Data Analysts Interview Questions Data Science

Data Validation is basically the process of validating data. This step plays one of the important roles in the process of data analysis. It mainly involves two processes namely, Data Screening and Data Verification.

• Data Screening: Various algorithms are used in this step in order to screen the entire data and find out all inaccurate values.
• Data Verification: This step is mainly to evaluate each and every suspected value in various use-cases and then decide whether to include those values in the data or not or suppose the values have to be rejected as invalid or if they have to be replaced with some redundant values.

• Tableau
• RapidMiner
• OpenRefine
• KNIME
• Google Search Operators
• Solver
• NodeXL
• io
• Wolfram Alpha’s
• Google Fusion tables

The different types of hypothesis testing are as follows:

• T-test: T-test is used when the standard deviation is unknown and the sample size is comparatively small.
• Chi-Square Test for Independence: These tests are used to find out the significance of the association between categorical variables in the population sample.
• Analysis of Variance (ANOVA): This kind of hypothesis testing is used to analyze differences between the means in various groups. This test is often used similarly to a T-test but, is used for more than two groups.
• Welch’s T-test: This test is used to find out the test for equality of means between two population samples.

Usually, data is distributed in different ways with a bias to the left or to the right or it can be all spilled across. However, there are chances that data is distributed around a central value without any bias to the left or right and reaches normal distribution in the form of a bell-shaped curve. A normal distribution is an arrangement of a data set in which most values cluster in the middle of the range and the rest taper off symmetrically toward either extreme.

The random variables are distributed in the form of a symmetrical bell-shaped curve.

Properties of Normal Distribution:

1. Unimodal - one mode
2. Symmetrical - left and right halves are mirror images
3. Bell-shaped - maximum height (mode) at the mean
4. Mean, Mode, and Median are all located in the centre
5. Asymptotic

In order to understand the alternate hypothesis, it is necessary to first know the null hypothesis. The null hypothesis is a statistical phenomenon which is used to test for possible rejection under the assumption that the result of chance would be true.

An alternative hypothesis, on the other hand, is also a statistical phenomenon which contradicts the null hypothesis. It is considered that the observations are a result of an effect with some chance of variation.

A data scientist must have the following skills

• Database knowledge
  • Database management
  • Data Blending
  • Querying
  • Data manipulation
• Predictive Analytics
  • Basic descriptive statistics
  • Predictive modelling
  • Advanced analytics
• Big Data Knowledge
  • Big data analytics
  • Unstructured data analysis
  • Machine learning
• Presentation skill
  • Data visualization
  • Insight presentation
  • Report design

Statistical methods that are useful for a data scientist are:

• Bayesian method
• Markov process
• Spatial and cluster processes
• Rank statistics, percentile, outliers detection
• Imputation techniques, etc.
• Simplex algorithm
• Mathematical optimization

Some of the missing patterns which are frequently observed by data analyst professionals are -

• Missing at Random
• Missing completely at Random
• Missing that depends on the missing value itself
• Missing that depends on the unobserved input variable

The three types of analysis methodologies have single, double or multiple variables.

• Univariate analysis: This has only one variable and therefore there are no relationships, causes. The main aspect of the univariate analysis is to summarize the data and find the patterns within it to make actionable decisions.
• Bivariate analysis: This deals with the relationship between two sets of data. These sets of paired data come from related sources, or samples. Some of the tools used to analyze such data includes chi-squared tests and t-tests when the data have a correlation. The strength of the correlation between the two data sets will be tested in a Bivariate analysis.
• Multivariate analysis:  This is similar to bivariate analysis. It is a set of techniques used for the analysis of data sets that contain more than one variable, and the techniques are especially valuable when working with correlated variables.

Linear regression is a statistical model which attempts to fit the best possible straight line between the independent and the dependent variables when a set of input features are given. As the output is continuous, the cost function measures the distance from the observed to the predicted values. It can be said to be an appropriate choice to solve regression problems, for example, predicting sales number.

On the other hand, Logistic regression gives probability as its output. By definition, it is a bounded variable between zero and one, due to the sigmoid activation function. It is more appropriate to solve classification problems, for example, predicting whether a transaction is a fraud or not.

Eigenvectors are used to understand linear transformations and are calculated for a correlation or a covariance matrix. Eigenvectors are basically the directions along which a specific linear transformation acts either by compressing, flipping or stretching.

Eigenvalues refer to the strength of the transformation or the factor by which the compression occurs in the direction of eigenvectors.

Time series analysis is a statistical technique which deals with time series data or trend analysis. It helps to understand the underlying forces leading to a particular trend in the time series data points. Time series data is the data that is in a series of particular time periods or intervals. The types of data considered are -

• Time series data - This is a set of observations on the values that a variable takes at different times
• Cross-sectional data - This is the data of one or more variables that are collected at the same point in time
• Pooled data - This is a combination of both time series data and cross-sectional data

Time series analysis can be performed in two domains - frequency domain and time domain

The process of Imputation involves replacing missing data with substituted values. There are mainly three problems caused by missing data -

1. Missing data can result in a substantial amount of bias
2. It can make the handling and analysis of the data more arduous
3. It can create reductions in efficiency

Due to these problems, missing data can create problems for analyzing data. Imputation avoids such problems involved with listwise deletion of cases which have missing values. The different types of imputation techniques are -

Single Imputation (the missing value is replaced by a value)

• Hot-deck imputation - With the help of a punch card, a missing value is imputed from a randomly selected similar record
• Cold deck imputation - It is similar to hot-deck imputation except it is more advanced and it selects donors from another dataset
• Mean imputation - It replaces the missing values with the mean of that variable for all other cases
• Regression imputation -  It replaces the missing value with the predicted values of a variable based on other variables
• Stochastic regression - It is similar to regression imputation, it adds the average regression variance to regression imputation

Multiple Imputation (method for handling missing data in multivariate analysis)

• Multiple imputations estimate the values multiple times. It follows three steps:
  1. Imputation – Missing values are imputed similar to single imputation. Here, the imputed values are drawn m times from a distribution instead of just once. There are m completed datasets after the end of this step.
  2. Analysis – Each of these m datasets is analyzed in this step and at the end of this step there are m analyses.
  3. Pooling –  Here, the m results are consolidated into one single result after calculating the mean, variance and confidence interval of the concerned variable.

In the case of unstructured data, the iterative process should be used to classify data. Taking some data samples and modifying the model accordingly in order to evaluate the same for accuracy. It is necessary to always use the basic process for data mapping. Also, data mining, data visualization techniques, algorithm designing and so on needs to be performed properly. It becomes easier to convert unstructured data into well-documented data files as per customer trends if all these processes are performed accordingly.

KPI or Key Performance Indicator can be defined as the metric which consists of a combination of charts, reports, spreadsheets or business processes.

Design of experiment is the initial process which is used to split data, data sampling or data setup for statistical analysis.

80/20 rule, also known as the Pareto principle is the law of the vital few or the principle of factor sparsity. It states that, for many events, roughly 80% of the effects come from 20% of the causes.

Clustering is defined as the task of dividing the population or data points into a number of groups such that data points in the same group are more similar to other data points in the same group than those in other groups. Basically, the main aim is to segregate groups with similar traits and assign them into clusters.

Clustering can be divided into two subgroups :

• Hard Clustering: In hard clustering, each data point either belongs to a cluster completely or not.
• Soft Clustering: In soft clustering, instead of putting each data point into a separate cluster, a probability or likelihood of that data point to be in those clusters is assigned.

Correlation is a statistic that measures the strength and direction of the associations between two or more variables.

On the other hand, causation is a relationship that describes cause and effect.

“Correlation does not imply causation.” This statement warns us about the dangers of the common practice of looking at a strong correlation and assuming causality. A strong correlation may manifest without causation in the following cases:

• Lurking variable: An unobserved variable that affects both variables of interest, causing them to exhibit a strong correlation, even when there is no direct relationship between them.
• Confounding variable: A confounding variable is one that cannot be isolated from one or more of the variables of interest. Therefore we cannot explain if the result observed is caused by the variation of the variable of interest or of the confounding variable.
• Spurious correlation: Sometimes due to coincidence, variables can be correlated even though there is no reasonably logical relationship.

Causation is tricky to be inferred. The most usual solution is to set up a randomized experiment, where the variable that’s a candidate to be the cause is isolated and tested. Unfortunately, in many fields running such an experiment is impractical or not viable, so using logic and domain knowledge becomes crucial to formulating reasonable conclusions.

There are plenty of alternatives to handle missing data, although none of them is perfect or fits all cases. Some of them are:

• Dropping incomplete rows: Simplest of all, it can be used if the amount of missing data is small and seemingly random.
• Dropping variables: This can be used if the proportion of missing data in a feature is too big and the feature is of little significance to the analysis. In general, it should be avoided, as it usually throws away too much information.
• Considering “not available” (NA) to be a value: Sometimes missing information is information in itself. Depending on the problem domain, missing values are sometimes non-random: Instead, they’re a byproduct of some underlying pattern.
• Value imputation: This is the process of estimating the value of a missing field given other information from the sample. There are various viable kinds of imputation. Some examples are mean/mode/median imputation, KNN, regression models, and multiple imputations.

Data is visually represented with the help of a chart using locations in the image (height, width and depth). Dealing with beyond three dimensions, we need to add more information and make use of other visual cues.

Some of the most common cues are -

• Colour - It is a visually appealing and an intuitive way to depict both continuous and categorical data.
• Size - Marker size can be used to represent continuous data. It can also be used for categorical data but as the size differences are more difficult to detect than colour, this cue might not be an appropriate choice for this type of data.
• Shape - Shapes are an effective way to represent different classes in a data set.

While performing data analysis, there are few problem data analysts face, some of them are mentioned below:

• Poor formatted data file or the presence of duplicate entries in the data file leads to a reduction in the quality of data. Also, there can unexpected commas and black spaces in columns, which results in incomplete and inconsistent data.
• While you extract data from a poor source, it can be a problem as you will end up spending a lot of time in data cleaning.
• As we extract data from various sources, there are chances that the data might vary in representation. In such cases, you might have to combine those data from all other sources which will result in a delay.
• Misclassified and incomplete data sometimes tends to be a big problem while you perform data analysis.

There are few criteria mentioned below which can guide you to decide whether the model developed is good or not:

• As per the dataset, a good model should be able to predict as accurately as possible.
• A good model should have the ability to adapt easily according to business requirements.
• Suppose there is a change or addition to the data, a good model should be able to scale accordingly.
• A model is said to be good when it can easily be consumed by the clients for actionable and profitable results.

In statistics, a variance is said to be the spread of a data set. It is a measurement which is used to identify how far each number in the data set is from the mean. During market research, variance plays an important role in calculating the probabilities of future events. It is used to find all the possible values and likelihoods that a random variable can take within a given advantage. If the value of variance is zero, it means that all the values within a data set are identical, and the variances which are not equal will come in the form of positive numbers. The larger the value of variance, the more spread in the data set. If the variance is large, it means the number in the data set are far from the mean and each other whereas a smaller variance implies that the number is closer together in value.

On the other hand, covariance provides insights into the relationship between two variables. It is the measurement of how two random variables in a data set will change together. There are two types of covariance - positive covariance and negative covariance. A positive covariance means that two variables are positively related and are moving in the same direction. Similarly, negative covariance is when the variables are inversely related or are moving in opposite directions.