Understanding Exploratory Data Analysis in Python – Types, Importance, and More

Want to understand everything about exploratory data analysis in Python? Let’s start with the what first.

In data science and especially predictive analytics, a model-building cycle goes through three stages: Pre-modeling, Modeling, and Post-modeling. While the Modeling stage deals with algorithm selection, model building, etc., and Post-modeling with model evaluation, validation, and deployment, the Pre-Modeling stage can make or break the model. It’s because predictive models work as well as the data they are built upon.

Several steps are taken in the Pre-Modeling stage to ensure that the data being fed to the model is of good quality. These include Exploratory Data Analysis (EDA), Data Preprocessing, and Feature Engineering (FE).

In this article, you will learn about EDA and its process, tools, and techniques, the difference between Data Preprocessing and Feature Engineering, and how to do Exploratory Data Analysis in Python.

Also Read: Data Analysis vs. Interpretation: What is the Difference?

What is Exploratory Data Analysis (EDA)?

EDA is the first step when developing a data model or analyzing data. EDA encompasses all the various subtasks that allow you to understand the data you are dealing with. These tasks include describing, summarizing, visualizing, and understanding the relationship between data features.

Therefore, as the term suggests, EDA refers to all the steps undertaken by a data science professional to “explore” and consequently understand the traits of data and its variables.

EDA typically aims to answer specific preliminary questions before any significant changes or manipulation can be done to the data and be fed to a predictive model.

These questions include-

• What kind of data am I dealing with?

Here the user tries to identify the fundamental characteristics of the data, such as its file format, volume, number of rows and columns, metadata, structure, type and data types of columns, etc.)

• What is the complexity level of the data?

Data may comprise many files and be connected through primary and secondary keys. The user, therefore, needs to understand such relations and find if the data is nested or if some fields have nested data.

• Does the data serve my purpose?

Data is extracted to serve a purpose. For example, if you want to develop a predictive model that forecasts Sales, then the data should have the sales number for the model to analyze patterns and extrapolate them. Therefore the data must be able to answer the business questions for which the data is being used.

• Is my data clean?

Users need to check if the data is unclean or not. Unclean data refers to data that has missing values, outliers, incorrect data types, unconventional column names, etc. The identification of such drawbacks in the data allows users to mark them and solve these issues at a later stage.

• What is the relationship between the features of the data?

Predictive models often have multicollinearity issues while seeking a strong relationship between the independent features and the dependent variables. Therefore, before starting, a user needs to understand the features’ relationship among themselves and how strong this relationship is.

While these are some of the critical questions that EDA tries to answer, it is not an exhaustive list. Let us look at the motivation behind performing EDA. 

Motivation for Exploratory Data Analysis

EDA doesn’t describe the data just for a better understanding of it. Instead of just describing, summarizing, and visualizing the data, it achieves multiple goals crucial in the later steps of Data Preprocessing, Feature Engineering, Model Development, Model Validation, Model Evaluation, and Model Deployment.

The main reasons why EDA is performed, apart from the few discussed above, are the following-

1. Identifying outliers and missing values
2. Identifying trends in time and space
3. Uncover patterns related to the target (dependent variable)
4. Creating hypotheses and testing them through experiments
5. Identifying new sources of data
6. Enable data-driven decision
7. Test underlying assumptions (especially for implementing statistical models)
8. Identify essential and unnecessary features
9. Examine distributions such as examination of Normality
10. Assess data quality
11. Graphical representation of data for quick analysis

Types of Exploratory Data Analysis in Python

The most crucial aspect of EDA is understanding the datasets and their variables. While certain EDA steps aim to provide some general information about the dataset, such as the number of rows and columns, column names, column data types, number of duplicate rows, etc.,  EDA’s crux understands the variables in-depth .

An EDA professional tries to describe the data and understand its features and characteristics and their relationship with other variables. Understanding the variables is crucial in identifying the right preprocessing and feature engineering required for the data.

The variable can be understood using statistics (non-graphical) and graphs (graphical) and by analyzing one (univariate), two (bivariate), or more (multivariate) variables together.

Therefore, the following types of EDA for understanding data are as follows-

Univariate non-graphical

Univariate analysis is the most basic analysis, where only one variable is understood at a time. In non-graphical univariate analysis, descriptive statistics are used along with some hypothesis tests, and basic exploratory functions are used to understand the data. The typical calculations done here include the following-

• data type
• number of duplicates
• type of variable
• mean
• median
• mode
• minimum and maximum
• interquartile range
• variance
• coefficient of variance
• standard deviation
• skewness
• kurtosis
• one sample t-test
• dependent t-test
• one-way ANOVA (F-test)
• number of missing values
• identification of outliers

Univariate graphical

Next, you can understand the variables by visualizing them. Various types of graphs can be created for this purpose, such as-

• Histogram
• Frequency Bar Chart
• Frequency Pie Chart
• Box Plot
• Univariate Line Chart

Also Read: How Univariate Analysis helps in Understanding Data

Bivariate non-graphical

Under non-graphical bivariate analysis, you analyze two variables using descriptive and mainly inferential statistics to understand their relationship. The typical steps performed here include the following-

• Aggregation and summarization using categorical and a numerical variable
• Cross frequency tables
• Correlation Coefficient
• Dependent t-Test
• Chi-Square Test

Bivariate graphical

In bivariate graphical analysis, those charts are created that use two variables to understand their connection. Typical graphs created at this stage include the following-

• Bar Chart
• Pie Chart
• Line Chart
• Scatter Plot
• Box Plot

Multivariate non-graphical

Often more than one variable needs to be analyzed together better to understand the level of complexity in the data. Common steps taken at this stage include

• Correlation Matrix
• Frequency or Statistical Cross Tables
• Simple Linear Regression

Multivariate graphical

Various graphs can also be created using more than two variables to help us understand how the variables are related. Standard multivariate charts include-

• Scatter Plot
• Heat Map
• Stacked / Dodged Bar Chart
• Multiline Chart

Performing Exploratory Data Analysis in Python

By taking an example dataset, let’s now address how to do EDA in Python [step-by-step guide].

Step 1: Define objective, acquire, and collate data

Many EDA projects in Python are out there that look at various kinds of datasets and explore them. However, this article considers data from a car insurance company. For example, there is a need to perform market analysis.

For that, the data science team needs to understand better their customer base and eventually create a predictive model that predicts the claim amount for a customer.

A customer 360 dataset is required to comprehend the customers fully where all information about the customer is there, from information about them to the information regarding the insurance products they bought.

Import data

A data science professional needs to get hold of the data that can solve complex business problems. The required data can often be fragmented and needs to be collected and combined into a single dataset for analysis. The data sources vary from surveys and social media analytics to customer reviews and log files. 

For example, the required data is in three separate sources, which you identify, access, and then import into your jupyter notebook to start the analysis.

The details of the three datasets are as follows-

• Dataset #1: Having customer’s demographic information
• Dataset #2: Having information about the insurance policy the customer has bought
• Dataset #3: Having information about the insurance claims

Once the required modules and the dataset are imported, you can view the dataset to understand its contents and how to combine them.

View data

The first data has details regarding the customer, such as their gender, education, income, etc.

The second dataset has information regarding the nature of the policy they have purchased.

The third data is information regarding the insurance claim, raised complaints, etc.

Collate data

‘Customer ID’ is the common variable in all three datasets and is the primary key to combining all three datasets into one. From this point forward, the EDA in Python will be performed on this dataset.

Step 2: Basic information

Column names

You first start by looking at all the column names to understand the kind of information in the data. Let’s say you notice the names in the data are not as per Python conventions and have space and other special characters.

The scope of EDA is to explore the data and identify such issues so that they can be addressed when performing data preprocessing and feature engineering. However, you can switch back and forth from EDA to other steps and perform preprocessing and feature engineering while performing EDA. Therefore, you can make the column per Python conventions in this case.

Removing the unnecessary characters from the column names makes it easier to operate on such columns when performing EDA in Python.

Check duplicate rows

You start by checking for any duplicate rows, as when performing data exploration in Python, duplicate rows can cause issues. In your case, you don’t have any duplicate rows.

No. of rows and columns

# finding number of rows and columns
print("number of rows: ", df.shape[0])
print("number of columns: ", df.shape[1])

The next important thing in Python data exploration is to know the dimensions of the data, as it allows you to use functionalities like .iloc if required. In your case, you have 9,134 rows and 26 columns.

Data type of columns

When performing exploratory data analysis in Python, you use various functions, often dependent on the column’s data type. Therefore identifying the data type of columns is important.

Combining the identified similar variable

You can perform another feature engineering step in the middle of EDA by looking at the columns above; a ‘Total_Claim Amount’ column can be created by summing all the claim amount columns.

Such a column will later be the dependent variable if you make a predictive model using this data. Here you add such a step of feature engineering to help you perform EDA.

Different Types of Exploratory Data Analysis in Python

Univariate non-graphical EDA

Let’s start with univariate non-graphical analysis, exploring one column at a time using statistics and other basic Python EDA functions.

Column Data Type

Interestingly, the column ‘Income_USD’ data type is being shown as ‘O’, i.e., Object, which refers to a string data type which should not be the case. As per the column name, the data type should be numeric, like float.

By exploring the variable or by looking at the data dictionary, you find that ‘NILL’ is being mentioned in place of 0 and ‘$’ and ‘, ‘ is in the data causing it to have a string-based data type.

You can replace the characters with appropriate values and perform feature engineering by converting the column’s data type.

Similarly, you can look at ‘Customer_Lifetime_Value_USD’ also. There too, seems to be a mismatch of data type.

Here it seems that in place of missings, some specials characters ‘$$$’ have been mentioned.

If required, you can perform feature engineering here, replace the special character with the missing value, and convert the column’s data type to an appropriate one.

Type of variable

The next important step in Python data exploration is to understand the type of variable you are dealing with. The type of variables are different from the data type of columns as here you try to understand the nature of data stored in it.

Common types of variables include numerical (continuous or discrete) and categorical (binary, nominal, or ordinal).

By looking at the categories of the column ‘Gender’, it is a nominal categorical variable that has been label encoded.

array(['Midsize', 'Small', 'Large'], dtype=object)

The column ‘Vehicle_Size’ categories suggest that it is an ordinal categorical variable.

Counting missing variables

Knowing the column’s missing values and its quantum is crucial when working on EDA projects in Python. If a column has an extremely high amount of missing values, then such a column may be dropped.

20

In your dataset, for example, the column ‘State’ seems to have 20 missing values. You can identify the missing values for all the columns and treat them using the appropriate imputation method based on the variable type or drop them.

Outlier identification

Statistics and statistical functions play an important role in Python data exploration, and they are highly susceptible to outliers. Therefore one needs to identify the outliers (and cap them as part of feature engineering) so that the subsequent results of EDA that use statistics are correct.

For example, you identify the upper outliers for the ‘Income_USD’ column using the IQR method and find two such outliers.

You can perform feature engineering here and cap them on the spot.

Duplicate values

series ([], Name: Customer_ID, dtype: bool)

In datasets that are at a customer level, you need to ensure that each row provides information about a unique customer. Therefore, here you check for duplicates in the customer id; luckily, there are no duplicates in the column.

Descriptive summary Statistics

Descriptive statistics play a crucial role in data exploration in Python. As seen above, various Python functions can describe a variable through its mean, median, mode, min, max, standard deviation, etc.

One sample T-test

In univariate analysis, some hypothesis testing can be done. Here you run a one-sample t-test to check if the average ‘Total_Claim_Amount’ is statistically different from $500.

Independent T-test

An Independent t-test allows you to understand whether the mean of two categories of a variable is similar. Here, for example, there is little proof that the income of male and female customers is statistically the same.

One-way ANOVA (F-test)

One Way Analysis of Variance, a kind of F-test, allows you to understand if the mean of two or more categories of a variable is similar. Here, for example, there is strong proof that the Income of individuals from different educational backgrounds is statistically similar.

F_onewayResult(statistic=15.32657653737315, pvalue=1.6967129487932849e-12)

Univariate Graphical EDA

The exploratory data analysis in Python is continued using one variable (univariate); however, now you use graphs to describe the variables.

• Histogram

If you create the histogram of ‘Total_Claim_Amount’, you can understand its distribution. Suppose you intend to use this variable as a dependent variable with a statistical algorithm like linear regression.

In that case, you may want to transform it to ensure its distribution is normal (Gaussian), which presently is not the case.

• Boxplot

The outliers can also be visualized using a boxplot. As seen before using the IQR outlier detection method, there are two outliers in ‘Income_USD.’

• Frequency Bar Chart

Frequency bar charts allow you to understand the count of each value in a categorical variable. As per the chart, most customers seem to be from California.

• Frequency Pie Chart

Just like frequency bar charts, frequency pie charts can also be created. As per the chart, most customers seem to be from a suburban location.

• Line Chart

A univariate line chart uses an index to form one axis. Such graphs are helpful when datetime is used to index data.

Bivariate Non-graphical EDA

The next step in exploratory data analysis in Python is to use two variables to analyze their relationship. Let’s start with bivariate analysis using statistics.

• Cross-frequency Tables

A cross-frequency table lets you know the frequency of each unique combination of categories from two categorical variables. For example, most customers are married and are located in suburban areas.

• Aggregation and Summarization

Cross tables can be created using one numerical and one categorical variable. For example, you calculate the sum of ‘Total_Claim_Amount’ for each location type. It seems customers make most claims from suburban areas.

• Correlation Coefficient

-0.3549593958604483

You can use the correlation coefficient to understand the relationship between two numerical columns. Your data shows a negative correlation between income and claim amount, i.e., as the income increases, the claim amount decreases.

• Chi-Square Test

You use the chi-square test to understand the relationship between two categorical columns. Here you convert the claim amount into bins such that they indicate the claim amount level.

If you run a chi-square to test the relationship between the claim level and location code, then it seems location code strongly influences the claim level.

• Dependent T-Test

Ttest_relResult(statistic=142.7974436778173, pvalue=0.0)

A dependent (paired) t-test allows you to understand whether the mean of two numerical variables is statistically the same. For example, there is strong proof that the Claim_Amount_1 and Claim_Amount_2 columns statistically have the same mean.

Bivariate Graphical EDA

Next, you try to understand the relationship between two columns by creating various graphs.

• Bar Chart

You can create a bar chart to understand the numerical and categorical relationship. Below – on the x-axis, you have location type; on the y-axis, the total claim amount.

• Pie Chart

Similar to a bar chart, a pie chart can also be created. Here the income % of all those customers who own different kinds of cars is identified.

• Line Chart

You can create a line chart to understand the relationship between two numerical columns. Above, the monthly premium is plotted against the number of complaints which shows that the premium decreases as the complaints increase.

• Scatterplot

The scatterplot is the most prominently used chart to understand the relationship between two numerical columns. Above, the scatterplot is created using income and claim amount.

• Boxplot

Bivariate boxplots can also be created where a boxplot is created using a numerical variable for each category of a categorical variable. In the example above, boxplots for each category of ‘Location_code’ is created using the income variable.

Multi-variate Non-Graphical EDA

Lastly, one can perform EDA by simultaneously using and analyzing more than two variables. Such an EDA is known as multivariate EDA. Let’s start with non-graphical EDA.

• Correlation Matrix

You can create a correlation matrix where for every variable, you get the correlation values against all other variables in the dataset. Such matrices are crucial when addressing the issue of multicollinearity and checking various assumptions for statistical algorithms.

• Multivariate Cross Table

A multivariate cross table can be created using frequency or summary statistics. Above, for example, a cross table is created using two categorical and four numerical columns.

• Multivariate Regression

You can run a simple linear regression to understand how certain variables are related to one variable. Above, for example, it seems that the income, gender, and number of complaints don’t seem to have a strong linear relationship with the claim amount.

Multivariate Graphical EDA

Let’s end the discussion on exploratory data analysis in Python by performing multivariate EDA using graphs.

• Scatterplot

You can easily create a scatterplot using two numerical and a categorical variable. Above, for example, the scatterplot between income and claim amount is created where each data point (dot) also indicates the location it belongs to.

• Stacked/Dodged Bar Chart

You can use stacked or dodged bar charts to visualize two categorical and one numerical variable. For example, the vehicle class and size are plotted against the claim amount, which shows that the claim amount is most for luxury cars, with all their sizes having roughly the same contribution.

• Multiline Chart

Multiline charts allow you to plot multiple numerical columns against another numerical or ordinal categorical column. Above, for example, the different claim amounts are plotted against the number of complaints.

• Heatmap

Heatmaps can be created to visualize tables such as a correlation matrix. Above, a heat map is created using the correlation matrix created during multivariate non-graphical EDA. Such a graphical matrix representation helps identify the highly correlated variables.

As you have seen, much work goes into performing EDA in Python. There are, however, few libraries that help you to come up with quick EDA reports. While they may not be as flexible as writing all the required code manually (as done above), they still help give you a good starting point.

A few of the important libraries are listed in the next section.

Tools & Libraries for Exploratory Data Analysis

While many tools can be used to perform EDA, the two primary tools are Python and R. Both are open-source programming languages that allow you to perform EDA and all sorts of data science operations, from data mining and cleaning to model development and deployment.

As seen above, In Python, EDA plays a crucial part when understanding the imported data, and one needs to write numerous functions. However, there are now dedicated libraries in Python and R that allow you to develop a detailed EDA report by writing a single line of code.

The most prominent EDA libraries are: 

Python Libraries

1. Pandas Profiling
2. Sweetviz
3. Autoviz
4. D-Tale
5. Skim
6. Dataprep
7. Klib
8. Dabl
9. SpeedML
10. Datatile

Also read: Python libraries to Learn for Machine Learning

R Libraries

1. Data Explorer
2. SmartEDA
3. DataMaid
4. Dlookr
5. GGally
6. Skimr

Also Read: 50 Ultimate Python Data Science Libraries to Learn

If you would run these libraries or if you would have noticed in the types of EDA section, descriptive statistics play a crucial role, especially in univariate analysis.

Therefore, to perform EDA effectively, you need to understand descriptive statistics and how it helps us understand the data. Let’s explore this topic next.

Understanding Descriptive Statistics

Statistics plays a crucial role in EDA. Statistics can be broadly divided into descriptive and inferential statistics. Descriptive statistics is where you try to summarize a variable/data by describing it.

In inferential statistics, several hypothesis tests are used to determine probabilities and assess the relationship of variables with the population.

A cube can be described by measuring its length, breadth, height, and mass, and a variable can be described by measuring frequency, central tendency, variability, and shape.

Also Read: Descriptive vs. Inferential Statistics

• Measure of Frequency

This measure calculates the count of each value in a variable. For example, in a categorical variable, the count of each category is calculated, while the same is done for a numerical variable where the frequency of each is calculated.

In the measure of frequency, the result can be presented in the form of cross tabs or through graphs such as histograms (for numerical variables) or bar charts and pie charts (for categorical variables).

• Measure of Central Tendency

The next crucial statistic that helps in describing data is its central value. A central value of the data can help you summarize it and give you a sense of it.

For example, if a variable has a data scientist’s salary in Mumbai, then a central value can easily define it.

However, how to calculate this central value is something that you need t understand as there is more than one way of calculating this value.

• Measure of Variability (Dispersion)

The next important measure is the measure of variability. Rather than calculating the center value, the data is summarized based on how much it varies. There are multiple measures of variability, such as-

1. Range:The simplest measure of variability/dispersion is a range which is the difference between the minimum and maximum value of the data. The issue with such a value is that it is unreliable as it only considers two values rather than all. Therefore two different data having two completely different levels of dispersion can have the same range as their minimum and maximum values are the same.
   
2. Mean absolute deviation:Another way of calculating deviation is the Mean Absolute Deviation (MAD), where the mean of the data is considered the central value, and the deviation of each value in the data from the central value (mean) is calculated. Once the deviation is calculated, their mean is calculated (mean deviation).However, as the sum of all deviations is always zero, the signs are ignored, and the absolute deviations are considered. The issue with such a method is that you violate certain algebraic rules by forcefully ignoring the negative sign and making it difficult to plug into a normal distribution.
   
3. Variance:Variance resolves the issue posed by MAD. Here rather than taking the absolute of the deviation, you square the deviation, thereby getting rid of the negative value. The problem with adding a square is that unit of variance measurement becomes in the square.For example, if the original variable were kg, the variance would be kg2. Another issue is that the outlier has a larger adverse impact by taking square.
   
4. Standard Deviation:To resolve the above issues to a certain degree, Standard Deviation is used, which is the square root of variance.

• Measure of Shape

Lastly, data can also be summarized based on the shape. Each numerical data has a distribution that can be understood by creating a histogram and analyzing its shape. The shape can then help you have a vague idea about the underlying probability distribution that can help you know various data characteristics.

A data can have a symmetric or asymmetric distribution where in symmetrical data, the data can be divided from the middle that returns two mirror opposites. Asymmetrical data, however, is skewed.

Common symmetrical data distributions include Uniform, Normal (Gaussian), etc.

So, for example, if you come across symmetric-shaped data with a bell-shaped curve, it can be a normal distribution where as per the 3-sigma rule, 68.27% of data falls between one, 95.25% between two, and 99.73% falls between three standard deviations.

Also, the mean, median, and mode are equal for a normal distribution. However, even if the distribution looks normal, one must ensure that the skewness and kurtosis are zero (i.e., the distribution has no abnormal peaks).

Data Pre-processings & Feature Engineering

Exploratory Data Analysis, Data Preprocessing, and Feature Engineering are fundamentally different techiness.

The confusion regarding them comes from all these techniques being performed during the Pre-Modeling stage, and the numerous sub-tasks performed under them often overlap.

Also, these techniques are not always performed one after the other, and it’s common to have a few EDA steps followed by Preprocessing, followed by some more EDA, followed by Feature Engineering, followed by EDA, and so on.

 To understand their differences, let’s explore each technique and the tasks performed under them.  

• Exploratory Data Analysis

As already discussed, EDA is used to examine and describe the data you are dealing with. It uses descriptive and inferential statistics along with the use of various graphs to make sense of the data and identify the issues with it.

• Data Pre-processing

If you would have noticed, in EDA, you always describe the data and identify the issues but never resolve them.

For example, during EDA, you would identify a few irrelevant unnecessary values in a variable; however, it’s under data preprocessing that you manipulate the data and eliminate such values.

Therefore the issues identified in the EDA are tried to be resolved under Data Preprocessing.

Common Preprocessing tasks can include

1. Removal of tags and other unnecessary values
2. Replacing abbreviations with their more meaningful full forms
3. Removal of Stopwords
4. Replacing keywords denoting missing value with NULL

• Feature Engineering

There is a huge overlap between Data Preprocessing and Feature Engineering. In feature engineering, we transform and thereby fundamentally alter the feature. Many of the issues identified in the EDA step are always rectified here.

For example, you may have calculated the mean of a variable and the number of missing values it has during EDA. However, under Feature Engineering, you will perform mean value imputation to plug this issue.

Also, feature engineering is an iterative process as features affect the model’s output, and you must constantly experiment with how you engineer the features to get the optimal result.

The idea behind Feature Engineering is to make the features ready for the model. Common tasks performed under Feature Engineering include-

1. Missing Value Treatment (dropping missing values, mean value imputation, etc.)
2. Outlier Capping
3. Scaling Data (standardization, normalization, min-max, etc.)
4. Converting categorical features into numerical features using Label Encoding or One-Hot Encoding
5. Derive new Features such as KPIs
6. Transforming the dependent variable to make it have a normal distribution
7. Feature Reduction (RFE, K-best, Stepwise Regression, etc.)
8. Feature Extraction (PCA, LDA, etc.)

Conclusion

EDA is vital in Data Science and is typically the first step when performing data analytics or predictive analytics.

You can perform exploratory data analysis in Python in a non-graphical (statistical) or graphical manner.

An effective EDA allows users to assess the data quality and identify the steps needed during the data preprocessing and feature engineering stage to prepare the predictive model.

Therefore, if you are working or plan to work in data science, you must understand what EDA is and its importance and explore how EDA can be performed.

FAQs

• What is EDA used for?

EDA is used to understand and summarize the data that is to be used for data analytics and predictive analytics.

• What are the types of EDA in Python?

EDA can be divided into non-graphical and graphical. In non-graphical EDA, various descriptive and inferential statistics are performed to understand the data, whereas, in graphical EDA, multiple charts are created to explore the data and describe it.

Each non-graphical and graphical EDA can be further of three types based on the number of variables used- univariate, bivariate, and multivariate.

• What is EDA in coding?

Python and R are typically used for performing EDA, where codes are written to understand the data.

I hope this article helped you understand EDA, its type, typical steps, tools, functions, and libraries to perform it effectively. If you want to know more about EDA or learn languages like Python and R that allow you to perform EDA, write back to us.