What are the key steps you follow when performing Exploratory Data Analysis on a new dataset?

Question

Accepted Answer

When approaching a new dataset, I follow a systematic process to ensure thorough exploration. Each step builds on the previous one to develop a complete understanding of the data.

The first step is understanding the dataset structure. I examine the shape of the data using df.shape to see the number of rows and columns. I review column names and data types with df.info() to understand what variables are available and how they are stored. I use df.head() and df.tail() to preview actual data values and get a sense of what each column contains.

Next, I generate summary statistics using df.describe() to calculate measures like mean, median, standard deviation, minimum, maximum, and quartiles for numerical columns. For categorical columns, I use value_counts() to see the distribution of categories and identify the most common values. These statistics provide a quick numerical summary of each variable.

The third step involves checking data quality. I assess missing values with df.isnull().sum() to identify columns with incomplete data. I check for duplicates using df.duplicated().sum() and investigate their cause. I look for inconsistent values, such as negative ages or future dates, that indicate data entry errors.

Then I analyze individual variables through univariate analysis. For numerical variables, I create histograms to understand distributions and identify skewness. For categorical variables, I use bar charts to visualize category frequencies. Box plots help identify outliers and understand the spread of data.

The fifth step explores relationships between variables through bivariate and multivariate analysis. Scatter plots reveal relationships between pairs of numerical variables. Correlation matrices show the strength and direction of linear relationships. Cross-tabulations and grouped statistics compare categorical variables.

Finally, I document findings and formulate hypotheses. I note patterns, anomalies, and interesting relationships discovered during exploration. These findings guide subsequent data cleaning, feature engineering, and model selection decisions.

Throughout this process, I iterate as needed. New discoveries often prompt revisiting earlier steps with fresh questions.