Pandas: An Essential Python Data Analysis Library

Pandas is a free, open-source Python library built for data manipulation and analysis. It treats tabular data the way a spreadsheet does, except every operation runs at the speed of compiled code. The pandas library gives you two core data structures and a consistent way to clean, filter, and reshape data before it reaches a machine learning pipeline or a computer vision workflow.

Wes McKinney created pandas in 2008 for financial data analysis. It has since become one of the most downloaded packages on the Python Package Index, and it now sits at version 3.0, a release that changes some long-standing defaults. This guide covers the core data structures, the functions you’ll use daily, and what’s new in pandas 3.0.

The Core Data Structures in Pandas

Pandas offers two core data structures: the Series (one-dimensional) and the DataFrame (two-dimensional), and nearly everything else in the library builds on them.

• DataFrame: A DataFrame is a two-dimensional table with labeled rows and columns, similar in spirit to R’s data.frame objects. Thanks to its size mutability, columns can be inserted or deleted without rebuilding the whole structure.
• Series: A Series is a one-dimensional labeled array holding a single data type, such as integers or strings, with every element carrying a matching label.

Pandas also converts ragged or differently-indexed NumPy arrays into DataFrame objects, and it imports and exports tabular data across CSV, JSON, Excel (.xlsx), and SQL database formats, so the same DataFrame can move between a notebook and a report without manual reformatting.

Installing Pandas: pip and Conda

Pandas is distributed through the Python Package Index, so getting started takes one command in a terminal or a Jupyter Notebook cell.

1. Install pandas from PyPI: pip install pandas, or from conda-forge: conda install -c conda-forge pandas
2. Import pandas: import pandas as pd
3. Create a DataFrame
4. Create a Series

#creating dataframe
data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Age': [25, 30, 28]}
df = pd.DataFrame(data)
print(df)

#Creating Series
fruits = ['apple', 'banana', 'orange']
series = pd.Series(fruits)
print(series)

Core Functionalities in Pandas

Pandas covers every stage of a typical data science pipeline:

1. Selection and Indexing: By label (loc), position (iloc), or a mix of both.
2. Data Cleaning: Handling of missing data, duplicates, and inconsistent entries.
3. Data Transformation: Flexible reshaping and pivoting, merging, and sorting.
4. Data Filtering: Boolean-based subsetting.
5. Statistical Analysis: Descriptive statistics and explicit data alignment.
6. Group Operations: A groupby engine to perform split-apply-combine, aggregating and transforming data by category.
7. Time Series: Date range generation, frequency conversion, moving window statistics, date shifting.

Data Selection and Indexing

Pandas offers several ways to select rows and columns:

Using loc for Label-Based Selection

• Single column: df.loc[:, ‘column_name’]
• Multiple columns: df.loc[:, [‘column_name1’, ‘column_name2’]]
• Single row: df.loc[‘row_label’]
• Multiple rows: df.loc[[‘row_label1’, ‘row_label2’]]

Using iloc for Position-Based Selection

• Single column: df.iloc[:, 2]
• Multiple columns: df.iloc[:, [1, 3]]
• Single row: df.iloc[4]
• Multiple rows: df.iloc[[1, 3]]

Boolean Indexing

• By condition: df[df[‘column_name’] > value]
• loc with a condition: df.loc[df[‘column_name’] == ‘value’, [‘column1’, ‘column2’]]

import pandas as pd
import numpy as np

# Creating a sample DataFrame
data = {
    'Name': ['John', 'Anna', 'Peter', 'Linda'],
    'Age': [28, 34, 29, 32],
    'City': ['New York', 'Paris', 'Berlin', 'London'],
    'Salary': [68000, 72000, 71000, 69000]
}
df = pd.DataFrame(data)

# Using loc for label-based selection
multiple_columns_loc = df.loc[:, ['Name', 'Age']]

# Using iloc for position-based selection
multiple_columns_iloc = df.iloc[:, [0, 1]]

# Rows based on condition
rows_based_on_condition = df[df['Age'] > 30]

# Using loc with a condition
loc_with_condition = df.loc[df['City'] == 'Paris', ['Name', 'Salary']]

#output 

Using loc for Label-Based Selection

    Name  Age
0   John   28
1   Anna   34
2  Peter   29
3  Linda   32

Rows Based on Condition (Age > 30)

    Name  Age    City  Salary
1   Anna   34   Paris   72000
3  Linda   32  London   69000

Using loc with a Condition (City == 'Paris')
   Name  Salary
1  Anna   72000

Data Cleaning and Handling Missing Values

Real-world data almost always has gaps and inconsistent entries. In pandas, missing values are represented as NaN (or NaT for dates), whether the column holds floating-point or non-floating-point data.

Identifying Missing Values

• Check: df.isnull() or df.isna() return a boolean DataFrame flagging NaN entries.
• Count: df.isnull().sum() counts missing entries per column.

Handling Missing Values

• Remove: df.dropna() for rows, df.dropna(axis=1) for columns.
• Fill: df.fillna(value), or df[‘column’].fillna(df[‘column’].mean()) for a column’s own mean.

Data Transformation

• Duplicates: df.drop_duplicates()
• Renaming: df.rename(columns={‘old_name’: ‘new_name’})
• Type changes: df.astype({‘column’: ‘dtype’})
• Custom logic: df.apply(lambda x: func(x))

# Sample data with missing values
data = {'Name': ['John', 'Anna', 'Peter', None],
        'Age': [28, np.nan, 29, 32],
        'City': ['New York', 'Paris', None, 'London']}
df = pd.DataFrame(data)

# Check for missing values
missing_values_check = df.isnull()

# Count missing values
missing_values_count = df.isnull().sum()

# Remove rows with any missing values
cleaned_df_dropna = df.dropna()

# Fill missing values with a specific value
filled_df = df.fillna({'Age': df['Age'].mean(), 'City': 'Unknown'})

# Removing duplicates (assuming df has duplicates for demonstration)
deduped_df = df.drop_duplicates()

# Renaming columns
renamed_df = df.rename(columns={'Name': 'FirstName'})

# Changing data type of Age to integer (after filling missing values for demonstration)
df['Age'] = df['Age'].fillna(0).astype(int)

missing_values_check, missing_values_count, cleaned_df_dropna, filled_df, deduped_df, renamed_df, df

#ouputs


Check for missing values
   Name    Age   City
0  False  False  False
1  False   True  False
2  False  False   True
3   True  False  False

Counting Missing Values
Name 1
Age 1
City 1
dtype: int64

Removing the missing values

 Name Age City
0 John 28.0 New York

Data Filtering and Statistical Analysis

The same DataFrame can come from a CSV file, an API response, or a query against a SQL database using pd.read_sql(), and the workflow below applies regardless of where the data originated.

• Descriptive Statistics: describe(), mean(), and sum() summarize central tendency and spread.
• Correlation: corr() measures the relationship between variables.
• Aggregation: groupby() and agg() summarize data by category.

# Sample data creation
sales_data = {
    'Product': ['Table', 'Chair', 'Desk', 'Bed', 'Chair', 'Desk', 'Table'],
    'Category': ['Furniture', 'Furniture', 'Office', 'Furniture', 'Furniture', 'Office', 'Furniture'],
    'Sales': [250, 150, 200, 400, 180, 220, 300]
}
sales_df = pd.DataFrame(sales_data)

inventory_data = {
    'Product': ['Table', 'Chair', 'Desk', 'Bed'],
    'Stock': [20, 50, 15, 10],
    'Warehouse_Location': ['A', 'B', 'C', 'A']
}
inventory_df = pd.DataFrame(inventory_data)

# Merging sales and inventory data on the Product column
merged_df = pd.merge(sales_df, inventory_df, on='Product')

# Filtering data 
filtered_sales = merged_df[merged_df['Sales'] > 200]

# Statistical Analysis
# Basic descriptive statistics for the Sales column
sales_descriptive_stats = merged_df['Sales'].describe()

#ouputs

Filtered Sales Data (Sales > 200):
  Product   Category  Sales  Stock Warehouse_Location
0   Table  Furniture    250     20                  A
3     Bed  Furniture    400     10                  A
5    Desk     Office    220     15                  C
6   Table  Furniture    300     20                  A

Descriptive Statistics for Sales:
count      7.000000
mean     242.857143
std       84.599899
min      150.000000
25%      190.000000
50%      220.000000
75%      275.000000
max      400.000000
Name: Sales, dtype: float64

Data Visualization

Pandas wraps Matplotlib closely enough that basic charts need no extra import. The .plot() method turns a DataFrame or Series into a line, bar, or histogram in one line of code.

What’s New in Pandas 3.0

Pandas 3.0, released in January 2026, is the first major version bump in nearly three years, changing default behavior that pandas code has relied on for over a decade.

The biggest breaking change: chained assignment (df[‘col’][df[‘x’] > 5] = 100) no longer silently works. Every indexing operation now behaves as a copy, so updates must go through .loc in one step. In exchange, SettingWithCopyWarning is gone for good.

Install it with pip install –upgrade pandas, or via conda-forge with conda install -c conda-forge pandas=3.0. The official pandas 3.0 announcement and the PyPI project page cover the full migration notes.

Pandas AI: Query Your Data in Plain English

PandasAI is a third-party library built on pandas that lets you query a DataFrame in natural language instead of writing groupby() and merge() calls by hand. Here is the GitHub repo.

Install it with pip install pandasai. The example below ranks sales by country, including the United States, without writing a single aggregation:

import os
import pandas as pd
from pandasai import Agent

# Sample DataFrame
sales_by_country = pd.DataFrame({
    "country": ["United States", "United Kingdom", "France", "Germany", "Italy", "Spain", "Canada", "Australia", "Japan", "China"],
    "sales": [5000, 3200, 2900, 4100, 2300, 2100, 2500, 2600, 4500, 7000]
})

# By default, unless you choose a different LLM, it will use BambooLLM.
# You can get your free API key signing up at https://pandabi.ai (you can also configure it in your .env file)
os.environ["PANDASAI_API_KEY"] = "YOUR_API_KEY"

agent = Agent(sales_by_country)
agent.chat('Which are the top 5 countries by sales?')

output: China, United States, Japan, Germany, Australia

agent.chat(
    "What is the total sales for the top 3 countries by sales?"
)

output: The total sales for the top 3 countries by sales is 16500.

Real-World Use Cases of Pandas

Pandas shows up well beyond the data science team:

• Scientific Computing: Paired with NumPy and scikit-learn, pandas handles the tabular data behind most numerical Python work.
• Machine Learning: Pandas handles feature engineering, turning raw inputs into the columns a model trains on. See our guide to machine learning algorithms and our notes on evaluating model performance.
• Time Series: Retail and manufacturing teams forecast demand with the date range and frequency conversion tools above.
• Vision and Sensor Metadata: Computer vision pipelines generate structured data: detection logs, coordinates, timestamps, confidence scores. Pandas cleans that metadata before it reaches an agentic computer vision system, on-premises or at the network edge.
• Knowledge Graphs and NLP: Entity tables from natural language processing often land in a DataFrame before a knowledge graph.