In Pandas – we learnt to **create and access** data. Now in Pandas – II, we go deeper — **analysing, cleaning, and summarising** real data! Think of it as going from knowing how to read a report card, to actually calculating toppers, averages, and finding who was absent! 📊
> [!TIP]
> **How to use these notes:** Run every code example yourself in Python! Focus especially on **Section 2.4 (Descriptive Statistics)**, **Section 2.6 (Missing Data)**, and **Section 2.7 (groupby)** — these appear in EVERY board exam. Simple analogies are used throughout — no stress! 😊
---
## 2.1 🌟 Introduction
In this chapter, we move from just storing data to actually **working with it**:
```mermaid
graph LR
P1["📘 Pandas I\nCreate & Store Data\nSeries & DataFrames"]
P2["📗 Pandas II\nAnalyse & Clean Data\nStats, Iteration, Groups"]
P1 -->|"Next Level"| P2
style P1 fill:#2196F3,color:#fff
style P2 fill:#4CAF50,color:#fff
```
::: grid
::: card 🔄 | Iterating | Visit each row or column one by one | Like reading a register line by line
::: card ➕ | Binary Ops | Math between two DataFrames | Add marks of two tests together
::: card 📊 | Statistics | Mean, Median, Max, Min... | Instantly find class topper or average
::: card 🧹 | Missing Data | Handle NaN (absent students!) | Fill or remove missing entries
::: card 📦 | groupby() | Group data by category | Section-wise or city-wise analysis
:::
**Our Master DataFrame — used throughout this chapter:**
```python
# master_setup.py
import pandas as pd
import numpy as np
data = {
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
'Maths': [90, 85, 78, 92, 88],
'Science': [88, 95, np.nan, 90, 82], # Chirag was absent!
'English': [75, 80, 72, 85, 79],
'Section': ['A', 'A', 'B', 'B', 'A']
}
df = pd.DataFrame(data)
print(df)
```
**Output:**
```
Name Maths Science English Section
0 Arjun 90 88.0 75 A
1 Bina 85 95.0 80 A
2 Chirag 78 NaN 72 B
3 Divya 92 90.0 85 B
4 Esha 88 82.0 79 A
```
> NaN (Not a Number) in Chirag's Science row — he was **absent** for that exam!
---
## 2.2 🔄 Iterating Over a DataFrame
**Iteration** means visiting each part of the DataFrame one by one — like reading a register row by row or column by column.
```mermaid
graph LR
IT["Iterating\nover DataFrame"]
IR["iterrows()\nRow by Row\n(Horizontal)"]
IC["iteritems()\n/ items()\nColumn by Column\n(Vertical)"]
IT --> IR
IT --> IC
style IR fill:#FF9800,color:#fff
style IC fill:#9C27B0,color:#fff
```
---
### 2.2.1 Using `iterrows()` — Row by Row 🔄
`iterrows()` goes through the DataFrame **one row at a time**. Each time, it gives you:
- The **row index** (number or label)
- A **Series** containing all values in that row
> **Analogy:** Imagine a teacher reading the class register — going student by student from top to bottom. That's `iterrows()`!
```python
# iterrows_demo.py
import pandas as pd
import numpy as np
data = {
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya'],
'Maths': [90, 85, 78, 92],
'English': [75, 80, 72, 85]
}
df = pd.DataFrame(data)
print("Reading students one by one:")
for index, row in df.iterrows():
print(f"Row {index}: {row['Name']} → Maths={row['Maths']}, English={row['English']}")
```
**Output:**
```
Reading students one by one:
Row 0: Arjun → Maths=90, English=75
Row 1: Bina → Maths=85, English=80
Row 2: Chirag → Maths=78, English=72
Row 3: Divya → Maths=92, English=85
```
**Practical Example — Find students who passed (Maths ≥ 80):**
```python
# find_passers.py
print("Students who scored ≥ 80 in Maths:")
for index, row in df.iterrows():
if row['Maths'] >= 80:
print(f" ✅ {row['Name']} — {row['Maths']}")
```
**Output:**
```
Students who scored ≥ 80 in Maths:
✅ Arjun — 90
✅ Bina — 85
✅ Divya — 92
```
> [!NOTE]
> **iterrows() is slow for large data! 🧠**
> For small DataFrames (like in CBSE exams), iterrows() is fine. But for millions of rows, it's slow. Real data scientists prefer vectorized operations (like `df[df['Maths'] >= 80]`). For your board exam, always know iterrows() as the correct answer!
---
### 2.2.2 Using `iteritems()` — Column by Column 📊
`iteritems()` goes through the DataFrame **one column at a time**. Each time, it gives you:
- The **column name** (header)
- A **Series** containing all values in that column
> **Analogy:** Instead of reading row by row (student by student), now you're reading column by column (subject by subject) — first all Maths marks, then all English marks...
```python
# iteritems_demo.py
import pandas as pd
data = {
'Maths': [90, 85, 78, 92],
'Science': [88, 95, 70, 90],
'English': [75, 80, 72, 85]
}
df = pd.DataFrame(data)
print("Reading subjects one by one:")
for col_name, col_data in df.items(): # Note: use .items() in newer Pandas
print(f"\nSubject: {col_name}")
print(f" All marks: {col_data.values}")
print(f" Class Average: {col_data.mean():.2f}")
```
**Output:**
```
Reading subjects one by one:
Subject: Maths
All marks: [90 85 78 92]
Class Average: 86.25
Subject: Science
All marks: [88 95 70 90]
Class Average: 85.75
Subject: English
All marks: [75 80 72 85]
Class Average: 78.00
```
> [!WARNING]
> **iteritems() vs items() — Important!**
> `iteritems()` was the old name in Pandas. In **newer versions of Pandas (1.5+)**, it is **deprecated (removed)** and replaced with `items()`. Both do exactly the same thing.
> In CBSE exams: both `iteritems()` and `items()` are accepted in answers!
**Comparison Table:**
| Feature | `iterrows()` | `iteritems()` / `items()` |
| :--- | :--- | :--- |
| **Traversal** | Row by row | Column by column |
| **Direction** | Horizontal (left-right) | Vertical (top-bottom) |
| **Gives you** | (index, Series of row) | (col_name, Series of column) |
| **Use when** | Processing student-wise | Processing subject-wise |
> [!IMPORTANT]
> **Board Exam Tip**
> "What is the difference between iterrows() and iteritems()?" — **2-mark** question!
> Answer: `iterrows()` iterates **row by row** (returns index and row as Series). `iteritems()` iterates **column by column** (returns column name and column as Series).
---
## 2.3 ➕ Binary Operations in a DataFrame
**Binary operations** are mathematical operations (+, -, *, /) performed between **two DataFrames** (or a DataFrame and a number/Series).
> **Analogy:** Imagine you have **Test 1 marks** in one sheet and **Test 2 marks** in another sheet. Adding them gives you **Total marks**. That's a binary operation between two DataFrames!
```mermaid
graph LR
D1["📊 DataFrame 1\nTest 1 Marks"]
D2["📊 DataFrame 2\nTest 2 Marks"]
OP["➕ Binary Operation\n(Addition)"]
RES["📊 Result\nTotal Marks"]
D1 --> OP
D2 --> OP
OP --> RES
style OP fill:#FF9800,color:#fff
style RES fill:#4CAF50,color:#fff
```
**Basic Binary Operations:**
```python
# binary_ops_demo.py
import pandas as pd
test1 = pd.DataFrame({
'Math': [45, 38, 42],
'Science': [40, 45, 35]
}, index=['Arjun', 'Bina', 'Chirag'])
test2 = pd.DataFrame({
'Math': [48, 40, 38],
'Science': [42, 48, 40]
}, index=['Arjun', 'Bina', 'Chirag'])
print("Test 1:\n", test1)
print("\nTest 2:\n", test2)
print("\nTotal (test1 + test2):")
print(test1 + test2)
print("\nDifference (test2 - test1):")
print(test2 - test1)
```
**Output:**
```
Test 1:
Math Science
Arjun 45 40
Bina 38 45
Chirag 42 35
Test 2:
Math Science
Arjun 48 42
Bina 40 48
Chirag 38 40
Total (test1 + test2):
Math Science
Arjun 93 82
Bina 78 93
Chirag 80 75
Difference (test2 - test1):
Math Science
Arjun 3 2
Bina 2 3
Chirag -4 5
```
**What happens when indexes DON'T match? — NaN appears! ⚠️**
```python
# mismatched_binary.py
import pandas as pd
df1 = pd.DataFrame({'A': [10, 20], 'B': [30, 40]}, index=[0, 1])
df2 = pd.DataFrame({'A': [5, 5], 'B': [5, 5]}, index=[1, 2])
print("DF1:\n", df1)
print("\nDF2:\n", df2)
print("\nDF1 + DF2:")
print(df1 + df2)
```
**Output:**
```
DF1:
A B
0 10 30
1 20 40
DF2:
A B
1 5 5
2 5 5
DF1 + DF2:
A B
0 NaN NaN ← index 0 in DF1, but NOT in DF2 → NaN
1 25.0 45.0 ← index 1 in BOTH → added normally!
2 NaN NaN ← index 2 in DF2, but NOT in DF1 → NaN
```
**The `fill_value` trick — replace NaN with 0 during operation:**
```python
print(df1.add(df2, fill_value=0))
```
**Output:**
```
A B
0 10.0 30.0 ← 10 + 0 (fill_value fills missing df2 values with 0)
1 25.0 45.0
2 5.0 5.0 ← 0 + 5 (fill_value fills missing df1 values with 0)
```
**All Binary Operation Functions:**
| Operation | Symbol | Function | Example |
| :--- | :--- | :--- | :--- |
| Addition | + | `add()` | `df1.add(df2, fill_value=0)` |
| Subtraction | - | `sub()` | `df1.sub(df2, fill_value=0)` |
| Multiplication | * | `mul()` | `df1.mul(df2, fill_value=1)` |
| Division | / | `div()` | `df1.div(df2, fill_value=1)` |
> [!IMPORTANT]
> **Board Exam Tip**
> "What happens when you perform a binary operation between two DataFrames with different row indexes?" — **2-mark** question!
> Answer: Pandas aligns data by **both row index and column label**. Where matching labels exist, the operation is performed normally. Where labels exist in one DataFrame but not the other, the result is **NaN** (missing value).
---
## 2.4 📊 Descriptive Statistics with Pandas
**Descriptive Statistics** = functions that summarise your data — finding the average, highest, lowest, middle value, etc. Pandas makes all of this incredibly easy!
> **Analogy:** After a class test, a teacher wants to know: Who scored highest? What's the class average? What's the middle score? All of this is "descriptive statistics" — and Pandas answers all these questions in one line each!
**Using this DataFrame for all examples:**
```python
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Maths': [90, 85, 78, 92, 88],
'Science': [88, 95, np.nan, 90, 82],
'English': [75, 80, 72, 85, 79]
}, index=['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'])
```
> **Remember:** All statistical functions **automatically skip NaN values** unless told otherwise!
---
### 2.4.1 Functions `min()` and `max()` 📉📈
```python
# min_max_demo.py
# Column-wise (default) — min/max of each subject
print("Lowest marks per subject:")
print(df.min())
print("\nHighest marks per subject:")
print(df.max())
# Row-wise (axis=1) — min/max per student
print("\nLowest marks per student:")
print(df.min(axis=1))
```
**Output:**
```
Lowest marks per subject:
Maths 78.0
Science 82.0
English 72.0
dtype: float64
Highest marks per subject:
Maths 92.0
Science 95.0
English 85.0
dtype: float64
Lowest marks per student:
Arjun 75.0
Bina 80.0
Chirag 72.0
Divya 85.0
Esha 79.0
dtype: float64
```
> `axis=0` (default) → calculates **down the column** (gives one value per column)
> `axis=1` → calculates **across the row** (gives one value per row)
---
### 2.4.2 Functions `mode()`, `mean()`, `median()` 📐
**Understanding the three "M"s:**
::: grid
::: card 📊 | Mean (Average) | Add all values ÷ count | (90+85+78+92+88)/5 = 86.6
::: card 🎯 | Median (Middle) | Sort and pick middle value | 78, 85, 88, 90, 92 → Middle = 88
::: card 🏆 | Mode (Most Common) | The value that appears most often | If 85 appears 3 times, mode = 85
:::
```python
# mean_median_mode.py
print("MEAN (Average marks per subject):")
print(df.mean())
print("\nMEDIAN (Middle value per subject):")
print(df.median())
print("\nMODE (Most common value per subject):")
print(df.mode())
```
**Output:**
```
MEAN (Average marks per subject):
Maths 86.6
Science 88.75
English 78.2
dtype: float64
MEDIAN (Middle value per subject):
Maths 88.0
Science 89.0
English 79.0
dtype: float64
MODE (Most common value per subject):
Maths Science English
0 78.0 82.0 72.0
```
> [!NOTE]
> **mode() returns a DataFrame, not a Series! 🧠**
> Because there can be MULTIPLE modes (e.g., if both 85 and 90 appear twice, both are modes!). That's why `mode()` returns a DataFrame with all possible mode values, while `mean()` and `median()` return a single Series.
---
### 2.4.3 Functions `count()` and `sum()` 🔢
```python
# count_sum_demo.py
print("COUNT (non-NaN values per subject):")
print(df.count())
print("\nSUM (total marks per subject):")
print(df.sum())
print("\nSUM per student (axis=1):")
print(df.sum(axis=1))
```
**Output:**
```
COUNT (non-NaN values per subject):
Maths 5
Science 4 ← Only 4! Chirag's NaN is not counted
English 5
dtype: int64
SUM (total marks per subject):
Maths 433.0
Science 355.0
English 391.0
dtype: float64
SUM per student (axis=1):
Arjun 253.0
Bina 260.0
Chirag 150.0 ← Only Maths + English (Science is NaN)
Divya 267.0
Esha 249.0
dtype: float64
```
> [!IMPORTANT]
> **Board Exam Tip**
> "What is the difference between count() and sum()?" — **1-mark** question!
> `count()` → returns the **number of non-NaN values** in each column/row.
> `sum()` → returns the **total (addition) of all non-NaN values** in each column/row.
---
### 2.4.4 Functions `quantile()`, `std()`, and `var()` 📉
These three sound scary, but let's explain them simply:
::: grid
::: card 📊 | std() — Standard Deviation | How spread out the marks are | Small std = everyone scored similarly; Large std = big differences
::: card 📐 | var() — Variance | std squared (std²) | Another measure of spread; variance = std × std
::: card 📏 | quantile() — Percentile | What value is at X% of the data | 0.25 = 25th percentile (25% of students scored below this)
:::
```python
# std_var_quantile_demo.py
print("STANDARD DEVIATION (how spread out marks are):")
print(df.std())
print("\nVARIANCE (std squared):")
print(df.var())
print("\n25th Percentile (Q1) of each subject:")
print(df.quantile(0.25))
print("\n50th Percentile (Median) of each subject:")
print(df.quantile(0.50))
print("\n75th Percentile (Q3) of each subject:")
print(df.quantile(0.75))
```
**Output:**
```
STANDARD DEVIATION:
Maths 5.177
Science 5.315
English 4.764
dtype: float64
VARIANCE:
Maths 26.80
Science 28.25
English 22.70
dtype: float64
25th Percentile (Q1):
Maths 85.0
Science 83.5
English 75.5
dtype: float64
50th Percentile (Median):
Maths 88.0
Science 89.0
English 79.0
dtype: float64
75th Percentile (Q3):
Maths 90.0
Science 91.25
English 80.5
dtype: float64
```
**What does Standard Deviation actually mean?**
```
Maths std = 5.177 → marks are close together (78 to 92 range)
If std were 30 → huge variation (some scored 10, some scored 100!)
```
> [!IMPORTANT]
> **Board Exam Tip**
> "What does `df.std()` return?" — **1-mark** question!
> Answer: `std()` returns the **Standard Deviation** of each column — it measures how much the values spread away from the mean. A small standard deviation means values are clustered close to the average; a large one means they are spread out widely.
---
### 2.4.5 The `describe()` Function — The Swiss Army Knife! 🛡️
`describe()` gives you **8 statistics at once** for all numeric columns — it's the most powerful summary function in Pandas!
```python
# describe_demo.py
print("Complete Statistical Summary:")
print(df.describe())
```
**Output:**
```
Maths Science English
count 5.000000 4.000000 5.000000
mean 86.600000 88.750000 78.200000
std 5.177370 5.315073 4.764452
min 78.000000 82.000000 72.000000
25% 85.000000 83.500000 75.500000
50% 88.000000 89.000000 79.000000
75% 90.000000 91.250000 80.500000
max 92.000000 95.000000 85.000000
```
**What each row means:**
| Row in describe() | Meaning |
| :--- | :--- |
| **count** | Number of non-NaN values |
| **mean** | Average of all values |
| **std** | Standard Deviation |
| **min** | Smallest value |
| **25%** | 25th percentile (Q1) |
| **50%** | Median (middle value) |
| **75%** | 75th percentile (Q3) |
| **max** | Largest value |
> [!IMPORTANT]
> **Board Exam Tip**
> "Which single function gives you count, mean, std, min, max of a DataFrame?" — **1-mark** question!
> Answer: **`df.describe()`** — it gives all 8 statistical summaries at once.
---
## 2.5 🔧 Some Other Essential Functions and Functionality
### 2.5.1 Inspection Function `info()` 🔍
`info()` gives you a **structural summary** of the DataFrame — not the data values, but information ABOUT the DataFrame.
```python
# info_demo.py
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya'],
'Maths': [90, 85, 78, 92],
'Science': [88, 95, np.nan, 90],
'Section': ['A', 'A', 'B', 'B']
})
df.info()
```
**Output:**
```
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 4 non-null object
1 Maths 4 non-null int64
2 Science 3 non-null float64 ← Only 3 non-null (Chirag = NaN!)
3 Section 4 non-null object
dtypes: float64(1), int64(1), object(2)
memory usage: 256.0+ bytes
```
**What `info()` tells you:**
- Number of rows and columns
- Column names and their data types
- Count of non-NaN values per column
- Memory usage
> **Difference: `info()` vs `describe()`**
> `info()` → tells you ABOUT the structure (types, NaN counts, memory)
> `describe()` → tells you ABOUT the data (mean, max, std...)
---
### 2.5.2 DataFrame's Top and Bottom Rows — `head()` and `tail()` 📋
```python
# head_tail_demo.py
import pandas as pd
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha', 'Farhan', 'Gita'],
'Marks': [90, 85, 78, 92, 88, 65, 95]
})
print("FIRST 3 rows (head):")
print(df.head(3))
print("\nLAST 3 rows (tail):")
print(df.tail(3))
print("\nDefault head (5 rows):")
print(df.head())
```
**Output:**
```
FIRST 3 rows (head):
Name Marks
0 Arjun 90
1 Bina 85
2 Chirag 78
LAST 3 rows (tail):
Name Marks
4 Esha 88
5 Farhan 65
6 Gita 95
Default head (5 rows):
Name Marks
0 Arjun 90
1 Bina 85
2 Chirag 78
3 Divya 92
4 Esha 88
```
| Function | Returns | Default |
| :--- | :--- | :--- |
| `df.head(n)` | First n rows | 5 |
| `df.tail(n)` | Last n rows | 5 |
---
### 2.5.3 Cumulative Calculation Functions 📈
**Cumulative** means adding up (or finding max/min) as you go — each row shows the running total up to that point.
> **Analogy:** Imagine counting your savings week by week:
> Week 1: ₹500 → Total so far: ₹500
> Week 2: ₹300 → Total so far: ₹800
> Week 3: ₹700 → Total so far: ₹1500
> That's exactly what `cumsum()` does!
```python
# cumulative_demo.py
import pandas as pd
sales = pd.DataFrame({
'Week': ['W1', 'W2', 'W3', 'W4', 'W5'],
'Sales': [500, 300, 700, 200, 600]
})
sales = sales.set_index('Week')
print("Original Sales:")
print(sales)
print("\nCumulative Sum (running total):")
print(sales.cumsum())
print("\nCumulative Max (highest seen so far):")
print(sales.cummax())
print("\nCumulative Min (lowest seen so far):")
print(sales.cummin())
print("\nCumulative Product:")
print(sales.cumprod())
```
**Output:**
```
Original Sales:
Sales
Week
W1 500
W2 300
W3 700
W4 200
W5 600
Cumulative Sum (running total):
Sales
Week
W1 500 ← Just W1: 500
W2 800 ← W1+W2: 500+300=800
W3 1500 ← W1+W2+W3: 800+700=1500
W4 1700 ← +200=1700
W5 2300 ← +600=2300
Cumulative Max (highest seen so far):
Sales
Week
W1 500
W2 500 ← Max(500,300) = 500
W3 700 ← Max(500,700) = 700
W4 700 ← Max(700,200) = 700 (no change)
W5 700 ← Max(700,600) = 700 (no change)
Cumulative Min (lowest seen so far):
Sales
Week
W1 500
W2 300 ← Min(500,300) = 300
W3 300 ← Min(300,700) = 300 (no change)
W4 200 ← Min(300,200) = 200
W5 200 ← Min(200,600) = 200 (no change)
```
**Four Cumulative Functions:**
| Function | What it does | Real use |
| :--- | :--- | :--- |
| `cumsum()` | Running total | Track total sales/savings |
| `cummax()` | Running maximum | Track record high score |
| `cummin()` | Running minimum | Track lowest temperature ever |
| `cumprod()` | Running product | Compound interest calculations |
> [!IMPORTANT]
> **Board Exam Tip**
> "What does cumsum() do?" — **2-mark** question.
> Answer: `cumsum()` returns a DataFrame where each value is the **cumulative sum** — the sum of all values from the beginning up to and including the current row. For example, if values are [10, 20, 30], cumsum gives [10, 30, 60].
---
### 2.5.4 Applying Functions on a Subset of DataFrame 🎯
You can apply a function to specific **rows or columns** using `apply()`:
```python
# apply_demo.py
import pandas as pd
df = pd.DataFrame({
'Maths': [90, 85, 78, 92, 88],
'Science': [88, 95, 70, 90, 82],
'English': [75, 80, 72, 85, 79]
}, index=['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'])
# Apply a function to each COLUMN (axis=0):
print("Max of each subject:")
print(df.apply(max)) # Applies Python's max() to each column
print("\nCustom function — range (max-min) per subject:")
print(df.apply(lambda x: x.max() - x.min()))
# Apply to each ROW (axis=1):
print("\nTotal marks per student:")
print(df.apply(sum, axis=1))
print("\nBest subject score per student:")
print(df.apply(max, axis=1))
```
**Output:**
```
Max of each subject:
Maths 92
Science 95
English 85
dtype: int64
Custom function — range (max-min) per subject:
Maths 14
Science 25
English 13
dtype: int64
Total marks per student:
Arjun 253
Bina 260
Chirag 220
Divya 267
Esha 249
dtype: int64
Best subject score per student:
Arjun 90
Bina 95
Chirag 78
Divya 92
Esha 88
dtype: int64
```
**Applying on a Subset of Columns:**
```python
# Only apply to Maths and Science columns:
subset = df[['Maths', 'Science']]
print("Mean of Maths and Science only:")
print(subset.mean())
```
> [!NOTE]
> **`apply()` is very powerful! 🧠**
> `apply()` lets you run ANY function across rows or columns — whether it's Python's built-in `max`, NumPy functions, or even your own custom `lambda` functions. It's the most flexible statistical tool in Pandas!
---
## 2.6 🧹 Handling Missing Data
**Missing data** (shown as `NaN`) is very common in real-world datasets:
- A student was absent for an exam
- A survey respondent skipped a question
- A sensor failed to record a reading
Pandas provides tools to **detect**, **remove**, and **fill** missing data.
```mermaid
graph TD
MISS["❓ Missing Data (NaN)"]
DET["🔍 Detect\nisnull() / notnull()"]
DROP["❌ Drop\ndropna()"]
FILL["✅ Fill\nfillna()"]
MISS --> DET
MISS --> DROP
MISS --> FILL
style MISS fill:#F44336,color:#fff
style DET fill:#FF9800,color:#fff
style DROP fill:#9E9E9E,color:#fff
style FILL fill:#4CAF50,color:#fff
```
**Our DataFrame with missing values:**
```python
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
'Maths': [90, 85, np.nan, 92, 88],
'Science': [88, np.nan, 70, 90, np.nan],
'English': [75, 80, 72, 85, 79]
})
print(df)
```
**Output:**
```
Name Maths Science English
0 Arjun 90.0 88.0 75
1 Bina 85.0 NaN 80
2 Chirag NaN 70.0 72
3 Divya 92.0 90.0 85
4 Esha 88.0 NaN 79
```
---
### 2.6.1 Detecting / Filtering Missing Data 🔍
```python
# detect_missing.py
# isnull() — returns True where data is MISSING
print("Where is data missing?")
print(df.isnull())
# notnull() — returns True where data IS PRESENT
print("\nWhere is data present?")
print(df.notnull())
# Count missing values per column:
print("\nCount of missing values per column:")
print(df.isnull().sum())
# Which rows have ANY missing value?
print("\nRows with at least one missing value:")
print(df[df.isnull().any(axis=1)])
```
**Output:**
```
Where is data missing?
Name Maths Science English
0 False False False False
1 False False True False
2 False True False False
3 False False False False
4 False False True False
Count of missing values per column:
Name 0
Maths 1
Science 2
English 0
dtype: int64
Rows with at least one missing value:
Name Maths Science English
1 Bina 85.0 NaN 80
2 Chirag NaN 70.0 72
4 Esha 88.0 NaN 79
```
---
### 2.6.2 Handling Missing Data — Dropping Missing Values ❌
`dropna()` removes rows (or columns) that contain NaN values:
```python
# dropna_demo.py
print("Original DataFrame:")
print(df[['Name', 'Maths', 'Science']])
print("\nAfter dropna() — removes rows with ANY NaN:")
print(df.dropna()[['Name', 'Maths', 'Science']])
print("\nAfter dropna(how='all') — removes rows where ALL values are NaN:")
print(df.dropna(how='all')[['Name', 'Maths', 'Science']])
# Drop columns with NaN:
print("\nDrop COLUMNS with NaN (axis=1):")
print(df.dropna(axis=1))
# Drop rows where specific column has NaN:
print("\nDrop rows where Maths is NaN:")
print(df.dropna(subset=['Maths']))
```
**Output:**
```
Original DataFrame:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 NaN
2 Chirag NaN 70.0
3 Divya 92.0 90.0
4 Esha 88.0 NaN
After dropna() — removes rows with ANY NaN:
Name Maths Science
0 Arjun 90.0 88.0
3 Divya 92.0 90.0
After dropna(how='all'):
(same as original — no row where ALL are NaN)
Drop COLUMNS with NaN (axis=1):
Name English
0 Arjun 75
1 Bina 80
2 Chirag 72
3 Divya 85
4 Esha 79
Drop rows where Maths is NaN:
Name Maths Science English
0 Arjun 90.0 88.0 75
1 Bina 85.0 NaN 80
3 Divya 92.0 90.0 85
4 Esha 88.0 NaN 79
```
**`dropna()` Parameters:**
| Parameter | Default | Meaning |
| :--- | :--- | :--- |
| `axis` | 0 | 0=drop rows, 1=drop columns |
| `how` | 'any' | 'any'=drop if ANY NaN; 'all'=drop only if ALL are NaN |
| `subset` | None | Only check specific columns for NaN |
| `inplace` | False | True = modify original; False = return new df |
---
### 2.6.3 Handling Missing Data — Filling Missing Values ✅
Instead of deleting rows with NaN, you can **replace NaN with a meaningful value** using `fillna()`:
```python
# fillna_demo.py
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
'Maths': [90, 85, np.nan, 92, 88],
'Science': [88, np.nan, 70, 90, np.nan],
})
print("Original:")
print(df)
# Fill ALL NaN with a fixed value:
print("\nFill NaN with 0:")
print(df.fillna(0))
# Fill NaN with the COLUMN MEAN (most common approach):
print("\nFill NaN with column mean:")
print(df.fillna(df.mean()))
# Fill NaN forward (use previous row's value):
print("\nForward fill (ffill):")
print(df.fillna(method='ffill'))
# Fill NaN backward (use next row's value):
print("\nBackward fill (bfill):")
print(df.fillna(method='bfill'))
# Fill specific columns with specific values:
print("\nFill Maths NaN with 0, Science NaN with 60:")
print(df.fillna({'Maths': 0, 'Science': 60}))
```
**Output:**
```
Original:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 NaN
2 Chirag NaN 70.0
3 Divya 92.0 90.0
4 Esha 88.0 NaN
Fill NaN with 0:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 0.0 ← NaN replaced with 0
2 Chirag 0.0 70.0 ← NaN replaced with 0
3 Divya 92.0 90.0
4 Esha 88.0 0.0 ← NaN replaced with 0
Fill NaN with column mean:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 82.67 ← Mean of (88,70,90) = 82.67
2 Chirag 88.75 70.0 ← Mean of (90,85,92,88) = 88.75
3 Divya 92.0 90.0
4 Esha 88.0 82.67 ← Mean again
Forward fill:
Name Maths Science
0 Arjun 90.0 88.0
1 Bina 85.0 88.0 ← Takes value from row above (88)
2 Chirag 85.0 70.0 ← Takes Maths from row above (85)
3 Divya 92.0 90.0
4 Esha 88.0 90.0 ← Takes value from row above (90)
```
**`fillna()` Strategies — When to use which:**
| Strategy | Code | When to use |
| :--- | :--- | :--- |
| Fixed value | `fillna(0)` | Zero means "not attempted" |
| Column mean | `fillna(df.mean())` | Best estimate for numeric data |
| Column median | `fillna(df.median())` | When data has outliers |
| Forward fill | `fillna(method='ffill')` | Time-series data (carry last known value) |
| Backward fill | `fillna(method='bfill')` | Use next known value |
> [!IMPORTANT]
> **Board Exam Tip**
> "Write code to fill all NaN values in a DataFrame with the column's mean." — **2-mark** question!
> Answer: `df.fillna(df.mean())`
>
> "Write code to drop all rows that have any missing value." — **2-mark** question!
> Answer: `df.dropna()` or `df.dropna(how='any')`
---
## 2.7 📦 Function `groupby()` — Group and Analyse!
`groupby()` is one of the most powerful functions in Pandas. It lets you:
1. **Split** data into groups based on a column
2. **Apply** a function (sum, mean, count, etc.) to each group
3. **Combine** results into a new DataFrame
> **Super Simple Analogy:** Imagine dividing your class into **Section A and Section B** and then finding the average marks of each section separately. That's exactly what `groupby('Section').mean()` does!
```mermaid
graph LR
ALL["📋 All Students\n(Mixed Sections)"]
GRP["groupby('Section')"]
A["📊 Section A\nArjun, Bina, Esha"]
B["📊 Section B\nChirag, Divya"]
AGG["apply mean()"]
RES["📊 Result\nA: avg marks\nB: avg marks"]
ALL --> GRP
GRP --> A
GRP --> B
A --> AGG
B --> AGG
AGG --> RES
style GRP fill:#FF9800,color:#fff
style RES fill:#4CAF50,color:#fff
```
**Basic `groupby()` example:**
```python
# groupby_basic.py
import pandas as pd
import numpy as np
df = pd.DataFrame({
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha', 'Farhan'],
'Section': ['A', 'A', 'B', 'B', 'A', 'B'],
'Maths': [90, 85, 78, 92, 88, 74],
'English': [75, 80, 72, 85, 79, 68]
})
print("Original DataFrame:")
print(df)
# Group by Section and find MEAN of each group:
print("\nSection-wise AVERAGE:")
print(df.groupby('Section').mean())
# Group by Section and find MAX of each group:
print("\nSection-wise MAXIMUM:")
print(df.groupby('Section').max())
# Group by Section and find COUNT:
print("\nSection-wise COUNT (number of students):")
print(df.groupby('Section').count())
```
**Output:**
```
Original DataFrame:
Name Section Maths English
0 Arjun A 90 75
1 Bina A 85 80
2 Chirag B 78 72
3 Divya B 92 85
4 Esha A 88 79
5 Farhan B 74 68
Section-wise AVERAGE:
Maths English
Section
A 87.67 78.00
B 81.33 75.00
Section-wise MAXIMUM:
Name Maths English
Section
A Esha 90 80
B Farhan 92 85
Section-wise COUNT:
Name Maths English
Section
A 3 3 3
B 3 3 3
```
**More `groupby()` Examples:**
```python
# groupby_advanced.py
# Sum of marks per section:
print("Total marks per section:")
print(df.groupby('Section')[['Maths', 'English']].sum())
# Groupby and get specific statistic:
print("\nMinimum Maths marks per section:")
print(df.groupby('Section')['Maths'].min())
# Get complete statistics per group:
print("\nFull statistics per section:")
print(df.groupby('Section').describe())
```
**Output:**
```
Total marks per section:
Maths English
Section
A 263 234
B 244 225
Minimum Maths marks per section:
Section
A 85
B 74
Name: Maths, dtype: int64
```
**Groupby with multiple columns:**
```python
# Add City column to demo multiple groupby:
df['City'] = ['Delhi', 'Mumbai', 'Delhi', 'Pune', 'Mumbai', 'Delhi']
print("Section AND City-wise average:")
print(df.groupby(['Section', 'City'])['Maths'].mean())
```
> [!IMPORTANT]
> **Board Exam Tip**
> "Write code to find the average marks of each section using groupby()." — **3-mark** question, extremely common!
> Answer:
> ```python
> result = df.groupby('Section').mean()
> print(result)
> ```
---
## ⚠️ Common Errors and Misconceptions
| Mistake | What Goes Wrong | Correct Approach |
| :--- | :--- | :--- |
| ❌ Using `and` instead of `&` in conditions | TypeError | ✅ `df[(df['M']>80) & (df['S']=='A')]` |
| ❌ Forgetting NaN skipping behaviour | Think NaN = 0 | ✅ NaN is SKIPPED in mean(), sum(), etc. |
| ❌ `iteritems()` in new Pandas | DeprecationWarning/Error | ✅ Use `items()` in Pandas 1.5+ |
| ❌ `dropna()` not changing original | df unchanged | ✅ Use `inplace=True` or `df = df.dropna()` |
| ❌ Confusing `count()` and `size()` | Wrong answer | ✅ `count()` skips NaN; `size` includes NaN |
| ❌ mode() returning a Series | Confused by DataFrame output | ✅ mode() returns a DataFrame (multiple modes possible!) |
---
## 🔑 Quick Revision — Exam Ready!
**Iteration:**
- `iterrows()` → row by row → `(index, Series)`
- `items()` / `iteritems()` → column by column → `(col_name, Series)`
**Statistical Functions:**
| Function | Returns |
| :--- | :--- |
| `min()` / `max()` | Smallest/largest per column |
| `mean()` | Average per column |
| `median()` | Middle value per column |
| `mode()` | Most common value (DataFrame) |
| `sum()` | Total per column |
| `count()` | Non-NaN count per column |
| `std()` | Standard Deviation |
| `var()` | Variance (std²) |
| `quantile(q)` | Value at q-th percentile |
| `describe()` | All 8 stats at once |
**Missing Data:**
- `isnull()` → True where NaN | `notnull()` → True where not NaN
- `dropna()` → remove rows/columns with NaN
- `fillna(value)` → replace NaN with value
**groupby():**
- `df.groupby('col').mean()` → group by col, find average
- `df.groupby('col').sum()` → group by col, find total
- `df.groupby('col').count()` → count per group
**Cumulative:**
- `cumsum()` → running total
- `cummax()` → running maximum
- `cummin()` → running minimum
---
## 🎯 Sample Board Exam Questions
### Q1: Very Short Answer [1 mark each]
a) Which function gives all 8 statistics (count, mean, std, min, 25%, 50%, 75%, max) in one call?
**→ `df.describe()`**
b) What does `isnull()` return?
**→ A DataFrame of True/False — True where values are missing (NaN), False where present.**
c) What does `cumsum()` do?
**→ Returns the cumulative running sum — each value is the sum of all elements up to that position.**
d) Which function removes rows containing NaN?
**→ `dropna()`**
e) What is the difference between `count()` and `sum()`?
**→ `count()` returns the number of non-NaN values; `sum()` returns the total (addition) of all non-NaN values.**
---
### Q2: Short Answer [2 marks]
**Q: Differentiate between `dropna()` and `fillna()`.**
`dropna()` **removes** rows (or columns) that contain NaN values — the affected rows are deleted from the result.
`fillna(value)` **replaces** NaN values with a specified value (e.g., 0, the column mean, or 'Unknown') — no rows are removed.
```python
df.dropna() # Removes Chirag's row (NaN in Science)
df.fillna(0) # Keeps Chirag but replaces his NaN with 0
```
---
### Q3: Program Writing [3 marks]
**Q: Create a DataFrame of 4 students with Maths and Science marks (one NaN). Find section-wise average using groupby(). Also fill NaN with the column mean.**
```python
import pandas as pd
import numpy as np
data = {
'Name': ['Arjun', 'Bina', 'Chirag', 'Divya'],
'Section': ['A', 'A', 'B', 'B'],
'Maths': [90, 85, np.nan, 92],
'Science': [88, 95, 70, 90]
}
df = pd.DataFrame(data)
print("Original:")
print(df)
# Fill NaN with mean:
df_filled = df.fillna(df.mean())
print("\nAfter filling NaN with mean:")
print(df_filled)
# Section-wise average:
print("\nSection-wise average:")
print(df.groupby('Section').mean())
```
---
### Q4: Output Based [3 marks]
**Q: What is the output?**
```python
import pandas as pd
import numpy as np
s = pd.Series([10, 20, 30, 40, 50])
print(s.cumsum())
print(s.std())
```
**Output:**
```
0 10
1 30
2 60
3 100
4 150
dtype: int64
15.811388300841896
```
*(cumsum: running totals 10, 10+20=30, 30+30=60, 60+40=100, 100+50=150)*
*(std: standard deviation of [10,20,30,40,50])*
---
## ✏️ Practice Problems
1. Create a DataFrame of 5 products with their prices and quantities. Use `describe()` to get the statistical summary.
2. Write code to iterate over each row of a student DataFrame and print "Pass" if marks ≥ 40, else "Fail".
3. Create two DataFrames with matching and non-matching indexes. Add them using `.add(fill_value=0)` and explain the result.
4. From a DataFrame with NaN values, write code to: (a) count NaN per column, (b) drop rows with NaN, (c) fill NaN with the column median.
5. Create a DataFrame with students from different cities. Use `groupby('City')` to find the maximum marks per city.
6. Find the cumulative sum of a Series [5, 10, 15, 20] and predict the output before running.
7. Write code to apply a custom function that calculates (max - min) for each column of a DataFrame.
8. What is the difference between `iterrows()` and `items()`? Write one example of each.
9. A DataFrame has columns 'Month' and 'Sales'. Use `cumsum()` to show running total sales and `cummax()` to track the best month so far.
10. Create a DataFrame, then: (a) find mode of each column, (b) find the 75th percentile (Q3) using quantile(), (c) find standard deviation.
Back to List
Calculating...
UNIT 1 : CH 2
Dec 14, 2025
📊 Data Handling Using Pandas – II
🎯 CBSE BOARD EXAM 2027
Complete Preparation Resources
Prepare smarter with the right combination of CSIP12's FREE digital resources + optional printed study books.
INFORMATICS PRACTICES — 065
2026–27
LEARN — Core Theory
Sumita Arora — Informatics Practices with Python
Primary2026–27 Session Edition • Dhanpat Rai & Co.
PRACTISE — Pandas, SQL & Case Studies
Together With CBSE Class 12 Informatics Practices Question Bank
PrimaryChapterwise & Topicwise | For 2027 Exam
Additional Option
Arihant All in One Informatics Practices Class 12
View on Amazon
You don't need every book. Choose resources that complement your preparation. CSIP12 provides notes, programs, quizzes, solved questions and previous-year papers for FREE.
Learning Support
Need Help With This Chapter?
Save key topics for exam revision, ask questions to teachers, or submit content corrections.
Verified Doubts & Teacher Answers
Loading resolved questions for this note...