📊 Data Handling Using Pandas – II

In Pandas – I we learnt to create and access data. Pandas – II is about analysing, cleaning and summarising it: iterating over a DataFrame, binary operations, descriptive statistics, other essential functions, handling missing data, and grouping data. In report-card terms, it is the step from reading the marks to calculating toppers and averages and finding who was absent.

2.1 🌟 Introduction

In this chapter, we move from just storing data to actually working with it:

graph LR
    P1["📘 Pandas I\nCreate & Store Data\nSeries & DataFrames"]
    P2["📗 Pandas II\nAnalyse & Clean Data\nStats, Iteration, Groups"]

    P1 -->|"Next Level"| P2

    style P1 fill:#2196F3,color:#fff
    style P2 fill:#4CAF50,color:#fff

Our Master DataFrame — used throughout this chapter:

# master_setup.py
import pandas as pd
import numpy as np

data = {
    'Name':    ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
    'Maths':   [90, 85, 78, 92, 88],
    'Science': [88, 95, np.nan, 90, 82],   # Chirag was absent!
    'English': [75, 80, 72, 85, 79],
    'Section': ['A', 'A', 'B', 'B', 'A']
}

df = pd.DataFrame(data)
print(df)

Output:

     Name  Maths  Science  English Section
0   Arjun     90     88.0       75       A
1    Bina     85     95.0       80       A
2  Chirag     78      NaN       72       B
3   Divya     92     90.0       85       B
4    Esha     88     82.0       79       A

NaN (Not a Number) in Chirag's Science row — he was absent for that exam!

2.2 🔄 Iterating Over a DataFrame

Iteration means visiting each part of the DataFrame one by one — like reading a register row by row or column by column.

graph LR
    IT["Iterating\nover DataFrame"]
    IR["iterrows()\nRow by Row\n(Horizontal)"]
    IC["iteritems()\n/ items()\nColumn by Column\n(Vertical)"]

    IT --> IR
    IT --> IC

    style IR fill:#FF9800,color:#fff
    style IC fill:#9C27B0,color:#fff

2.2.1 Using iterrows() — Row by Row 🔄

iterrows() goes through the DataFrame one row at a time. Each time, it gives you:

• The row index (number or label)
• A Series containing all values in that row

Analogy: Imagine a teacher reading the class register — going student by student from top to bottom. That's iterrows()!

# iterrows_demo.py
import pandas as pd
import numpy as np

data = {
    'Name':  ['Arjun', 'Bina', 'Chirag', 'Divya'],
    'Maths': [90, 85, 78, 92],
    'English': [75, 80, 72, 85]
}
df = pd.DataFrame(data)

print("Reading students one by one:")
for index, row in df.iterrows():
    print(f"Row {index}: {row['Name']} → Maths={row['Maths']}, English={row['English']}")

Output:

Reading students one by one:
Row 0: Arjun → Maths=90, English=75
Row 1: Bina → Maths=85, English=80
Row 2: Chirag → Maths=78, English=72
Row 3: Divya → Maths=92, English=85

Practical Example — Find students who passed (Maths ≥ 80):

# find_passers.py
print("Students who scored ≥ 80 in Maths:")
for index, row in df.iterrows():
    if row['Maths'] >= 80:
        print(f"  ✅ {row['Name']} — {row['Maths']}")

Output:

Students who scored ≥ 80 in Maths:
  ✅ Arjun — 90
  ✅ Bina — 85
  ✅ Divya — 92

2.2.2 Using iteritems() — Column by Column 📊

iteritems() goes through the DataFrame one column at a time. Each time, it gives you:

• The column name (header)
• A Series containing all values in that column

Analogy: Instead of reading row by row (student by student), now you're reading column by column (subject by subject) — first all Maths marks, then all English marks...

# iteritems_demo.py
import pandas as pd

data = {
    'Maths':   [90, 85, 78, 92],
    'Science': [88, 95, 70, 90],
    'English': [75, 80, 72, 85]
}
df = pd.DataFrame(data)

print("Reading subjects one by one:")
for col_name, col_data in df.items():     # Note: use .items() in newer Pandas
    print(f"\nSubject: {col_name}")
    print(f"  All marks: {col_data.values}")
    print(f"  Class Average: {col_data.mean():.2f}")

Output:

Reading subjects one by one:

Subject: Maths
  All marks: [90 85 78 92]
  Class Average: 86.25

Subject: Science
  All marks: [88 95 70 90]
  Class Average: 85.75

Subject: English
  All marks: [75 80 72 85]
  Class Average: 78.00

Comparison Table:

2.3 ➕ Binary Operations in a DataFrame

Binary operations are mathematical operations (+, -, *, /) performed between two DataFrames (or a DataFrame and a number/Series).

Analogy: Imagine you have Test 1 marks in one sheet and Test 2 marks in another sheet. Adding them gives you Total marks. That's a binary operation between two DataFrames!

graph LR
    D1["📊 DataFrame 1\nTest 1 Marks"]
    D2["📊 DataFrame 2\nTest 2 Marks"]
    OP["➕ Binary Operation\n(Addition)"]
    RES["📊 Result\nTotal Marks"]

    D1 --> OP
    D2 --> OP
    OP --> RES

    style OP fill:#FF9800,color:#fff
    style RES fill:#4CAF50,color:#fff

Basic Binary Operations:

# binary_ops_demo.py
import pandas as pd

test1 = pd.DataFrame({
    'Math': [45, 38, 42],
    'Science': [40, 45, 35]
}, index=['Arjun', 'Bina', 'Chirag'])

test2 = pd.DataFrame({
    'Math': [48, 40, 38],
    'Science': [42, 48, 40]
}, index=['Arjun', 'Bina', 'Chirag'])

print("Test 1:\n", test1)
print("\nTest 2:\n", test2)

print("\nTotal (test1 + test2):")
print(test1 + test2)

print("\nDifference (test2 - test1):")
print(test2 - test1)

Output:

Test 1:
        Math  Science
Arjun    45       40
Bina     38       45
Chirag   42       35

Test 2:
        Math  Science
Arjun    48       42
Bina     40       48
Chirag   38       40

Total (test1 + test2):
        Math  Science
Arjun    93       82
Bina     78       93
Chirag   80       75

Difference (test2 - test1):
        Math  Science
Arjun     3        2
Bina      2        3
Chirag   -4        5

What happens when indexes DON'T match? — NaN appears! ⚠️

# mismatched_binary.py
import pandas as pd

df1 = pd.DataFrame({'A': [10, 20], 'B': [30, 40]}, index=[0, 1])
df2 = pd.DataFrame({'A': [5, 5],  'B': [5, 5]},  index=[1, 2])

print("DF1:\n", df1)
print("\nDF2:\n", df2)
print("\nDF1 + DF2:")
print(df1 + df2)

Output:

DF1:
    A   B
0  10  30
1  20  40

DF2:
   A  B
1  5  5
2  5  5

DF1 + DF2:
      A     B
0   NaN   NaN    ← index 0 in DF1, but NOT in DF2 → NaN
1  25.0  45.0    ← index 1 in BOTH → added normally!
2   NaN   NaN    ← index 2 in DF2, but NOT in DF1 → NaN

The fill_value trick — replace NaN with 0 during operation:

print(df1.add(df2, fill_value=0))

Output:

      A     B
0  10.0  30.0    ← 10 + 0 (fill_value fills missing df2 values with 0)
1  25.0  45.0
2   5.0   5.0    ← 0 + 5 (fill_value fills missing df1 values with 0)

All Binary Operation Functions:

2.4 📊 Descriptive Statistics with Pandas

Descriptive Statistics = functions that summarise your data — finding the average, highest, lowest, middle value, etc. Pandas makes all of this incredibly easy!

Analogy: After a class test, a teacher wants to know: Who scored highest? What's the class average? What's the middle score? All of this is "descriptive statistics" — and Pandas answers all these questions in one line each!

Using this DataFrame for all examples:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Maths':   [90, 85, 78, 92, 88],
    'Science': [88, 95, np.nan, 90, 82],
    'English': [75, 80, 72, 85, 79]
}, index=['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'])

Remember: All statistical functions automatically skip NaN values unless told otherwise!

2.4.1 Functions min() and max() 📉📈

# min_max_demo.py

# Column-wise (default) — min/max of each subject
print("Lowest marks per subject:")
print(df.min())

print("\nHighest marks per subject:")
print(df.max())

# Row-wise (axis=1) — min/max per student
print("\nLowest marks per student:")
print(df.min(axis=1))

Output:

Lowest marks per subject:
Maths      78.0
Science    82.0
English    72.0
dtype: float64

Highest marks per subject:
Maths      92.0
Science    95.0
English    85.0
dtype: float64

Lowest marks per student:
Arjun     75.0
Bina      80.0
Chirag    72.0
Divya     85.0
Esha      79.0
dtype: float64

axis=0 (default) → calculates down the column (gives one value per column) axis=1 → calculates across the row (gives one value per row)

2.4.2 Functions mode(), mean(), median() 📐

Understanding the three "M"s:

# mean_median_mode.py

print("MEAN (Average marks per subject):")
print(df.mean())

print("\nMEDIAN (Middle value per subject):")
print(df.median())

print("\nMODE (Most common value per subject):")
print(df.mode())

Output:

MEAN (Average marks per subject):
Maths      86.6
Science    88.75
English    78.2
dtype: float64

MEDIAN (Middle value per subject):
Maths      88.0
Science    89.0
English    79.0
dtype: float64

MODE (Most common value per subject):
   Maths  Science  English
0   78.0     82.0     72.0

2.4.3 Functions count() and sum() 🔢

# count_sum_demo.py

print("COUNT (non-NaN values per subject):")
print(df.count())

print("\nSUM (total marks per subject):")
print(df.sum())

print("\nSUM per student (axis=1):")
print(df.sum(axis=1))

Output:

COUNT (non-NaN values per subject):
Maths      5
Science    4    ← Only 4! Chirag's NaN is not counted
English    5
dtype: int64

SUM (total marks per subject):
Maths      433.0
Science    355.0
English    391.0
dtype: float64

SUM per student (axis=1):
Arjun     253.0
Bina      260.0
Chirag    150.0    ← Only Maths + English (Science is NaN)
Divya     267.0
Esha      249.0
dtype: float64

2.4.4 Functions quantile(), std(), and var() 📉

These three sound scary, but let's explain them simply:

# std_var_quantile_demo.py

print("STANDARD DEVIATION (how spread out marks are):")
print(df.std())

print("\nVARIANCE (std squared):")
print(df.var())

print("\n25th Percentile (Q1) of each subject:")
print(df.quantile(0.25))

print("\n50th Percentile (Median) of each subject:")
print(df.quantile(0.50))

print("\n75th Percentile (Q3) of each subject:")
print(df.quantile(0.75))

Output:

STANDARD DEVIATION:
Maths      5.177
Science    5.315
English    4.764
dtype: float64

VARIANCE:
Maths      26.80
Science    28.25
English    22.70
dtype: float64

25th Percentile (Q1):
Maths      85.0
Science    83.5
English    75.5
dtype: float64

50th Percentile (Median):
Maths      88.0
Science    89.0
English    79.0
dtype: float64

75th Percentile (Q3):
Maths      90.0
Science    91.25
English    80.5
dtype: float64

What does Standard Deviation actually mean?

Maths std = 5.177  → marks are close together (78 to 92 range)
If std were 30     → huge variation (some scored 10, some scored 100!)

2.4.5 The describe() Function — The Swiss Army Knife! 🛡️

describe() gives you 8 statistics at once for all numeric columns — a quick way to summarise a DataFrame.

# describe_demo.py

print("Complete Statistical Summary:")
print(df.describe())

Output:

           Maths    Science    English
count   5.000000   4.000000   5.000000
mean   86.600000  88.750000  78.200000
std     5.177370   5.315073   4.764452
min    78.000000  82.000000  72.000000
25%    85.000000  83.500000  75.500000
50%    88.000000  89.000000  79.000000
75%    90.000000  91.250000  80.500000
max    92.000000  95.000000  85.000000

What each row means:

2.5 🔧 Some Other Essential Functions and Functionality

2.5.1 Inspection Function info() 🔍

info() gives you a structural summary of the DataFrame — not the data values, but information ABOUT the DataFrame.

# info_demo.py
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Name':    ['Arjun', 'Bina', 'Chirag', 'Divya'],
    'Maths':   [90, 85, 78, 92],
    'Science': [88, 95, np.nan, 90],
    'Section': ['A', 'A', 'B', 'B']
})

df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 4 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   Name     4 non-null      object
 1   Maths    4 non-null      int64
 2   Science  3 non-null      float64   ← Only 3 non-null (Chirag = NaN!)
 3   Section  4 non-null      object
dtypes: float64(1), int64(1), object(2)
memory usage: 256.0+ bytes

What info() tells you:

• Number of rows and columns
• Column names and their data types
• Count of non-NaN values per column
• Memory usage

Difference: info() vs describe() info() → tells you ABOUT the structure (types, NaN counts, memory) describe() → tells you ABOUT the data (mean, max, std...)

2.5.2 DataFrame's Top and Bottom Rows — head() and tail() 📋

# head_tail_demo.py
import pandas as pd

df = pd.DataFrame({
    'Name': ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha', 'Farhan', 'Gita'],
    'Marks': [90, 85, 78, 92, 88, 65, 95]
})

print("FIRST 3 rows (head):")
print(df.head(3))

print("\nLAST 3 rows (tail):")
print(df.tail(3))

print("\nDefault head (5 rows):")
print(df.head())

Output:

FIRST 3 rows (head):
     Name  Marks
0   Arjun     90
1    Bina     85
2  Chirag     78

LAST 3 rows (tail):
     Name  Marks
4    Esha     88
5  Farhan     65
6    Gita     95

Default head (5 rows):
     Name  Marks
0   Arjun     90
1    Bina     85
2  Chirag     78
3   Divya     92
4    Esha     88

2.5.3 Cumulative Calculation Functions 📈

Cumulative means adding up (or finding max/min) as you go — each row shows the running total up to that point.

Analogy: Imagine counting your savings week by week: Week 1: ₹500 → Total so far: ₹500 Week 2: ₹300 → Total so far: ₹800 Week 3: ₹700 → Total so far: ₹1500 That's exactly what cumsum() does!

# cumulative_demo.py
import pandas as pd

sales = pd.DataFrame({
    'Week':  ['W1', 'W2', 'W3', 'W4', 'W5'],
    'Sales': [500, 300, 700, 200, 600]
})
sales = sales.set_index('Week')

print("Original Sales:")
print(sales)

print("\nCumulative Sum (running total):")
print(sales.cumsum())

print("\nCumulative Max (highest seen so far):")
print(sales.cummax())

print("\nCumulative Min (lowest seen so far):")
print(sales.cummin())

print("\nCumulative Product:")
print(sales.cumprod())

Output:

Original Sales:
      Sales
Week
W1      500
W2      300
W3      700
W4      200
W5      600

Cumulative Sum (running total):
       Sales
Week
W1      500     ← Just W1: 500
W2      800     ← W1+W2: 500+300=800
W3     1500     ← W1+W2+W3: 800+700=1500
W4     1700     ← +200=1700
W5     2300     ← +600=2300

Cumulative Max (highest seen so far):
       Sales
Week
W1      500
W2      500     ← Max(500,300) = 500
W3      700     ← Max(500,700) = 700
W4      700     ← Max(700,200) = 700 (no change)
W5      700     ← Max(700,600) = 700 (no change)

Cumulative Min (lowest seen so far):
       Sales
Week
W1      500
W2      300     ← Min(500,300) = 300
W3      300     ← Min(300,700) = 300 (no change)
W4      200     ← Min(300,200) = 200
W5      200     ← Min(200,600) = 200 (no change)

Four Cumulative Functions:

2.5.4 Applying Functions on a Subset of DataFrame 🎯

You can apply a function to specific rows or columns using apply():

# apply_demo.py
import pandas as pd

df = pd.DataFrame({
    'Maths':   [90, 85, 78, 92, 88],
    'Science': [88, 95, 70, 90, 82],
    'English': [75, 80, 72, 85, 79]
}, index=['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'])

# Apply a function to each COLUMN (axis=0):
print("Max of each subject:")
print(df.apply(max))        # Applies Python's max() to each column

print("\nCustom function — range (max-min) per subject:")
print(df.apply(lambda x: x.max() - x.min()))

# Apply to each ROW (axis=1):
print("\nTotal marks per student:")
print(df.apply(sum, axis=1))

print("\nBest subject score per student:")
print(df.apply(max, axis=1))

Output:

Max of each subject:
Maths      92
Science    95
English    85
dtype: int64

Custom function — range (max-min) per subject:
Maths      14
Science    25
English    13
dtype: int64

Total marks per student:
Arjun     253
Bina      260
Chirag    220
Divya     267
Esha      249
dtype: int64

Best subject score per student:
Arjun     90
Bina      95
Chirag    78
Divya     92
Esha      88
dtype: int64

Applying on a Subset of Columns:

# Only apply to Maths and Science columns:
subset = df[['Maths', 'Science']]
print("Mean of Maths and Science only:")
print(subset.mean())

2.6 🧹 Handling Missing Data

Missing data (shown as NaN) is very common in real-world datasets:

• A student was absent for an exam
• A survey respondent skipped a question
• A sensor failed to record a reading

Pandas provides tools to detect, remove, and fill missing data.

graph TD
    MISS["❓ Missing Data (NaN)"]
    DET["🔍 Detect\nisnull() / notnull()"]
    DROP["❌ Drop\ndropna()"]
    FILL["✅ Fill\nfillna()"]

    MISS --> DET
    MISS --> DROP
    MISS --> FILL

    style MISS fill:#F44336,color:#fff
    style DET fill:#FF9800,color:#fff
    style DROP fill:#9E9E9E,color:#fff
    style FILL fill:#4CAF50,color:#fff

Our DataFrame with missing values:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Name':    ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
    'Maths':   [90, 85, np.nan, 92, 88],
    'Science': [88, np.nan, 70, 90, np.nan],
    'English': [75, 80, 72, 85, 79]
})
print(df)

Output:

     Name  Maths  Science  English
0   Arjun   90.0     88.0       75
1    Bina   85.0      NaN       80
2  Chirag    NaN     70.0       72
3   Divya   92.0     90.0       85
4    Esha   88.0      NaN       79

2.6.1 Detecting / Filtering Missing Data 🔍

# detect_missing.py

# isnull() — returns True where data is MISSING
print("Where is data missing?")
print(df.isnull())

# notnull() — returns True where data IS PRESENT
print("\nWhere is data present?")
print(df.notnull())

# Count missing values per column:
print("\nCount of missing values per column:")
print(df.isnull().sum())

# Which rows have ANY missing value?
print("\nRows with at least one missing value:")
print(df[df.isnull().any(axis=1)])

Output:

Where is data missing?
    Name  Maths  Science  English
0  False  False    False    False
1  False  False     True    False
2  False   True    False    False
3  False  False    False    False
4  False  False     True    False

Count of missing values per column:
Name       0
Maths      1
Science    2
English    0
dtype: int64

Rows with at least one missing value:
     Name  Maths  Science  English
1    Bina   85.0      NaN       80
2  Chirag    NaN     70.0       72
4    Esha   88.0      NaN       79

2.6.2 Handling Missing Data — Dropping Missing Values ❌

dropna() removes rows (or columns) that contain NaN values:

# dropna_demo.py

print("Original DataFrame:")
print(df[['Name', 'Maths', 'Science']])

print("\nAfter dropna() — removes rows with ANY NaN:")
print(df.dropna()[['Name', 'Maths', 'Science']])

print("\nAfter dropna(how='all') — removes rows where ALL values are NaN:")
print(df.dropna(how='all')[['Name', 'Maths', 'Science']])

# Drop columns with NaN:
print("\nDrop COLUMNS with NaN (axis=1):")
print(df.dropna(axis=1))

# Drop rows where specific column has NaN:
print("\nDrop rows where Maths is NaN:")
print(df.dropna(subset=['Maths']))

Output:

Original DataFrame:
     Name  Maths  Science
0   Arjun   90.0     88.0
1    Bina   85.0      NaN
2  Chirag    NaN     70.0
3   Divya   92.0     90.0
4    Esha   88.0      NaN

After dropna() — removes rows with ANY NaN:
    Name  Maths  Science
0  Arjun   90.0     88.0
3  Divya   92.0     90.0

After dropna(how='all'):
(same as original — no row where ALL are NaN)

Drop COLUMNS with NaN (axis=1):
     Name  English
0   Arjun       75
1    Bina       80
2  Chirag       72
3   Divya       85
4    Esha       79

Drop rows where Maths is NaN:
    Name  Maths  Science  English
0  Arjun   90.0     88.0       75
1   Bina   85.0      NaN       80
3  Divya   92.0     90.0       85
4   Esha   88.0      NaN       79

dropna() Parameters:

2.6.3 Handling Missing Data — Filling Missing Values ✅

Instead of deleting rows with NaN, you can replace NaN with a meaningful value using fillna():

# fillna_demo.py
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Name':    ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha'],
    'Maths':   [90, 85, np.nan, 92, 88],
    'Science': [88, np.nan, 70, 90, np.nan],
})

print("Original:")
print(df)

# Fill ALL NaN with a fixed value:
print("\nFill NaN with 0:")
print(df.fillna(0))

# Fill NaN with the COLUMN MEAN (most common approach):
print("\nFill NaN with column mean:")
print(df.fillna(df.mean()))

# Fill NaN forward (use previous row's value):
print("\nForward fill (ffill):")
print(df.fillna(method='ffill'))

# Fill NaN backward (use next row's value):
print("\nBackward fill (bfill):")
print(df.fillna(method='bfill'))

# Fill specific columns with specific values:
print("\nFill Maths NaN with 0, Science NaN with 60:")
print(df.fillna({'Maths': 0, 'Science': 60}))

Output:

Original:
     Name  Maths  Science
0   Arjun   90.0     88.0
1    Bina   85.0      NaN
2  Chirag    NaN     70.0
3   Divya   92.0     90.0
4    Esha   88.0      NaN

Fill NaN with 0:
     Name  Maths  Science
0   Arjun   90.0     88.0
1    Bina   85.0      0.0   ← NaN replaced with 0
2  Chirag    0.0     70.0   ← NaN replaced with 0
3   Divya   92.0     90.0
4    Esha   88.0      0.0   ← NaN replaced with 0

Fill NaN with column mean:
     Name  Maths  Science
0   Arjun   90.0     88.0
1    Bina   85.0     82.67  ← Mean of (88,70,90) = 82.67
2  Chirag   88.75    70.0   ← Mean of (90,85,92,88) = 88.75
3   Divya   92.0     90.0
4    Esha   88.0     82.67  ← Mean again

Forward fill:
     Name  Maths  Science
0   Arjun   90.0     88.0
1    Bina   85.0     88.0   ← Takes value from row above (88)
2  Chirag   85.0     70.0   ← Takes Maths from row above (85)
3   Divya   92.0     90.0
4    Esha   88.0     90.0   ← Takes value from row above (90)

fillna() Strategies — When to use which:

2.7 📦 Function groupby() — Group and Analyse!

groupby() is one of the most useful functions in Pandas. It lets you:

1. Split data into groups based on a column
2. Apply a function (sum, mean, count, etc.) to each group
3. Combine results into a new DataFrame

Super Simple Analogy: Imagine dividing your class into Section A and Section B and then finding the average marks of each section separately. That's exactly what groupby('Section').mean() does!

graph LR
    ALL["📋 All Students\n(Mixed Sections)"]
    GRP["groupby('Section')"]
    A["📊 Section A\nArjun, Bina, Esha"]
    B["📊 Section B\nChirag, Divya"]
    AGG["apply mean()"]
    RES["📊 Result\nA: avg marks\nB: avg marks"]

    ALL --> GRP
    GRP --> A
    GRP --> B
    A --> AGG
    B --> AGG
    AGG --> RES

    style GRP fill:#FF9800,color:#fff
    style RES fill:#4CAF50,color:#fff

Basic groupby() example:

# groupby_basic.py
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'Name':    ['Arjun', 'Bina', 'Chirag', 'Divya', 'Esha', 'Farhan'],
    'Section': ['A', 'A', 'B', 'B', 'A', 'B'],
    'Maths':   [90, 85, 78, 92, 88, 74],
    'English': [75, 80, 72, 85, 79, 68]
})

print("Original DataFrame:")
print(df)

# Group by Section and find MEAN of each group:
print("\nSection-wise AVERAGE:")
print(df.groupby('Section').mean())

# Group by Section and find MAX of each group:
print("\nSection-wise MAXIMUM:")
print(df.groupby('Section').max())

# Group by Section and find COUNT:
print("\nSection-wise COUNT (number of students):")
print(df.groupby('Section').count())

Output:

Original DataFrame:
     Name Section  Maths  English
0   Arjun       A     90       75
1    Bina       A     85       80
2  Chirag       B     78       72
3   Divya       B     92       85
4    Esha       A     88       79
5  Farhan       B     74       68

Section-wise AVERAGE:
         Maths    English
Section
A        87.67    78.00
B        81.33    75.00

Section-wise MAXIMUM:
          Name  Maths  English
Section
A         Esha     90       80
B        Farhan    92       85

Section-wise COUNT:
         Name  Maths  English
Section
A           3      3        3
B           3      3        3

More groupby() Examples:

# groupby_advanced.py

# Sum of marks per section:
print("Total marks per section:")
print(df.groupby('Section')[['Maths', 'English']].sum())

# Groupby and get specific statistic:
print("\nMinimum Maths marks per section:")
print(df.groupby('Section')['Maths'].min())

# Get complete statistics per group:
print("\nFull statistics per section:")
print(df.groupby('Section').describe())

Output:

Total marks per section:
         Maths  English
Section
A          263      234
B          244      225

Minimum Maths marks per section:
Section
A    85
B    74
Name: Maths, dtype: int64

Groupby with multiple columns:

# Add City column to demo multiple groupby:
df['City'] = ['Delhi', 'Mumbai', 'Delhi', 'Pune', 'Mumbai', 'Delhi']

print("Section AND City-wise average:")
print(df.groupby(['Section', 'City'])['Maths'].mean())

result = df.groupby('Section').mean()
print(result)

⚠️ Common Errors and Misconceptions

🔑 Quick Revision

Iteration:

• iterrows() → row by row → (index, Series)
• items() / iteritems() → column by column → (col_name, Series)

Statistical Functions:

Missing Data:

• isnull() → True where NaN | notnull() → True where not NaN
• dropna() → remove rows/columns with NaN
• fillna(value) → replace NaN with value

groupby():

• df.groupby('col').mean() → group by col, find average
• df.groupby('col').sum() → group by col, find total
• df.groupby('col').count() → count per group

Cumulative:

• cumsum() → running total
• cummax() → running maximum
• cummin() → running minimum

🎯 Sample Board Exam Questions

Q1: Very Short Answer [1 mark each]

a) Which function gives all 8 statistics (count, mean, std, min, 25%, 50%, 75%, max) in one call? → df.describe()

b) What does isnull() return? → A DataFrame of True/False — True where values are missing (NaN), False where present.

c) What does cumsum() do? → Returns the cumulative running sum — each value is the sum of all elements up to that position.

d) Which function removes rows containing NaN? → dropna()

e) What is the difference between count() and sum()? → count() returns the number of non-NaN values; sum() returns the total (addition) of all non-NaN values.

Q2: Short Answer [2 marks]

Q: Differentiate between dropna() and fillna().

dropna() removes rows (or columns) that contain NaN values — the affected rows are deleted from the result. fillna(value) replaces NaN values with a specified value (e.g., 0, the column mean, or 'Unknown') — no rows are removed.

df.dropna()         # Removes Chirag's row (NaN in Science)
df.fillna(0)        # Keeps Chirag but replaces his NaN with 0

Q3: Program Writing [3 marks]

Q: Create a DataFrame of 4 students with Maths and Science marks (one NaN). Find section-wise average using groupby(). Also fill NaN with the column mean.

import pandas as pd
import numpy as np

data = {
    'Name':    ['Arjun', 'Bina', 'Chirag', 'Divya'],
    'Section': ['A', 'A', 'B', 'B'],
    'Maths':   [90, 85, np.nan, 92],
    'Science': [88, 95, 70, 90]
}
df = pd.DataFrame(data)

print("Original:")
print(df)

# Fill NaN with mean:
df_filled = df.fillna(df.mean())
print("\nAfter filling NaN with mean:")
print(df_filled)

# Section-wise average:
print("\nSection-wise average:")
print(df.groupby('Section').mean())

Q4: Output Based [3 marks]

Q: What is the output?

import pandas as pd
import numpy as np

s = pd.Series([10, 20, 30, 40, 50])
print(s.cumsum())
print(s.std())

Output:

0     10
1     30
2     60
3    100
4    150
dtype: int64
15.811388300841896

(cumsum: running totals 10, 10+20=30, 30+30=60, 60+40=100, 100+50=150) (std: standard deviation of [10,20,30,40,50])

✏️ Practice Problems

1. Create a DataFrame of 5 products with their prices and quantities. Use describe() to get the statistical summary.

2. Write code to iterate over each row of a student DataFrame and print "Pass" if marks ≥ 40, else "Fail".

3. Create two DataFrames with matching and non-matching indexes. Add them using .add(fill_value=0) and explain the result.

4. From a DataFrame with NaN values, write code to: (a) count NaN per column, (b) drop rows with NaN, (c) fill NaN with the column median.

5. Create a DataFrame with students from different cities. Use groupby('City') to find the maximum marks per city.

6. Find the cumulative sum of a Series [5, 10, 15, 20] and predict the output before running.

7. Write code to apply a custom function that calculates (max - min) for each column of a DataFrame.

8. What is the difference between iterrows() and items()? Write one example of each.

9. A DataFrame has columns 'Month' and 'Sales'. Use cumsum() to show running total sales and cummax() to track the best month so far.

10. Create a DataFrame, then: (a) find mode of each column, (b) find the 75th percentile (Q3) using quantile(), (c) find standard deviation.