pandas: a table with named columns, and the four things to look at first

What a DataFrame is

Module 5 read CSV files with the csv module, one row at a time as dicts. That is right for streaming and for files whose rows you process independently. When you want to ask questions across rows — averages by group, joins, missing-value counts — you want the whole table in memory as columns, and that is pandas.

import pandas as pd

df = pd.read_csv("orders.csv")

A DataFrame is best understood as a dict of columns, where each column is a NumPy-backed Series with one dtype, and all of them share an index — the row labels, which default to 0, 1, 2. Everything fast in pandas is fast because a column is an array; everything slow is slow because you made it loop over rows.

The four things to look at first

Before any analysis, four commands, every time:

df.shape          # (rows, columns)
df.head()         # first five rows — does it look like what you expected?
df.dtypes         # one line per column: int64, float64, object, bool, datetime64
df.isna().sum()   # missing values per column

dtypes is where the surprises are. read_csv guesses each column's type from its contents, and the guesses fail in predictable ways:

• object means "strings, or a mix". A numeric column with one stray value — N/A, 7002A, a trailing space — becomes object for every row. Arithmetic on it fails; a merge against an int64 column of the same IDs matches nothing, because 7002 != "7002".
• IDs read as numbers. 007002 becomes 7002, losing the leading zeros that a phone number, a postcode or an account ID needs. Pass dtype={"account": str} to keep them.
• Dates read as strings. 2026-09-04 is object until you say parse_dates=["created_at"] or convert afterwards with pd.to_datetime.

Fix these at read time, not afterwards:

df = pd.read_csv("orders.csv", dtype={"order_id": str, "postcode": str},
                 parse_dates=["created_at"], na_values=["N/A", "-", ""])

df.info() prints dtypes, non-null counts and memory in one report; df.describe() gives min, max, mean and quartiles for numeric columns, which catches a price column with a maximum of 999999.

Selecting

df["price"]                     # one column: a Series
df[["order_id", "price"]]       # several columns: a DataFrame
df[df["price"] > 100]           # rows by condition — a boolean mask, as in NumPy
df.loc[df["price"] > 100, "order_id"]      # rows by condition, one column
df.iloc[0:5, 0:3]               # rows and columns by position
df.loc[42]                      # the row whose index label is 42

loc selects by label — index values and column names. iloc selects by integer position. They differ the moment the index is not 0..n−1, which happens after filtering or sorting: df.iloc[0] is the first row now; df.loc[0] is the row that was labelled 0 before, if it survived. Combining conditions uses &, |, ~ with brackets, exactly as NumPy did.

df.query("price > 100 and status == 'paid'") is the same filter as a string, easier to read for long conditions.

Adding and changing columns

df["total"] = df["price"] * df["quantity"]              # vectorised
df["month"] = df["created_at"].dt.month                 # datetime accessor
df["email_domain"] = df["email"].str.split("@").str[1]  # string accessor

Whole-column operations. The .str and .dt accessors apply string and date methods across a column without a loop. If you find yourself writing for i, row in df.iterrows(), stop: iterrows is the slowest way to touch a table, hundreds of times slower than a column expression, and the column expression almost always exists. df.apply(func, axis=1) is a little better and still a loop; use it only for logic that cannot be written by column.

The copy warning

paid = df[df["status"] == "paid"]
paid["flag"] = True
# SettingWithCopyWarning: A value is trying to be set on a copy of a slice...

paid may be a view of df or a copy, and pandas is telling you it cannot promise which, so your assignment may or may not reach df. The fix is to say what you mean: paid = df[df["status"] == "paid"].copy() if you want an independent table, or df.loc[df["status"] == "paid", "flag"] = True if you want to change df. Recent pandas versions with copy-on-write enabled make every such selection a copy and the warning disappears; the habit of .copy() or .loc assignment is right either way.

Getting back out

df.to_csv("clean.csv", index=False)
df.to_parquet("clean.parquet")            # needs pyarrow; smaller, typed, fast
df["price"].to_numpy()                    # a NumPy array
df[["x", "y"]].to_numpy(dtype=np.float32) # a (n, 2) matrix for a model

index=False on to_csv stops pandas writing the row numbers as a column, which otherwise comes back as Unnamed: 0 on the next read. Parquet preserves dtypes, so the ID-as-string and the parsed dates survive a round trip; CSV forgets them and you re-fix them on every load.

Try this now

Load any CSV with at least one ID column and one date. Run the four commands. Find every object column and decide whether it should be. Re-read with dtype= and parse_dates= until dtypes says what you mean, then write it to Parquet and read it back.