Fast and Correct pandas 3 Code

What you'll build

A before-and-after of common pandas habits: the copy-on-write behaviour that changed in pandas 3, the dedicated str dtype, memory savings with category, and a vectorised replacement for a slow apply.

The idea

• Copy-on-write (default in pandas 3): every selection behaves like a copy; modifying a subset never changes the original, and chained assignment never works. Always modify with one loc call on the DataFrame you mean.
• String dtype: text columns use the str dtype by default in pandas 3 instead of generic object.
• Categoricals store repeated labels once: large memory savings for columns like country or status.
• Vectorise: column arithmetic, .str and .dt accessors, np.select beat apply(axis=1).

The code

import numpy as np
import pandas as pd

df = pd.DataFrame({"country": ["DE", "US", "DE"] * 1000, "amount": np.arange(3000, dtype=float)})
subset = df[df["country"] == "DE"]
subset.loc[:, "amount"] = 0              # modifies only subset (copy-on-write)
print("original untouched:", df["amount"].sum() > 0)
df.loc[df["country"] == "DE", "amount"] = 0    # the right way to change df
print("DE rows now zero:", (df.loc[df["country"] == "DE", "amount"] == 0).all())

print("text dtype:", df["country"].dtype)
as_category = df["country"].astype("category")
print("memory bytes:", df["country"].memory_usage(deep=True), "->", as_category.memory_usage(deep=True))

df["band"] = np.select([df["amount"] == 0, df["amount"] < 1000], ["zero", "low"], default="high")
print(df["band"].value_counts().to_dict())

original untouched: True
DE rows now zero: True
text dtype: str
memory bytes: 81072 -> 3126
{'zero': 2000, 'high': 667, 'low': 333}

Common mistakes

• Relying on modifying a filtered DataFrame to change the original (worked by accident in some old cases, never in pandas 3).
• object columns full of strings in old code; convert with astype("str") (or let pandas 3 infer it).
• apply(lambda row: ..., axis=1) for logic that np.select, np.where or .str methods can express.
• Loading everything as float64/object when category or smaller numeric types would do.

Key points

• In pandas 3, selections behave like copies: assign with loc on the original.
• Text uses the str dtype; repeated labels belong in categories.
• Vectorised operations are faster and clearer than row-wise apply.

Further reading: Copy-on-Write · What's new in pandas 3.0 · Scaling to large datasets