Loading Data: read_csv Done Right

What you'll build

A loader for a messy export: semicolon separators, a comma as decimal separator, N/A and - meaning missing, dates in day.month.year format, and columns you don't need. The result is a clean DataFrame with correct types, ready for analysis.

The idea

read_csv has options for almost every real-world format problem:

Writing works the same way: to_csv(index=False), to_json(orient="records"), to_parquet for large data.

The code

from io import StringIO
import pandas as pd

raw = StringIO('''Kunde;Datum;Betrag;Status;Notiz
0042;15.01.2026;1.234,50;paid;-
0043;16.01.2026;N/A;open;call back
0044;17.01.2026;99,90;paid;-
''')
df = pd.read_csv(
    raw,
    sep=";", decimal=",", thousands=".",
    usecols=["Kunde", "Datum", "Betrag", "Status"],
    dtype={"Kunde": "str", "Status": "category"},
    parse_dates=["Datum"], date_format="%d.%m.%Y",
    na_values=["N/A", "-"],
)
df.columns = ["customer", "date", "amount", "status"]
print(df)
print(df.dtypes)
print(df.to_csv(index=False))

  customer       date  amount status
0     0042 2026-01-15  1234.5   paid
1     0043 2026-01-16     NaN   open
2     0044 2026-01-17    99.9   paid
customer               str
date        datetime64[us]
amount             float64
status            category
dtype: object
customer,date,amount,status
0042,2026-01-15,1234.5,paid
0043,2026-01-16,,open
0044,2026-01-17,99.9,paid

Common mistakes

• Letting IDs like 0042 be read as integers, losing leading zeros; read them as str.
• Parsing dates without a format, which can swap day and month for ambiguous values.
• Reading huge files fully into memory; use usecols, dtype, chunksize or Parquet.
• Writing CSVs with the index (to_csv() default), adding an unnamed extra column.

Key points

• Fix separators, decimals, types, dates and missing markers at load time.
• Keep IDs as strings and parse dates with an explicit format.
• Write with index=False; use Parquet for large or typed data.

Further reading: read_csv · IO tools