JSON: the format every API speaks

Four functions, and the difference between them

import json

text = json.dumps(data)              # object  -> string
data = json.loads(text)              # string  -> object

with open(p, "w", encoding="utf-8") as f:
    json.dump(data, f)               # object  -> file
with open(p, encoding="utf-8") as f:
    data = json.load(f)              # file    -> object

The s means string. dumps/loads work with text in memory; dump/load work with an open file. Mixing them up gives TypeError: the JSON object must be str, bytes or bytearray, not TextIOWrapper, which is the error telling you that you passed a file where a string was expected.

What maps to what

JSON's types are fewer than Python's, and the mapping is worth knowing exactly:

• object becomes dict, array becomes list
• string becomes str, number becomes int or float
• true/false become True/False, null becomes None

Round-tripping is not lossless. A tuple becomes a list. A set, a datetime, a Decimal and a NumPy number all raise TypeError: Object of type X is not JSON serializable. Dictionary keys that are integers come back as strings, because JSON object keys are always strings. That last one silently breaks code that looks up data[1] after a round trip.

Making it readable, and keeping the language

json.dumps(data, indent=2, ensure_ascii=False)

indent=2 produces the readable form for a config file or a debugging print. Leave it out for anything you send over a network — the whitespace is real bytes.

ensure_ascii=False matters more than it looks. By default, json.dumps escapes every non-ASCII character, so {"city": "मुंबई"} becomes {"city": "\u092e\u0941\u0902\u092c\u0908"}. That is valid JSON and any parser reads it back correctly, but the file is unreadable to a human and three times larger. For anything a person will open, pass ensure_ascii=False and write the file as UTF-8.

sort_keys=True gives a stable ordering, which is what makes two JSON files comparable with a diff tool and what makes a checksum meaningful.

Serialising what it refuses

from datetime import datetime, date
from decimal import Decimal

def encode(obj):
    if isinstance(obj, (datetime, date)):
        return obj.isoformat()
    if isinstance(obj, Decimal):
        return str(obj)
    if isinstance(obj, set):
        return sorted(obj)
    raise TypeError(f"cannot serialise {type(obj).__name__}")

json.dumps(data, default=encode)

default is called for anything the encoder does not recognise. Note two choices in there. Dates go out as ISO 8601 strings, because that is what every other language reads. Decimal becomes a string, not a float, because converting it to a float is exactly the precision loss you chose Decimal to avoid — and the reader must know to convert it back.

Nothing converts these back automatically on load. If you need real datetime objects, pass object_hook to loads, or convert after parsing.

JSON Lines, for anything large

One JSON object per line, no enclosing array:

with open("events.jsonl", "a", encoding="utf-8") as f:
    for event in events:
        f.write(json.dumps(event, ensure_ascii=False) + "\n")

This is the format for logs, datasets and model outputs, for three reasons: you can append to it without rewriting the file, you can read it one record at a time regardless of size, and one corrupt line costs you one record rather than the entire file. A 10 GB .json array must be parsed whole; a 10 GB .jsonl streams.

When the parse fails

json.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This nearly always means the text is not JSON at all. The usual cause is an API returning an HTML error page, or an empty body, with a status code you did not check. Print the first 200 characters before parsing and you will see a document type declaration staring back at you.

Trailing commas, single quotes and comments are all invalid JSON, even though Python accepts them in its own literals. JSON5 and jsonc are separate formats needing separate parsers.

Do not accept a binary Python object file from outside

Python's own binary serialisation format is convenient, and it is not a data format — it is a program that runs on load. Loading such a file from an untrusted source executes whatever code that file specifies, before you get a chance to inspect anything.

Use JSON for anything crossing a boundary. Keep the binary format for your own temporary files, on your own machine, and never load one that arrived from a download, an upload, or a colleague's email. The same warning applies to model checkpoint files in the older PyTorch format, which use that mechanism underneath — which is why the safer safetensors format was created and why model hubs now prefer it.