Running it every morning: cron, a lock file, and a job that is safe to run twice

The scheduler does very little

A scheduler starts your program at a time. That is all. Everything else — where it runs, what it can see, what happens if it is still running when the next start arrives, what happens if it runs twice — is your program's problem, and a script that works perfectly by hand fails under a scheduler for exactly those reasons.

cron, and the free alternatives

On macOS and Linux, crontab -e opens a file of lines:

# minute hour day month weekday   command
0 7 * * *   cd /home/asha/tagger && /home/asha/tagger/.venv/bin/python -m tagger.daily >> logs/daily.log 2>&1

Five time fields, then the command. 0 7 * * * is seven every morning. */15 * * * * is every fifteen minutes. The site crontab.guru decodes any line.

Three things in that command are deliberate. The cd sets the working directory, because cron starts in your home folder and module 5's relative paths will be wrong otherwise. The full path to the venv's Python selects the right interpreter and packages, because cron's PATH is nearly empty and python may resolve to nothing or to the system's. And >> logs/daily.log 2>&1 captures stdout and stderr to a file, because cron's output otherwise goes to a local mail spool nobody reads.

Windows has Task Scheduler with the same shape; macOS also has launchd, which survives the machine being asleep at the scheduled minute better than cron does. And GitHub Actions, in the next lesson, has an on: schedule: trigger with cron syntax that runs your job on their machines for free — the right choice for a job that should run even when your laptop is closed.

The environment is not yours

Cron does not read ~/.zshrc, ~/.bashrc or ~/.zprofile. Every variable you exported there — the API key, the database URL — is absent. The script fails with KeyError on the variable name, or worse, runs with a default and processes nothing.

Module 5 gave the fix: the job loads its own configuration. A .env file read by load_dotenv() at the top of main(), with its path derived from __file__ so the working directory does not matter. Or a line in the crontab itself: OPENAI_API_KEY=... above the schedule lines, though that puts the key in a file with wide read permissions on some systems, and the .env is better. Print bool(os.getenv("OPENAI_API_KEY")) as the first log line so a missing key is the first thing the log says.

Refusing to overlap

A job scheduled every fifteen minutes that takes twenty will run alongside itself, two copies processing the same items and doubling the bill. A lock file prevents it:

import fcntl, sys

def main():
    lock = open("/tmp/tagger-daily.lock", "w")
    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        log.warning("previous run still active; exiting")
        return 0
    run()

flock with LOCK_NB takes an exclusive lock or fails immediately. The operating system releases it when the process ends, even if it crashes — which a "create a file, delete it at the end" scheme does not, leaving a stale file that blocks every future run after one crash. On Windows, msvcrt.locking does the same; the filelock package (free) wraps both.

Safe to run twice

Sooner or later a job will run twice on the same data: the scheduler fires twice, you re-run it by hand after a partial failure, a machine restarts. A job that re-posts every summary, re-sends every email, or re-charges every call on the second run is a job you cannot safely retry, and one you cannot retry is one you cannot fix.

The property to design for is idempotency — module 6's word, applied to your own program: running it again produces the same final state, not twice the effects. The mechanism is a record of what has been done:

def run():
    done = load_done_ids("state/done.json")          # or a sqlite table
    for msg in fetch_new_messages():
        if msg.id in done:
            continue
        summary = summarise(msg)
        post(summary, idempotency_key=msg.id)
        done.add(msg.id)
        save_done_ids(done)                          # after each item, not at the end

Record each item as it completes, so a crash at item 30 of 50 resumes at 31. Pass the item's ID as an idempotency key to anything that accepts one, so even a duplicate post is absorbed at the far end. Make the record the source of truth about what is done, never the log.

The module-6 cache helps here too: a re-run that hits the cache for every already-summarised message costs nothing even if the done-record is lost.

Exit codes and logs

Return 0 on success and non-zero on failure from main(), and let module 3's sys.exit(main()) carry it out. Schedulers and CI record the code; it is how "did it work" becomes a query rather than a search through logs.

Log to a file, with rotation, so a job that runs for a year does not fill the disk:

from logging.handlers import RotatingFileHandler
handler = RotatingFileHandler("logs/daily.log", maxBytes=5_000_000, backupCount=5)

Log one line at start with the timestamp and configuration, one per item at INFO, and one at the end with counts: processed, skipped, failed, and the total cost from module 6's Budget. A job whose last line is done: 48 processed, 2 failed, $0.31 is a job you can check in five seconds.

When it silently stops

The worst failure is the job that stopped running three weeks ago and nobody noticed. The cheap defence is a heartbeat: the job's last act is to write the current time to a file or hit a URL, and something else — a second tiny cron line, a free monitoring service — alerts when the heartbeat is older than expected. This is the one addition that turns a script into something you can stop watching.

Try this now

Put your summariser under cron every two minutes with the venv path and log redirection. Watch it fail on the missing key, fix it with .env, then make the job take three minutes and watch the lock reject the overlap. Finally, run it twice by hand and confirm the second run posts nothing.