Skip to content

Job scheduling and delayed work

How a system runs work later or on a calendar: a durable record of due times, something that wakes due items and hands them to workers, and explicit rules for duplicates, missed runs and cancellation.

On this page

What it is

Delayed work is a job that should run once at a later moment: send a reminder the day before an appointment, release a held seat if payment hasn't arrived, retry a call after a backoff. Scheduled work is a job that recurs on a calendar: an hourly cleanup, a nightly aggregate. Both have the same shape underneath. Somewhere there is a record of jobs with their due times, something notices when a job becomes due, and workers run it; the designs on this page make that record durable so a restart doesn't lose it. What makes this different from a plain queue is that readiness depends on the clock rather than on arrival order, so we'd sort or index the store by due time to find what is due cheaply, and some component has to own the clock.

The smallest version is a delay built into the queue. In Amazon SQS, a delay queue keeps new messages invisible to consumers for a configured number of seconds; the default is 0 and the maximum is 15 minutes. A message timer sets the same DelaySeconds on one message instead of the whole queue. This differs from the visibility timeout: a delay hides a message when it arrives, while the visibility timeout hides it after a consumer has received it. For anything beyond 15 minutes, the SQS guide points to EventBridge Scheduler, which runs one-time or recurring actions without that limit.

When the horizon is longer, or you need to cancel or reschedule, we'd reach for a store you control. Each job is a row (or an entry in a sorted set) carrying its payload, a run_at, a status and a locked_until. A dispatcher loop asks for items whose run_at has passed and whose lease has expired, sets a new lease on each one it takes, and either runs it or publishes it to the regular work queue. When the handler finishes, the dispatcher marks the row done or writes the next run_at for a recurring job. The lease is what stops two dispatchers from taking the same row, and its expiry is what lets a job survive a dispatcher that dies mid-flight. The cost is lateness: a job can run as late as one poll interval after it is due, plus whatever the queue and worker add.

Workflow engines fold the delay into the program itself. In Temporal, a workflow calls sleep() or creates a timer for a fixed period, and the timer is persisted: a workflow can sleep for days, months or years, and if the worker or the Temporal service is down when the period ends, the call resolves once they are back. Sleeping is a light operation that doesn't hold a process, so one worker can own millions of timers. That fits a delay that is one step in a longer process, such as waiting for a payment before the next action.

Recurring schedules add a scheduler process. Celery 5.7's beat reads a schedule and sends tasks to workers at the configured times. Its documentation says to run a single scheduler per schedule or you get duplicate tasks; runs may overlap if the previous one hasn't finished (so take a lock when that matters); and beat keeps the last run times in a local file. Time zones bite here: the schedule defaults to UTC, and an interval schedule such as timedelta(days=1) measures elapsed time across a daylight-saving change, so a task last run at noon becomes due at 13:00 or 11:00 local the next day. A crontab schedule is how you keep a job at the same local hour.

Question to settleWhat goes wrong if you don'tWhat to decide

Duplicates

Two schedulers or two dispatchers both run the same job

A lease or a claim in the store; handlers that check state before acting

Missed runs

After an outage, a nightly job runs ten times at once, or not at all

Catch up once, catch up each missed run, or skip; per job

Lateness

A reminder lands minutes after the meeting

Poll interval and worker capacity sized to the tolerance

Cancellation

A reservation is released after it was paid

Delete the job, or have the handler re-check the current state

Herds

Jobs due at the top of the hour all fire together

Spread due times with jitter; cap how many run at once

When to use it

Reach for a scheduler when the trigger is time rather than an event, and the process that decides to do the work shouldn't be the one that waits for it: reminders, trial endings, holds that expire, retries with a backoff longer than the queue's own delay, and periodic rollups. Pick the mechanism by horizon and by what you need to do with the job while it waits. A delay of seconds to a few minutes on a queue you already run fits a queue-native delay. Days or months, or a job you may need to cancel or move, fits a store you control. A wait that sits between two other steps of the same business process fits a durable timer in a workflow engine, where the code reads as one sequence and the engine keeps the timer across restarts.

When not to

Don't schedule what you can compute at read time. A seat hold with an expires_at column can be treated as free by any reader whose clock has passed it, and a slow periodic sweeper deletes the stale rows in bulk; that removes one job per hold and the cancellation problem with it. Don't use a timer where an event exists: a job that wakes every minute to ask whether a payment arrived is a polling loop in disguise, and the payment event itself should start the next step. And a wait of a few hundred milliseconds inside a single request belongs in that request, not in a job store.

Don't build a scheduler for retries the queue already handles: if your broker already redelivers work a consumer didn't finish, a short backoff may fit inside that path; check its limits first. A recurring job on one host with no availability requirement is fine as plain cron. And for a single worker node, Celery allows embedding beat in the worker with -B, though that isn't recommended for production.

Compared

ApproachHorizonCancel or move a jobRight when

Queue-native delay (SQS delay queue, message timer)

Up to 15 minutes

Hard: the message is hidden inside the queue

Short backoffs on a queue you already run; nothing to cancel

Table with run_at and a polling dispatcher

Any

Easy: update or delete the row

Moderate volume, long horizons, jobs that may be canceled

In-memory sorted set keyed by due time

Any, if the store is durable

Easy: remove the member

High volume of short-lived timers where the store's durability is acceptable

Durable timer in a workflow engine (Temporal)

Days to years

Cancel the timer from the workflow

The wait is one step in a multi-step process

Cron-style scheduler (Celery beat, cron)

Recurring

Edit the schedule

Periodic jobs; one scheduler instance, with a lock if runs can overlap

In a picture

run_atAPIJobs table

Step 1 of 4

Persist the due time

The API writes the job with its run_at in the same transaction as the business change, so a crash after the response loses nothing.

How to say it in an interview

The common mistake

Quiz

A seat hold must be released 10 minutes after it is created unless it is paid. Which design is simplest and still correct?
A dispatcher polls a jobs table on a fixed interval. How late can a job start?
Two scheduler instances both come up after a deploy. What stops the nightly report from running twice?
You need a message delivered three days from now, and the team uses SQS. What is true?

Questions that use this

Related

Sources

  1. Amazon SQS Developer Guide: Amazon SQS delay queues
  2. Celery 5.7.0 documentation: Periodic Tasks
  3. Temporal documentation: Timers (Go SDK)
Sign in to report a mistake