I Kill My Workers on Purpose
Building a crash-safe workflow engine on PostgreSQL: one transaction per decision, leases instead of locks, and a test that kills its own worker

The test that kills a worker
The test I'm proudest of so far starts ten workflows, lets a worker claim all ten jobs, and then kills it with SIGKILL halfway through.
No catch block runs. No cleanup. No "I'm shutting down" message. The process is simply gone, holding ten jobs nobody else knows are abandoned.
Then the test starts a second worker and waits.
A few seconds later, all ten workflows are complete, and the history shows every step completed exactly once.
That test now runs on every pull request.
This post is about the guarantees that make it pass.
Canonik is the workflow engine I'm building: a neutral system of record for work performed by people, AI agents, and systems. In the first post, I explained why I decided to own the orchestration engine rather than adopt one. Since then I've built its core: a pure decision function, a PostgreSQL schema with tenant isolation enforced by the database, an atomic way to persist decisions, and workers that execute queued jobs and recover from crashes.
Four packages, 69 tests, and one deliberately violent test.
The happy path was never the hard part. Every interesting decision came from asking:
What happens when this fails?
Decide, then do
At the center of the engine is a pure function:
decide(definition, state, event) → { state, events, effects }
Give it a workflow definition, the current state, and something that just happened ("start" or "review completed"), and it returns:
- the new state
- the history events to record
- the work that should happen next
It never touches a database, a clock, or a network. It doesn't even mutate its inputs.
That sounds academic until something breaks.
Every rule of the engine is testable with plain objects in milliseconds. Today, 28 tests run in under 200 ms.
More importantly, every bug ends up in one of two places:
- the decision was wrong
- the infrastructure failed to record or execute the decision
The same function will eventually power workflow simulation, letting a workflow run against the real engine before anyone activates it.
Because deciding is separate from doing, the hardest problem, recording a decision safely, lives in exactly one place.
That place follows one rule:
Every decision is saved in a single transaction.
The new state, the history events, and one row in a jobs table for each effect commit together or not at all.
This is the transactional outbox pattern. It gives the guarantee I lean on everywhere else:
a job exists if and only if the decision that created it committed.
There's never a queued job for a step the database doesn't think started, and never a state change whose follow-up work was lost.
I proved it the blunt way: a test installs a trigger that makes the final write fail and then verifies that nothing written before it survives.
The worker-kill test depends on this guarantee. When another worker eventually finds a job, it can trust the workflow decision that created it actually committed.
When two things happen at once
A workflow stage can run steps in parallel. When a new customer signs up, Canonik checks them against a fraud blocklist while, in parallel, it starts identity verification. That step only sends the request to a verification provider; the result arrives later and is handled as its own step.
Both steps start at the same moment, run on different workers, take about the same time, and often finish within milliseconds of each other. At scale, and whenever a backlog of jobs drains at once, that's not a rare coincidence; it's routine.
Without care, two transactions read the same workflow state and each overwrite the other's progress. One completion disappears and the stage never advances.
Nothing errors.
The workflow just sits there forever.
The fix is a row lock.
Before deciding anything, the operation runs:
SELECT ...
FOR UPDATE
The second transaction waits. When it resumes, PostgreSQL shows it the state the first transaction just committed. It sees "fraud check completed" and correctly finishes the stage.
I also keep a revision number on every workflow, and it's tempting to think it serves the same purpose.
It doesn't.
The row lock handles simultaneous writers. The revision number catches a different problem: someone making a decision from a screen that became stale after it loaded.
In the end I had four defenses, each covering a different failure mode:
| Guard | Catches |
|---|---|
| Row lock | Two changes at the same instant |
| Expected revision | Decisions made on stale data |
| Step validation | Duplicate or late completions |
| Unique history constraint | Future bugs that bypass the lock |
The parallel-completion test runs 20 times in a row because one passing race proves very little.
Workers that never wait for each other
The outbox fills the jobs table. Workers empty it.
PostgreSQL is the queue, a choice I made on purpose. Jobs commit atomically with the decisions that create them, and there's only one system to operate.
The entire claiming protocol is a single SQL statement:
UPDATE jobs
SET status = 'running',
attempts = attempts + 1,
lease_expires_at = now() + make_interval(secs => $lease)
WHERE id IN (
SELECT id
FROM jobs
WHERE status = 'ready'
AND run_after <= now()
ORDER BY run_after
LIMIT $batch
FOR UPDATE SKIP LOCKED
)
RETURNING *;
Three ideas carry most of the weight.
SKIP LOCKED
If another worker is claiming a row at the same moment, skip it rather than wait.
Workers never block each other and never claim the same job.
A lease, not a lock
The database lock lasts only long enough to update the row.
After that, ownership is represented by status and a lease expiration timestamp.
That's important because the worker is about to call an external system. Holding a database lock across a network call is how one slow dependency freezes a database.
A fencing token
attempts + 1 creates a new claim number every time a job is acquired.
When the worker reports success, the update succeeds only if:
WHERE attempts = <the number I was given>
If the lease expired and another worker reclaimed the job, the number has already changed.
The late result matches nothing and is ignored.
The original design said:
Late results are never applied.
The fencing token is where that sentence became real, and it required no extra column.
Recording success is also atomic. Marking the job complete and advancing the workflow happen in the same transaction, so neither can happen without the other.
When things go wrong
Jobs fail in two fundamentally different ways.
The handler throws
The worker is alive and knows the job failed.
The job is rescheduled with exponential backoff:
- 2 seconds
- 4 seconds
- 8 seconds
- …
- capped at 5 minutes
After a maximum number of attempts, the job is marked failed and stops.
Poison jobs can't retry forever.
The worker disappears
Nobody reports anything.
The only evidence is an expired lease.
Every worker looks for expired leases as part of its normal polling loop and returns those jobs to the queue. Recovery requires no separate service.
One design choice turned out to be more important than it first appears:
Expired jobs also back off before being reclaimed.
An expired worker is not necessarily dead. It might simply be slow.
Immediate reclamation could put a second worker on the same job while the first is still running. And if the job is what crashed the worker, immediate retry just hands it to another worker, which crashes too.
A short delay reduces the first problem and prevents the second from cascading.
Before automating recovery, I watched it happen manually. With handlers configured to fail roughly half the time, 9 of 19 attempts failed and every workflow still completed.
Failures never moved a workflow forward.
Only successful outcomes did.
The kill -9 test
Then I automated it.
Worker A claims ten jobs with a two-second lease and starts ten-second handlers.
The test kills it with SIGKILL.
All ten jobs remain stuck in running, proving the worker died while holding work.
Worker B starts.
It notices the expired leases, reclaims the jobs, and completes them.
The test then verifies:
- every workflow reached completion
- every ticket job needed exactly two attempts
- every workflow history recorded each step completion exactly once
This is the test I care about most because it exercises almost every guarantee in the system at once:
- transactional persistence
- leases
- abandoned-job recovery
- fencing tokens
- exactly-once workflow progression
If any one of those breaks, the test fails.
There was one trap while writing it.
The usual way to run TypeScript, tsx, starts your script as a child process.
Kill tsx and you only kill the wrapper. The real worker keeps running, finishes the jobs, and the test passes without anything actually dying.
A recovery test that can't fail proves nothing.
The worker now launches with:
node --import tsx
which keeps everything inside a single process that the signal actually reaches.
What I got wrong along the way
Two mistakes taught me more than the successes.
A safeguard that wasn't doing the work
Tenant isolation is enforced through PostgreSQL Row-Level Security.
To prove the write check mattered, I deleted it and expected a test to fail.
It didn't.
PostgreSQL quietly applies a policy's read condition to writes when no explicit write condition exists, so the protection was still there.
Only replacing it with an explicit "allow anything" check made the test fail.
The lesson stuck:
If you haven't seen a safeguard fail, you don't know it's the one protecting you.
A test that ran code TypeScript would have rejected
One test failed with:
withTenant is not a function
I had imported it using import type, which TypeScript removes at runtime.
The test runner stripped types without performing type checking, so the mistake survived until execution.
Now every test run sits beside a type-check run, and CI executes both.
The honest limits
The database side of Canonik is exactly-once.
The outside world isn't.
If a worker creates a ticket in another system and dies before recording success, the job is reclaimed and the ticket may be created again.
That's at-least-once execution, a known limit for now.
It's also why every handler is still fake.
The fix is already planned:
- attach an idempotency key to every external call
- verify outcomes before retrying
Only then can external effects approach the guarantees the database already has.
What's next
So far Canonik is an engine you drive from a command line.
Next comes the first real application:
- an API
- organization sign-in
- a web UI for starting and completing work
- two demo organizations proving tenant isolation end to end
If you build systems where work crosses people, agents, and machines, I'd love to compare notes.
And if you've ever had a worker die mid-job in production, I'd love to hear what it cost you.
About the author
- I lead analytics and a growing portfolio of enterprise AI work, where I’ve learned that the hard part of AI isn’t the model. It’s accountability: who approved what, what an agent was allowed to do, and how you prove it. I’m building Canonik in public, a neutral orchestration engine for work done by people, AI agents, and systems, and writing here about the decisions behind it.
Comments
Join the discussionWhen the idempotency keys go in, I'd build them from the job id and step, never from the attempt number. The attempt counter is the right fence for your own tables, but if it ends up in the key, a reclaimed job hands the other system a key it has never seen, and the duplicate ticket gets created anyway. The fence has to change on every claim; the key has to stay the same across them.
The heartbeats you're planning can carry the same token. If the heartbeat runs UPDATE jobs SET lease_expires_at = ... WHERE id = $1 AND attempts = $2 and gets zero rows back, the worker has already lost the job and should stop before its next external call, instead of learning that when its success write matches nothing. For a system that doesn't accept idempotency keys, writing the job id into the record it creates, in an external reference field or a label, gives the retry something to search for before it creates another one.
This is really helpful. Thank you!
You're right. I was mentally carrying the attempt number around because it's already part of the claim lifecycle. Thinking about the idempotency key as the identity of the work itself, rather than a particular attempt, makes the distinction much clearer.
The plan already had fenced heartbeats, but not the part where losing the lease stops the handler before its next outside call. That’s a much better moment to find out than waiting for the final write to match nothing, so I’m adding that.
I also like the external reference idea for systems that don’t support idempotency keys. Search-before-create still leaves a small window if two attempts overlap, but it turns many duplicates from “create another record” into “find the one that’s already there.”
All three are going into the plan. You rock!
The point that a row lock and a revision number catch different failures is easy to miss. Simultaneous writers need the lock; a decision from a screen that went stale needs the revision. Treating them as one check would leave one of those gaps open. The fencing token from the claim number, with no extra column, is a clean way to drop a late result. And the backoff on expired leases is worth keeping: an overdue worker may only be slow, and an immediate retry of a job that crashed a worker can take the next one down with it. The honest limit at the end also belongs in the same picture: the database can be exactly once, while a call to another system may still run more than once if the worker disappears after the call and before the record.
The SIGKILL test is the honest proof here, since a clean shutdown never shows the real failure mode. One detail I would watch is the lease length: too short and a slow worker loses a job mid-step, too long and recovery waits, so I would tie it to the slowest step plus a margin.
Nice catch! I'd thought about slow handlers and lease length early on...then lost sight of it.
I should be able to get around the issue by having each handler declare its expected duration and workers send heartbeats to extend the lease while a long-running job is still healthy. That lets slow jobs keep their claim without forcing every lease to be long, so recovery can still happen quickly when a worker really does die.
I’ve added it to the plan. Thanks!
Heartbeats plus declared durations is a clean fix, since the lease can stay short for fast recovery while slow jobs keep their claim. One edge I would still test is a stalled heartbeat on a healthy handler, so a job is not reclaimed mid-step. Keen to see how it holds up in the next SIGKILL run.
The SIGKILL test is a great way to prove it: ten jobs claimed, the worker gone with no cleanup, and a second worker finishing all ten with every step completed exactly once. Keeping decide() pure, so every bug lands either in the decision or in recording it, is the part I would copy first.
We worked on the same promise at Ascendra Ventures for a seed-stage talent platform. Their hiring pipeline did not fail loudly, it just stopped. We rebuilt the profile processor as a durable workflow: urgent profiles run now, the rest are batched, and polling takes over when callbacks go quiet. The goal we wrote down was that every candidate finishes or visibly fails and never gets stuck in the middle. The numbers we shared afterwards were 99.99% system reliability and $0.004 per candidate imported, but the rule was what shaped the design.
How do you plan to handle a job that keeps crashing whichever worker picks it up, so it does not take down each new worker in turn?
Thanks! That’s a good question.
Most of the protection is already there. Every claim counts as an attempt, including reclaims after a worker dies. If a job keeps crashing whichever worker picks it up, it eventually hits the attempt cap and is marked failed instead of cycling forever.
Reclaimed jobs also back off before they’re retried, so a bad job can only take down one worker at a time. It doesn’t immediately get handed to the next worker and cascade through the whole pool.
The part I still want to improve is isolation. Right now, if a handler crashes the process, it takes the rest of that worker’s batch down with it. Running handlers in their own processes would shrink that blast radius a lot. That’s on the list.
Your “finishes or visibly fails” rule made me realize I’m only solving the infrastructure side of the problem today. The job fails, but the workflow isn’t told. From the request owner’s perspective, that’s still a thing stuck in the middle.
The fix I have planned is to turn terminal job failures into workflow events. Then the workflow knows the step failed and can route it to whoever can act on it, like the request owner when the input is bad or an admin when an external system is down.
Building Canonik
A public build log of Canonik, a neutral orchestration engine for work done by people, AI agents, and systems. Architecture decisions, trade-offs, failures, and what they taught me.
