# I Kill My Workers on Purpose

## The test that kills a worker

The test I'm proudest of so far starts ten workflows, lets a worker claim all ten jobs, and then kills it with `SIGKILL` halfway through.

No `catch` block runs. No cleanup. No "I'm shutting down" message. The process is simply gone, holding ten jobs nobody else knows are abandoned.

Then the test starts a second worker and waits.

A few seconds later, all ten workflows are complete, and the history shows every step completed exactly once.

That test now runs on every pull request.

This post is about the guarantees that make it pass.

Canonik is the workflow engine I'm building: a neutral system of record for work performed by people, AI agents, and systems. In the first post, I explained why I decided to own the orchestration engine rather than adopt one. Since then I've built its core: a pure decision function, a PostgreSQL schema with tenant isolation enforced by the database, an atomic way to persist decisions, and workers that execute queued jobs and recover from crashes.

Four packages, 69 tests, and one deliberately violent test.

The happy path was never the hard part. Every interesting decision came from asking:

**What happens when this fails?**

## Decide, then do

At the center of the engine is a pure function:

```ts
decide(definition, state, event) → { state, events, effects }
```

Give it a workflow definition, the current state, and something that just happened ("start" or "review completed"), and it returns:

- the new state
- the history events to record
- the work that should happen next

It never touches a database, a clock, or a network. It doesn't even mutate its inputs.

That sounds academic until something breaks.

Every rule of the engine is testable with plain objects in milliseconds. Today, 28 tests run in under 200 ms.

More importantly, every bug ends up in one of two places:

- the decision was wrong
- the infrastructure failed to record or execute the decision

The same function will eventually power workflow simulation, letting a workflow run against the real engine before anyone activates it.

Because deciding is separate from doing, the hardest problem, recording a decision safely, lives in exactly one place.

That place follows one rule:

**Every decision is saved in a single transaction.**

The new state, the history events, and one row in a `jobs` table for each effect commit together or not at all.

This is the transactional outbox pattern. It gives the guarantee I lean on everywhere else:

*a job exists if and only if the decision that created it committed.*

There's never a queued job for a step the database doesn't think started, and never a state change whose follow-up work was lost.

I proved it the blunt way: a test installs a trigger that makes the final write fail and then verifies that nothing written before it survives.

The worker-kill test depends on this guarantee. When another worker eventually finds a job, it can trust the workflow decision that created it actually committed.

## When two things happen at once

A workflow stage can run steps in parallel. When a new customer signs up, Canonik checks them against a fraud blocklist while, in parallel, it starts identity verification. That step only sends the request to a verification provider; the result arrives later and is handled as its own step.

Both steps start at the same moment, run on different workers, take about the same time, and often finish within milliseconds of each other. At scale, and whenever a backlog of jobs drains at once, that's not a rare coincidence; it's routine.

Without care, two transactions read the same workflow state and each overwrite the other's progress. One completion disappears and the stage never advances.

Nothing errors.

The workflow just sits there forever.

The fix is a row lock.

Before deciding anything, the operation runs:

```sql
SELECT ...
FOR UPDATE
```

The second transaction waits. When it resumes, PostgreSQL shows it the state the first transaction just committed. It sees "fraud check completed" and correctly finishes the stage.

![](https://cdn.hashnode.com/uploads/covers/6af8b738c3f8a2657ed92c78/7c5b7605-7fb0-43fe-90ca-4d1e67068430.png align="center")

I also keep a revision number on every workflow, and it's tempting to think it serves the same purpose.

It doesn't.

The row lock handles simultaneous writers. The revision number catches a different problem: someone making a decision from a screen that became stale after it loaded.

In the end I had four defenses, each covering a different failure mode:

| Guard | Catches |
| --- | --- |
| Row lock | Two changes at the same instant |
| Expected revision | Decisions made on stale data |
| Step validation | Duplicate or late completions |
| Unique history constraint | Future bugs that bypass the lock |

The parallel-completion test runs 20 times in a row because one passing race proves very little.

## Workers that never wait for each other

The outbox fills the `jobs` table. Workers empty it.

PostgreSQL is the queue, a choice I made on purpose. Jobs commit atomically with the decisions that create them, and there's only one system to operate.

The entire claiming protocol is a single SQL statement:

```sql
UPDATE jobs
SET status = 'running',
    attempts = attempts + 1,
    lease_expires_at = now() + make_interval(secs => $lease)
WHERE id IN (
  SELECT id
  FROM jobs
  WHERE status = 'ready'
    AND run_after <= now()
  ORDER BY run_after
  LIMIT $batch
  FOR UPDATE SKIP LOCKED
)
RETURNING *;
```

Three ideas carry most of the weight.

### SKIP LOCKED

If another worker is claiming a row at the same moment, skip it rather than wait.

Workers never block each other and never claim the same job.

### A lease, not a lock

The database lock lasts only long enough to update the row.

After that, ownership is represented by status and a lease expiration timestamp.

That's important because the worker is about to call an external system. Holding a database lock across a network call is how one slow dependency freezes a database.

### A fencing token

`attempts + 1` creates a new claim number every time a job is acquired.

When the worker reports success, the update succeeds only if:

```sql
WHERE attempts = <the number I was given>
```

If the lease expired and another worker reclaimed the job, the number has already changed.

The late result matches nothing and is ignored.

The original design said:

> Late results are never applied.

The fencing token is where that sentence became real, and it required no extra column.

Recording success is also atomic. Marking the job complete and advancing the workflow happen in the same transaction, so neither can happen without the other.

## When things go wrong

Jobs fail in two fundamentally different ways.

### The handler throws

The worker is alive and knows the job failed.

The job is rescheduled with exponential backoff:

- 2 seconds
- 4 seconds
- 8 seconds
- …
- capped at 5 minutes

After a maximum number of attempts, the job is marked `failed` and stops.

Poison jobs can't retry forever.

### The worker disappears

Nobody reports anything.

The only evidence is an expired lease.

Every worker looks for expired leases as part of its normal polling loop and returns those jobs to the queue. Recovery requires no separate service.

One design choice turned out to be more important than it first appears:

**Expired jobs also back off before being reclaimed.**

An expired worker is not necessarily dead. It might simply be slow.

Immediate reclamation could put a second worker on the same job while the first is still running. And if the job is what crashed the worker, immediate retry just hands it to another worker, which crashes too.

A short delay reduces the first problem and prevents the second from cascading.

![](https://cdn.hashnode.com/uploads/covers/6af8b738c3f8a2657ed92c78/7db21075-61d6-4ab7-810a-8f80a473374e.png align="center")

Before automating recovery, I watched it happen manually. With handlers configured to fail roughly half the time, 9 of 19 attempts failed and every workflow still completed.

Failures never moved a workflow forward.

Only successful outcomes did.

## The kill -9 test

Then I automated it.

Worker A claims ten jobs with a two-second lease and starts ten-second handlers.

The test kills it with `SIGKILL`.

All ten jobs remain stuck in `running`, proving the worker died while holding work.

Worker B starts.

It notices the expired leases, reclaims the jobs, and completes them.

The test then verifies:

- every workflow reached completion
- every ticket job needed exactly two attempts
- every workflow history recorded each step completion exactly once

![](https://cdn.hashnode.com/uploads/covers/6af8b738c3f8a2657ed92c78/c026c9ee-9427-4d34-804c-482a823e6fa3.png align="center")

This is the test I care about most because it exercises almost every guarantee in the system at once:

- transactional persistence
- leases
- abandoned-job recovery
- fencing tokens
- exactly-once workflow progression

If any one of those breaks, the test fails.

There was one trap while writing it.

The usual way to run TypeScript, `tsx`, starts your script as a child process.

Kill `tsx` and you only kill the wrapper. The real worker keeps running, finishes the jobs, and the test passes without anything actually dying.

A recovery test that can't fail proves nothing.

The worker now launches with:

```bash
node --import tsx
```

which keeps everything inside a single process that the signal actually reaches.

## What I got wrong along the way

Two mistakes taught me more than the successes.

### A safeguard that wasn't doing the work

Tenant isolation is enforced through PostgreSQL Row-Level Security.

To prove the write check mattered, I deleted it and expected a test to fail.

It didn't.

PostgreSQL quietly applies a policy's read condition to writes when no explicit write condition exists, so the protection was still there.

Only replacing it with an explicit "allow anything" check made the test fail.

The lesson stuck:

**If you haven't seen a safeguard fail, you don't know it's the one protecting you.**

### A test that ran code TypeScript would have rejected

One test failed with:

```plaintext
withTenant is not a function
```

I had imported it using `import type`, which TypeScript removes at runtime.

The test runner stripped types without performing type checking, so the mistake survived until execution.

Now every test run sits beside a type-check run, and CI executes both.

## The honest limits

The database side of Canonik is exactly-once.

The outside world isn't.

If a worker creates a ticket in another system and dies before recording success, the job is reclaimed and the ticket may be created again.

That's at-least-once execution, a known limit for now.

It's also why every handler is still fake.

The fix is already planned:

- attach an idempotency key to every external call
- verify outcomes before retrying

Only then can external effects approach the guarantees the database already has.

## What's next

So far Canonik is an engine you drive from a command line.

Next comes the first real application:

- an API
- organization sign-in
- a web UI for starting and completing work
- two demo organizations proving tenant isolation end to end

If you build systems where work crosses people, agents, and machines, I'd love to compare notes.

And if you've ever had a worker die mid-job in production, I'd love to hear what it cost you.
