Routines: let a Bot work on a schedule with nobody watching

[davidmckayv]

Raised by Mark alongside #192. Recording it; not scheduled.

The topic

Every run today starts because a person typed something. A routine is a Bot doing its job on a schedule with nobody in the room: check the overnight exceptions at 07:00, summarise what changed in the document library every Friday, sweep the open items each morning and post what needs a human.

The result lands in the Bot's own history, so a person reads it when they arrive rather than having to ask for it.

This is the feature where governance stops being paperwork. An interactive run has somebody watching who will notice a wrong tool call. An unattended one does not.

Horizontal from the first commit, not later

OpenBot is a template. Somebody will deploy it at a size where one box is not the answer, and the shape of this feature is decided by whether that was assumed on day one. Retrofitting a scheduler from single-process to multi-replica is a rewrite, because the single-process version is allowed to keep the answer in memory and the multi-replica one never is.

No leader, no in-process timer as the source of truth. The obvious build is setInterval in the server, and it is wrong here. We already do exactly that for the audit-retention sweep (server/src/audit-retention.ts), which is safe only because deleting old rows twice is the same as deleting them once. A routine fired by every replica is the same run billed N times, N times the tool calls, N posts in the channel.

The schedule and the ledger live in Postgres. Two tables, not one: what is supposed to run, and what did. They answer different questions and they change at different rates.

Work is claimed, not assigned. Any replica takes the next due routine with select ... for update skip locked, or a lease column with an expiry. No coordinator to be a single point of failure, and adding a replica adds throughput rather than a conflict.

A lease expires, so a replica dying is recoverable. A claim that is not renewed is reclaimed after a grace period. Whatever picks it up must be able to tell "this never started" from "this started and we lost the process", because the second one may have already called tools and spent money. That state is worth a name of its own rather than being folded into failure.

Firing is idempotent on (routine, scheduled time). A routine due at 07:00 runs once even if three replicas wake and a lease is reclaimed mid-flight. Without this, every recovery path is a duplicate-run path.

Nothing is held in a process that a person needs. A routine created on one replica is visible from every other on the next request, the way grants already are.

Governance, which is the point rather than the trim

A routine runs as a person, and the trail says nobody was watching. There is no live actor, so it must carry one: its owner. It sees what that person may see and no more. If the owner loses access to a connector, the routine loses it on the next run. An audit row for an unattended run has to be distinguishable from an interactive one, because "why did this Bot read that at 3am" is the question somebody will ask.

Same grants, same policy, same audit as any other run. No separate path. A routine is a run that nobody typed, not a run with fewer checks.

Caps, and they refuse rather than truncate. A cap on concurrent routine runs per deployment and per Bot. Unattended work is the easiest way to spend money without noticing, and a schedule that overruns its own interval will pile up on itself.

Failure stops rather than retries forever. Backoff, a cap on consecutive failures, then disabled with a reason a person can read. A routine failing every ten minutes for a week is a bill and a noisy trail and nobody notices either.

Delivery is recorded apart from the outcome. A run can succeed and its result reach nowhere — the channel is gone, the post failed. Collapsing those two into one status produces a trail that says a routine has been working fine while nobody has seen a result in a month, which is worse than one that says nothing.

A person can list, pause, and stop them. Including somebody else's, for an administrator. A routine somebody set up and forgot is the thing that outlives them.

The known constraint

A routine that needs a browser needs a computer, and computers are not horizontal yet — the supervisor gives a Bot a container, and that is per-host today. Routines that only call tools have no such problem and are most of the useful ones. Worth building the schedule so it does not assume a computer, and letting the computer-shaped ones wait for the fleet work rather than holding this back.

What we already have

• The schema opens with "bots, skills, routines, bot-to-bot handoff" (server/src/db/schema/coworker.ts). Routines were anticipated and never built.
• Postgres already carries everything that has to survive a replica, including channel activity, the policy and the snapshot store — the last of which exists precisely because holding it in one process broke across replicas.
• worker/ exists and does nothing but report status. If a routine runner should be its own process rather than the API server's, that is where it goes, and the claiming design means it can be more than one of them.
• Grants, policy and audit are already the path every tool call takes.

What is not decided

• Whether routines are authored in the product or shipped in the tenant package like skills now are. A template that ships useful routines demonstrates this on a clone; one that ships none needs somebody to think of it first.
• How a routine addresses its output: its own history, a channel, or a person's inbox.
• Whether a routine may use Bot-to-bot messaging: let a Bot hand work to another Bot #192 to bring in another Bot. Probably yes eventually, and the caps have to compose, because unattended plus fan-out is the expensive corner.

[zopeVaibhav]

Both this and #192 are marked "not scheduled", so this is a question about whether the shared piece underneath them is worth taking, not a claim on the issue.

Reading around it, the two issues do not need two builds. Both need one thing that does not exist yet: a way to start a run nobody typed. Today every run is driven by the browser — copilotkit.runAgent in app/src/components/channels/channel-chat.tsx:276. There is no worker, no queue and no leased work anywhere in server/src: skip locked and for update return nothing, and the only setInterval is the audit-retention sweep at audit-retention.ts:156, which this issue already cites as the thing not to copy.

So #192's "the message is a row, claimed the way any other queued work is" has nothing to sit on yet. The claimable-work runner is the whole of the overlap: a table of due work, a lease with an expiry, a claim that is idempotent, and a run that carries an owner rather than a live actor. This issue is that runner plus a schedule and the failure caps; #192 is that runner plus a tool and a target.

One thing worth knowing if the order is ever discussed: #192 opens by saying several Bots can be addressed in a channel with @ today. That is not true on main. A channel is pinned to one runtimeAgentId for the life of its thread, and channel-chat.tsx:379 says so directly — the mention is carried in the draft but nothing routes on it. routing/classify.ts picks the coworker at channel creation, not per message. So a second Bot's reply currently has nowhere to land, which is a change to the thread binding rather than to messaging. This issue has no such problem: a routine's result landing in the Bot's own history is exactly what one pinned thread already is.

Two questions:

1. Is the design here settled enough for somebody to take, or is it still being decided?
2. If it is taken, would you want the runner landing as its own change that both issues then sit on, rather than folded into whichever feature is built first?