Durable recovery and rollback

This runbook covers the optional customer-managed Databricks target. For the hosted Cloudflare deployment, use the Cloudflare release and PostgreSQL recovery procedures.

Production mutations are complete only after Temporal records them. Lakebase stores private/full artifacts; workflow history carries compact IDs. Restarting App compute therefore resumes work instead of recreating it.

Release sequence

1. Migrate an isolated staging Lakebase branch and verify migration checksums/catalog shape.
2. Register and remotely verify all Temporal search attributes.
3. deploy a V2 worker Build ID as compatible with the current set;
4. run generated V1/V2 histories plus the genuine pre-change V1 history through replay;
5. canary a dry Health & Governance pass and verify compact Delta lineage;
6. restart App compute while a workflow is parked, resolve it after restart, and confirm one decision;
7. rehearse rollback to the previous App deployment and worker compatibility set;
8. drain V1 only after inventory shows no unsafe running or approval-parked executions.

Databricks App startup validates the Prisma schema but never runs db push or performs DDL. Apply Prisma and GTM migrations as a separate deployment gate with the migration identity, then grant the App service principal runtime DML. For a cloned staging branch, use deploy/databricks/sql/grant-staging-runtime.sql with the canary PostgreSQL role.

Action-staging compatibility

Every canonical V2 action admitted through executeDurableAction() first persists its private parameters in gtm.staged_workspace_commands_v2; Temporal receives only the opaque staged-command ID. gtm.ingest_reply is the sole exception because it uses the dedicated gtm.staged_replies_v2 boundary.

Lakebase keeps that boundary fail-closed with the staged_workspace_commands_v2_action check constraint. Migration 49 synchronizes the constraint with the canonical GTM_ACTION_IDS registry, including provider/model recovery and reply-delivery reconciliation actions. The repository migration contract requires an exact match. Adding an action therefore requires a new additive migration and PostgreSQL coverage; never edit migration 49 or weaken the constraint to accept arbitrary strings.

The staged-command ID, lease timestamps, and workflow metadata stay outside the action's business parameters. Admission persists the exact supplied parameter object; it must not synthesize commandId, occurredAt, or another field for an action whose strict schema does not declare it. Actions that require those fields receive them from their caller and preserve them unchanged. The durable API contract test covers both sides of this rule.

Before deploying a build that adds an action:

1. Verify the new action is registered in the canonical Platform module and GTM_ACTION_IDS.
2. Add a new migration that expands the staged-action constraint using add, validate, and rename.
3. Run the migration and PostgreSQL conformance suites on an isolated Lakebase branch.
4. Apply the migration before the application or worker build that can request the action.
5. Canary the exact governed endpoint and confirm one staged command, one Platform invocation, and terminal audit evidence.

Operator commands

The namespace bootstrap requires a Temporal Cloud API key with namespace-administrator permission. Developer/runtime keys are intentionally insufficient. Run each registration once; the GTM worker performs a remote type-aware preflight and refuses to start when an attribute is missing or has the wrong type.

temporal operator search-attribute create --address "$TEMPORAL_ADDRESS" --namespace "$TEMPORAL_NAMESPACE" --api-key "$TEMPORAL_ADMIN_API_KEY" --tls --name gtmContractVersion --type Int
temporal operator search-attribute create --address "$TEMPORAL_ADDRESS" --namespace "$TEMPORAL_NAMESPACE" --api-key "$TEMPORAL_ADMIN_API_KEY" --tls --name gtmTenantId --type Keyword
temporal operator search-attribute create --address "$TEMPORAL_ADDRESS" --namespace "$TEMPORAL_NAMESPACE" --api-key "$TEMPORAL_ADMIN_API_KEY" --tls --name gtmOperation --type Keyword
temporal operator search-attribute create --address "$TEMPORAL_ADDRESS" --namespace "$TEMPORAL_NAMESPACE" --api-key "$TEMPORAL_ADMIN_API_KEY" --tls --name gtmEntityId --type Keyword
temporal operator search-attribute create --address "$TEMPORAL_ADDRESS" --namespace "$TEMPORAL_NAMESPACE" --api-key "$TEMPORAL_ADMIN_API_KEY" --tls --name gtmStatus --type Keyword
temporal operator search-attribute create --address "$TEMPORAL_ADDRESS" --namespace "$TEMPORAL_NAMESPACE" --api-key "$TEMPORAL_ADMIN_API_KEY" --tls --name gtmWaitingSince --type Datetime

# Register/verify through the application client after the namespace-admin step.
pnpm --filter @repo/gtm temporal:search-attributes
pnpm --filter @repo/gtm temporal:history:replay
pnpm --filter @repo/gtm temporal:recover -- --tenant <org> --status waiting_approval
pnpm --filter @repo/gtm campaign-readiness

After the canary worker has polled its isolated queues, inspect and promote its current Worker Deployment version. Never set a version current before the worker has started successfully.

pnpm --filter @repo/gtm temporal:deployment -- --operation describe --deployment gtm-brain-canary
pnpm --filter @repo/gtm temporal:deployment -- --operation current --deployment gtm-brain-canary --build-id <verified-build-id>

Recovery selection is bounded by tenant, workflow family, status/age, and result limits. Mutating recovery requires explicit confirmation and a ticket/reason. Sends never retry blindly. Approval resolution uses the compact V1/V2 signal appropriate to the discovered workflow contract.

Ambiguous provider outcomes

A network failure does not prove whether a provider accepted a request. GTM records that state as ambiguous and keeps the original tenant-bound input, hash, request identity, and capacity reservation. Do not create a replacement command or change its idempotency key.

• Reconcile a generic provider effect through the admin API POST /gtm/operations/provider-effects/reconcile. The governed gtm.reconcile_provider_effect_v2 action reuses the exact staged request and provider identity. Its provider-specific reconciler may look up the accepted request or perform an idempotent repair; it never invokes the original generic effect callback. Approval notifications maintain a durable, content-bound checkpoint per intended recipient and retry only recipients without completion evidence. Chat repair uses stable chat and message identities. Generic cleanup deletes only terminal effects; unresolved prepared and ambiguous effects remain visible until a governed resolution.
• When a provider has no lookup contract—for example a signal-acquisition endpoint that cannot search by the original request identity—an administrator may use POST /gtm/operations/provider-effects/resolve. The governed gtm.resolve_provider_effect_v2 action can record only definitive_failure or abandoned; it cannot claim success and does not call the provider. The command must bind the exact effect ID, kind, provider, immutable input hash, existing provider request IDs, reason code, and an operator evidence reference. The retained effect becomes terminal only after that evidence-bound action completes.
• Close an ambiguous model request through POST /gtm/operations/model-executions/resolve. The governed gtm.resolve_model_execution_v2 action permits only a definitive-failure resolution because a successful model body cannot be reconstructed safely without provider evidence.
• Reconcile Instantly delivery through gtm.reconcile_outreach_delivery_v2. A provider lookup uses the immutable campaign, recipient, and provider request binding; it never sends a replacement message.
• Reconcile an ambiguous reply—or a prepared attempt whose provider acknowledgement was not checkpointed—through POST /gtm/operations/reply-deliveries/reconcile. The governed gtm.reconcile_reply_delivery_v2 action requires the exact attempt, a caller-issued command identity, and operator evidence. Resolving sent requires the provider message ID; resolving not_sent closes the attempt as a definitive failure before a later governed send may be considered.

All operator endpoints require organization-admin access and a caller-issued idempotency key. Same key plus changed resolution parameters fails with conflict.

Private staged commands receive an active lease before Temporal starts and renew that lease when the activity consumes them. The scheduled bounded sweeper may remove expired abandoned or consumed staging, but never a currently leased command. Cleanup failure is observable and does not change the authoritative invocation result.

A terminal schema rejection is not an ambiguous provider outcome. First reconcile the original invocation and prove that no adapter attempt or provider request occurred. If its staging row is already consumed, do not edit the row, reopen the invocation, or reuse it with changed parameters. A subsequent operator retry is a separate explicit decision with a deterministic, audit-linked command identity; it must traverse the same governed action and adapter pipeline.

Canonical projection sequence gaps

Fabric Platform Host event sequences are monotonic identities, not a promise that every reserved number will be committed. A process can exit after reserving a sequence for a non-transactional lifecycle event and before inserting that event. The resulting permanent hole is valid canonical history and must not block later governed actions.

GTM folds the canonical Host ledger in sequence order and keeps two kinds of durable checkpoint in gtm.host_projection_checkpoints:

• one sparse high-water mark for each tenant, space, and projection; and
• one hashed subject checkpoint for each account whose state is projected.

Every domain event is still mirrored idempotently by its canonical event ID. A later event may advance past an absent sequence. If the missing event is subsequently committed, reconciliation mirrors it, but an older sequence cannot overwrite a newer account state. This preserves audit completeness and sequence-monotonic projections without inventing a synthetic event or editing the Platform ledger.

When diagnosing projection lag, compare the canonical fabric_platform.asset_events stream with the sparse high-water mark. Do not insert a placeholder event, renumber events, or update an account directly. Retry the original governed command after the current certified worker is deployed; reconciliation will fold every committed event and deterministically repair the projection.

Observability

Alert on workflow/activity failures, approval age, ambiguous provider/model effects, orphaned staging, task-queue backlog, migration drift, projection lag, and App restarts. Search attributes and telemetry must contain only tenant-safe IDs, statuses, categories, and versions—never message bodies, evidence quotes, reviewer notes, or credentials.