Production runbook

Cloudflare release order

Production runs on Cloudflare Containers with PostgreSQL. Apply the checked-in GTM migrations before deploying application code. In the production profile, the runtime verifies the exact checksummed migration ledger and never migrates on startup.

GTM_MIGRATION_DATABASE_URL=<production-migrator-url> \
GTM_POSTGRES_MIGRATION_APPROVED=1 \
GTM_BRIDGE_CONTROLLED_MIGRATION=1 \
pnpm --filter @repo/gtm postgres:migrate

GTM_DATABASE_URL=<production-runtime-url> \
GTM_POSTGRES_PREFLIGHT_APPROVED=1 \
pnpm --filter @repo/gtm postgres:runtime-preflight
pnpm cloudflare:ship:web

Use the dedicated migrator identity for the first command and the runtime identity for the preflight. Never place the migrator credential in the application Container. A release is not ready until /__fabric/health/ready returns 200. Process startup verifies the GTM migration ledger; recurring readiness verifies live database connectivity as well as Temporal dispatch.

pnpm cloudflare:ship:web accounts for Cloudflare Containers' non-transactional rollout order. Cloudflare activates the edge Worker before it finishes applying a changed image, so the release uses the immediate rollout mode and then waits for both the exact /api/health/image-revision Git SHA and application readiness. It repeats the exact deploy only after the first gate passes and verifies the stabilized release again. Each gate is bounded to two minutes and fails closed; never remove it to make a rollout appear complete. The application also retries one transient Next.js page-data failure in the browser before showing the manual reload action. This is recovery protection, not a substitute for the release gates.

If the first gate fails, stop and inspect the currently active deployment and Container rollout. Do not keep rerunning deploys: the previous Worker version remains the rollback target, and repeated activation can extend the mixed-revision window. Roll back the Worker when production is degraded, then diagnose image startup, database readiness, and revision evidence before attempting a new release.

If authenticated organization pages work but every GTM screen returns an internal server error, query max(version) and count(*) from gtm.schema_migrations. A ledger behind the source tree means the release order was reversed. Apply the missing checked-in migrations through the controlled command, run the runtime preflight, and reload the exact failing screen. Do not relax migration verification or add schema changes to application startup.

The Databricks procedure below is retained only for the explicit rollback profile.

Pre-launch gates

These four commands are the platform operator's gate before a controlled live launch. They are not part of a product user's path — nothing in Run your first campaign requires a terminal.

pnpm --filter @repo/gtm databricks:doctor
pnpm --filter @repo/gtm staging:preflight
pnpm --filter @repo/gtm go-live-check
pnpm --filter @repo/gtm campaign-readiness

Treat a failure as configuration work rather than something to bypass. The gates exist to stop an apparently healthy UI from hiding a missing model, database, inbox, webhook, or vendor connection.

Release checklist

1. Run relevant unit tests and TypeScript checks, then finish the production build before entering drain mode.
2. Run pnpm gtm:cost-guard. It must show only fabric-gtm-brain-prod as cost-active; the canary remains stopped.
3. Assign a new TEMPORAL_WORKER_BUILD_ID in the source snapshot. Never reuse the prior production Build ID.
4. Enter the durable deployment drain with that Build ID as the release ID. The drain rejects every API, agent-ingress, and script-initiated workflow start and pauses the three Temporal schedules. Reads, queries, approval signals, and recovery remain available.
5. Run the bounded quiescence check and wait for two consecutive observations with no pending Temporal activities or Nexus operations.
6. Validate and deploy the production Asset Bundle from the clean committed source tree.
7. Run the gtm_brain bundle target to create the snapshot App deployment, then wait for SUCCEEDED. databricks bundle deploy syncs resources and source but does not create the App deployment. databricks apps start is not a substitute: it can start the conservative source manifest without the bundle target's exact release environment overrides.
8. Wait for the new worker version to register, then promote that exact Build ID as the current version of the gtm-brain-prod Temporal Worker Deployment. The worker reports WORKER_UNDISPATCHABLE in the App logs until this promotion lands; that is expected, and the worker keeps polling instead of exiting. Readiness stays degraded until the promoted build is the one being polled.
9. Confirm App state is RUNNING and compute state is ACTIVE.
10. Promote the newly registered Temporal Worker Deployment build and run the verify-current gate. A Databricks App deployment is incomplete until gtm-brain-prod routes to that exact build ID.
11. Inspect logs for Prisma readiness, Next.js readiness, Temporal connection, schedule registration, and worker state RUNNING.
12. Run the live Databricks readiness doctor. The current gate is pnpm --filter @repo/gtm databricks:doctor; do not restore the removed @repo/databricks package.
13. If semantic Knowledge retrieval is in the release, verify the continuous AI Search index, exact App SELECT binding, tenant-filtered query, stale-hash rejection, and citation fields before setting knowledge_ai_search_enabled=1.
14. Clear the drain with the same release ID. Only schedules that were active before the deployment are resumed.
15. Run the cost guard again, then run the event projection job and query the Delta target.

worker_build_id has no default. A rollout must pass it explicitly, because a defaulted Build ID lets a deploy that forgot to bump succeed and ship new code under an identity Temporal has already promoted:

databricks bundle deploy -t gtm_brain \
  --var worker_build_id="gtm-prod-<change>-$(date -u +%Y%m%dT%H%M%SZ)"

The same resolved value must be committed to app.yaml; Databricks Apps does not expand variables in env values. pnpm run deployment-safety:test enforces both rules.

Detached canary snapshots

The committed App manifests use GTM_ENABLE_V2_LIVE_DELIVERY=0 and GTM_LIVE_DELIVERY_MODE=off. Databricks Apps does not pass ordinary operator-shell exports into the running process, so shell exports cannot activate or disable delivery. A release that enables governed delivery requires a separate reviewed source snapshot in addition to the narrow tenant authority created by a Campaigns launch.

For the accepted bounded canary, create a detached git archive snapshot and run pnpm gtm:live-canary-config against that directory. Activation requires the exact tenant, immutable campaign ID and version, an explicit maxSendsPerDay from 1 through 50, a fresh production Temporal Build ID, and an expiration between five minutes and four hours away. The renderer inserts the strict tenant/campaign/version/ceiling/natural-person binding into both App manifests and refuses to modify a Git worktree. Deploy it through the full drain, bundle, worker-promotion, readiness, and doctor transaction above.

After the rehearsal evidence is complete, create another detached archive with another unique Build ID and render --mode kill. Deploy and verify that snapshot through the same transaction. Do not treat an expired canary or a local shell export as rollback evidence; the deployed source must show delivery disabled and no canary binding before normal schedules resume.

Diagnosing "everything hangs"

If interactive actions never settle, distinguish a paused release from an undispatchable one before touching anything:

curl -sS https://<app-host>/api/health/ready | jq

On the Databricks rollback profile, governedExecution.reason names the blocked task queue and, when versioning is the cause, the Build ID new work routes to versus the Build IDs actually polling. The Cloudflare edge deliberately returns only status and ready; the detailed response remains inside the application Container and may be read only through an owner-approved container-internal diagnostic session. A degraded readiness response with a RUNNING App is the signature of a promotion that never completed — promote the Build ID a live worker is polling and re-run the verify-current gate. Do not restart the App: a restart destroys the worker whose promotion you are waiting on.

Inventory anything left in flight, including the workflows behind interactive writes:

pnpm --filter @repo/gtm temporal:recover -- --action list --workflow-family effect
pnpm --filter @repo/gtm temporal:recover -- --action list --workflow-family inference

Do not clear drain mode automatically after a failed deployment. Repair or roll back the one production App, verify its exact worker Build ID and readiness, and only then run exit. This fail-closed behavior prevents queued user actions or a schedule from starting against an unverified snapshot.

Production Databricks Apps set TEMPORAL_WORKER_DEFAULT_VERSIONING_BEHAVIOR=AUTO_UPGRADE. Databricks keeps one active App snapshot, so replay-tested workflows—including approval-parked campaign passes—move to the current worker on their next task rather than depending on a terminated snapshot to keep polling an older pinned build. Each rollout still uses a unique Build ID and promotes that worker only after both task queues are healthy.

If a workflow created before this policy remains pinned to a drained build, migrate only that known replay-compatible execution:

pnpm --filter @repo/gtm temporal:deployment -- \
  --operation workflow-auto-upgrade \
  --workflow-id <workflow-id>

The command records the override in Temporal workflow history. It does not approve, reject, publish, or deliver campaign content.

Approval passes created before the Platform Host checkpoint contract are replay-compatible. On resolution or expiry, the worker creates the missing idempotent gtm.authorize_draft_delivery_v2 checkpoint through the tenant's governed action runtime, records its compact invocation ID, and then resumes that exact checkpoint. New passes park the checkpoint before waiting and never take this compatibility path.

An upgrade from a snapshot that predates the deployment-control workflow is a one-time bootstrap exception. Pause all three GTM schedules, confirm that no business workflow is running, and stop the single App before replacing it. If the pre-export worker created an uninitialized failed gtm-deployment-control execution, terminate only that exact execution after preserving its history. The replacement worker creates a new run with the same durable workflow ID. Promote its unique Build ID, verify that temporal:deployment-drain status resolves the active run, and keep the schedules paused until every normal readiness gate passes.

Useful commands

RELEASE_ID="$TEMPORAL_WORKER_BUILD_ID"
pnpm gtm:cost-guard
pnpm --filter @repo/gtm temporal:deployment-drain \
  enter --release-id "$RELEASE_ID" --reason "production deployment"
pnpm --filter @repo/gtm temporal:deployment-drain \
  wait --timeout-seconds 900 --poll-seconds 5
pnpm --filter @repo/gtm temporal:deployment-drain status

databricks apps get fabric-gtm-brain-prod -p fabric-harness
databricks apps logs fabric-gtm-brain-prod --tail-lines 150 -p fabric-harness

cd deploy/databricks
databricks bundle deploy -t prod -p fabric-harness --var="..."
databricks bundle run gtm_brain -t prod -p fabric-harness --var="..."

pnpm --filter @repo/gtm databricks:doctor

pnpm --filter @repo/gtm temporal:deployment -- \
  --operation current \
  --deployment gtm-brain-prod \
  --build-id "$TEMPORAL_WORKER_BUILD_ID"
pnpm --filter @repo/gtm temporal:deployment-drain \
  exit --release-id "$RELEASE_ID"
pnpm gtm:cost-guard
databricks bundle run project_gtm_events -t prod -p fabric-harness --var="..."

pnpm --filter @repo/gtm databricks:knowledge-search

The AI Search command without --apply is a read-only gate. It fails when the source table, endpoint, or index is absent. Provisioning is a separate explicit deployment action and is never performed by App startup. A failed Knowledge indexing adapter does not roll back or erase the already committed Lakebase record; keep the feature dark, inspect the Platform invocation, correct the Databricks resource or grant, and recover the same idempotent adapter attempt. Do not rewrite the Knowledge event or use direct Delta SQL as an alternate authoring path.

To repair one missing or stale projection, inspect the governed maintenance plan first:

pnpm --filter @repo/gtm knowledge:reindex -- \
  --tenant <organization-id> \
  --knowledge-id <knowledge-id>

Add --apply only after verifying the tenant and canonical revision. For an already-deleted Knowledge ID, also supply --occurred-at <ISO-8601>; the action revalidates that the ID is still absent before removing its derived chunks. A revision race fails closed and must be retried against the new canonical revision.

Healthy startup markers

The database is already in sync with the Prisma schema.
Next.js ... Ready
gtm-agent-worker starting
schedule gtm-daily-pass updated to dispatcher
Worker state changed ... RUNNING
gtmDeploymentControlWorkflow

The App must launch pnpm --filter @repo/gtm-worker start. The former package-local GTM worker does not load Harness reviewer/generator activities and is no longer a supported entrypoint.

The worker uses a 45-second Temporal shutdown grace and a 55-second force bound. The App supervisor stays alive for up to 65 seconds while both the web child and worker child exit, then force-kills only the remaining processes. This ordering stops new HTTP work, stops Temporal polling, and gives heartbeat-aware activities time to report completion before the single Databricks App snapshot is replaced.

The V2 judge uses schema-pinned JSON text extraction rather than provider-specific structured-output tools. This is intentional: Unity AI Gateway models can return valid JSON text without implementing the AI SDK object-generation protocol. The worker accepts only one JSON object (optionally fenced), validates the exact bounded schema locally, retries within the activity budget, and otherwise abstains fail-closed. An explicit model abstention is canonicalized only in the restrictive direction to icpFit=abstain, triggerStrength=abstain, and outreachPlay=skip; it still requires a bounded reason. The adapter accepts at most one fenced JSON candidate and deterministically trims only the three bounded narrative fields before exact validation. It never repairs categories, identifiers, evidence, play, or offer bindings in the permissive direction.

Signal-sweep deadline safety

New V2 signal sweeps have a 10-minute start-to-close timeout. Provider acquisition is still cancelled after at most 105 seconds; the remaining envelope is reserved for governed per-command staging, invocation completion, and a clean activity return. The production adapters propagate the same abort signal through Apollo, Instantly, and Monid HTTP calls:

• Apollo requests time out after 15 seconds and independent organization job-posting reads run with stable-order concurrency capped at eight.
• Instantly lead reads retain their 10-second request timeout.
• Independent curated Monid bindings inspect and execute concurrently while preserving catalog order in diagnostics; every request and poll observes the activity cancellation signal.

No signal command may be staged after the activity enters its final 10-second safety window. A provider budget failure therefore completes as an explicit activity error well before Temporal's timeout and cannot continue ingesting signals after the workflow has closed. The longer activity envelope is behind the replay-safe gtm-v2-bounded-signal-ingestion-window-v1 patch; historical workflows retain their original three-minute command.

When investigating a failed sweep, compare the provider-effect failure timestamp with the Temporal activity event. There must be no later stageV2Signal or gtm.ingest_signal_v2 invocation from that attempt. Correct the provider latency or scope issue, then retry through the existing governed campaign/operator recovery path with its stable request identity. Do not create random request IDs to force a second mutation. Keep the production mode governed; the final boundary rejects a launch while the campaign is dry-run or while the ledger has unresolved reconciliation work.

Governed campaign launch

The ordinary GTM launch is completed inside Campaigns. After the safety check, prepare the final version, complete independent review, lock the exact campaign version, and select Approve campaign. The launch card then presents one action: Launch campaign.

In Campaigns, select Prepare final campaign on the current Launch row of the completed safety test. The governed clone creates exactly source version + 1 as dryRun: false, status: draft, copies only the exact artifact identities and revisions in the safe-test manifest, and removes the old execution manifest. A campaign-lineage lock refuses competing preparations. Run background review, approve the copied artifacts, select Confirm campaign details on its journey row, and then select Approve campaign on the current Launch row. Do not repurpose the existing dry-run version or reuse its human approvals for live delivery.

Selecting Launch campaign invokes the canonical gtm.request_campaign_pass action. For a live campaign the API first requires organization-admin membership, then the action requires a natural-person actor and the existing campaign.operator capability proof. It writes an idempotent authorization for the exact tenant, campaign version, and execution-manifest hash. The authorization expires after four hours, permits no more than one send, and requires natural-person approval of the generated message. The UI cannot mint or relax that authority by itself.

The governed launch capability requires a separately reviewed release with:

GTM_ENABLE_V2_LIVE_DELIVERY=1
GTM_LIVE_DELIVERY_MODE=governed

The committed migration release remains at 0 / off; the global enable flag remains the operational kill switch. The policy evaluator, Temporal decision activity, final gtm.send_approved_outreach_v2 Platform action, and provider boundary all reload and check the authorization. They also require the exact active non-dry campaign manifest and halt if a provider-accepted or unknown delivery attempt needs governed reconciliation.

After launch, wait for Review first email to appear. Check the qualified recipient and personalized message, then make the explicit approval decision. Nothing reaches Instantly before that decision. The normal product workflow does not require a GTM user to copy identifiers, edit deployment configuration, or contact a platform operator.

Detached canary recovery mode

The earlier strict canary remains available only for self-owned rehearsals and operator-led recovery. In that mode set GTM_LIVE_DELIVERY_MODE=canary and provide GTM_LIVE_DELIVERY_CANARY_JSON with the exact tenant, campaign ID, campaign version, send limit, issuance, expiry of no more than four hours, and requireHumanApproval: true. Unknown fields, future issuance, expiry, or scope drift fail closed.

For the self-addressed rehearsal only, render the canary with --rehearsal-recipient you@rehearsal.example. This adds an exact selfOwnedRehearsal recipient and literal operator.rehearsal source binding without changing the active prospect ICP. Qualification and both delivery boundaries fail closed on recipient, source, campaign, or expiry drift. After the rehearsal arrives and its reply is ingested, deploy disabled/off, suppress the rehearsal domain, and render the Apollo-qualified prospect canary without that flag. For the first real prospect send shown in the Campaigns launch card, omit --rehearsal-recipient.

For detached canary recovery, run the exact preflight:

pnpm --filter @repo/gtm go-live-check -- --tenant <tenant-id>

It must report the detached canary binding, active live campaign, and zero unresolved delivery outcomes. Do not grant the internal delivery action to an external agent; it is deliberately absent from the external catalog. Every delivered draft must be decided by an authorized human.

After the single send, verify the provider request, immutable recipient, atomic touch/cooldown row, GtmOutreachSent, reply webhook, invocation/event projection parity, and zero unresolved outcomes. After a detached recovery, return the deployment to the reviewed delivery posture for that release; the committed migration baseline remains GTM_ENABLE_V2_LIVE_DELIVERY=0 and GTM_LIVE_DELIVERY_MODE=off. Never change a command key or repeat a send to escape an ambiguous provider result; use the governed reconciliation action.

If the tenant-scoped production drafter returns malformed output or changes an exact approved proof/trigger source, the action uses the deterministic safe-draft template and runs the same lexical claim validation again. This fallback is available only through the trusted production composition root; injected test and external generators continue to fail closed.

Production startup fails when TEMPORAL_WORKER_BUILD_ID is absent. A Build ID identifies immutable worker code, so reusing one across App snapshots can route new activities to a stale rolling-deployment process even when the new source snapshot is healthy. The Build ID and source must be released atomically: sync a clean committed tree containing the new ID, deploy that snapshot, and only then promote the newly registered worker version. Promote it only after startup checks pass; keep the prior version available for pinned historical workflows until drain and replay verification are complete.

For container verification, ./aspire.sh publish builds the worker from apps/gtm-worker/Dockerfile using the repository root as its workspace context. The image therefore uses the same composition root as Aspire and Databricks deployments. Do not restore the removed packages/gtm/Dockerfile.manual path; it omits the governed Harness activities.

The current GTM schema marker is migration 53. Migrations 22–53 add delegated agent draft decisions, governed record identity and relationship commands, communication sync, tenant custom objects, the preview-first record import/export/reversal commands, durable full-workspace import jobs, governed cutover-certification commands, first-party inbound lead CRM, durable campaign-pass admission, bounded staged-command retention, delivery authorization and reconciliation, governed model/provider adapter effects, access-enforcement configuration and audit evidence, active staged-command execution leases, explicit ambiguous provider outcomes, governed tenant skill-run requests, governed exact agent-version certification and revocation, tenant-first keyset indexes for bounded agent control-plane reads, the commercial experiment control plane, immutable activation/assignment evidence, campaign treatment/draft-consumption bindings, atomic commercial delivery exposure, governed audit_24m exposure retention with privacy-safe tombstones, the exact canonical V2 staged-action constraint, durable external-agent enrollment and one-time credential claim, atomic campaign live-promotion staging, the external decision boundary (bindings, approvals, outbox, receipts), and durable draft-decision command admission staging. Migration 43 adds the operational event, invocation, subject, and operation-status read indexes. Migration 49 synchronizes the private staged-command allowlist with the canonical action registry. Migrations 50–53 add external-agent enrollment tables, campaign live-promotion staging, the external decision boundary, and draft-decision admission staging. The migrations only expand checksummed schemas and private staged-command allowlists; they do not copy credentials, record bodies, skill inputs, knowledge, or outputs into Temporal history. Apply them as the deployment owner before starting the new App/worker image, then verify the exact catalog signature and readiness marker before serving traffic.

Migration 43 builds six indexes over the event and invocation ledgers with ordinary transactional CREATE INDEX. Schedule it in a write-maintenance window after capturing a Lakebase branch/backup. The migration sets a five-second lock timeout so it fails instead of waiting behind long-running transactions, and a fifteen-minute statement timeout so an unexpectedly large build does not run without bound. Measure duration on a production-size staging branch before the production window. If it times out, leave the application on migration 42, investigate ledger size and blockers, and rerun the same checksummed migration later; do not manually mark migration 43 as applied or create differently named indexes.

As a lower-bound engineering check, PostgreSQL 17 built all six indexes in about two seconds over 500,000 synthetic events and 100,000 synthetic invocations on a local development host. A deliberately conflicting table lock failed the migration at the configured five-second lock timeout. Those numbers are not a production estimate: record the Lakebase branch duration, ledger sizes, and lock observations in the release evidence for the exact deployment SHA.

DATABRICKS_HOST=<workspace-url> \
DATABRICKS_CONFIG_PROFILE=<admin-profile> \
PGUSER=<workspace-user> \
GTM_LAKEBASE_HOST=<endpoint-host> \
GTM_LAKEBASE_DATABASE=gtm \
GTM_LAKEBASE_ENDPOINT_NAME=projects/<project>/branches/production/endpoints/primary \
pnpm --filter @repo/gtm databricks:migrate

The command uses a short-lived Lakebase OAuth credential obtained by the Harness Databricks runtime. It never prints or persists the credential-bearing database URL.

Operate the public agent ingress

https://gtm.fabric.pro/api/agents/gtm/<organization-id>/mcp is the only production endpoint documented for Grok Bot, Hermes, and other external agents. The docs Worker is a narrow HTTPS proxy to the Cloudflare application: it forwards Authorization: Bearer <registration-key> only on the exact agent gateway path, strips caller authorization elsewhere, preserves the compatibility x-gtm-agent-key, and lets the application derive the actor, tenant, action grant, and resource scope from that registration. It has no GTM registration, database credential, model key, or tenant authority of its own.

Keep the docs Worker's GTM_APP_INTERNAL service binding pointed at fabric-gtm-app. This avoids a public custom-domain loop and adds no caller credential. The proxy must never add a cloud-provider credential or replay a rejected request. The caller's stable GTM idempotency key is the sole retry identity. A probe without a registration key should return 401 with {"error":"invalid agent key"}; 502 means the proxy cannot reach the application and should alert. Do not work around either response by distributing application, database, or Cloudflare credentials.

Review failed_unknown delivery attempts after each initial batch. The state is deliberately terminal: an ambiguous provider response requires human reconciliation and must not be retried automatically.

Recover a record import job

Open Accounts, Contacts, Opportunities, or Meetings, select Import / Export, and inspect the job list. Each page checkpoint is an immutable Platform event and contains only offsets and counts. Worker restarts resume from the latest exact checkpoint, and long imports continue as new after 50 pages. Do not manually edit the offset or replay provider writes. Lightfield remains read-only in this release; an import conflict requires governed merge review or a corrected mapping and a new idempotency key.

The protected API exposes the same projection through GET /gtm/records/imports/jobs. A failed job keeps its last checkpoint and a bounded failure code. Diagnose the provider scope or mapping issue, then start a new job. Never delete the prior events: they are the audit and reconciliation evidence.

After completed imports exist for all four supported resources, open the Certification tab and run the full report. The protected API equivalents are GET /gtm/records/cutover/reports and POST /gtm/records/cutover/lightfield/reports. A completed report must account for every source record and will remain not_certified until Lightfield file bodies have their own approved storage, retention, and deletion path. Treat that state as an explicit migration boundary, not as an operator warning to ignore. The report does not authorize outbound Lightfield writes or campaign delivery.

Agent certification without live sending

Keep GTM_ENABLE_V2_LIVE_DELIVERY=0 and do not configure an agent grant for publication or delivery. Then run:

pnpm --filter @repo/gtm-agent-contracts test
pnpm --filter @repo/gtm-agents test
pnpm --filter @repo/gtm-worker test
pnpm --filter @repo/gtm-worker type-check
pnpm --filter @repo/gtm campaign-readiness
pnpm --filter @repo/gtm databricks:doctor

For a protected-workspace canary, create a disposable campaign brief, checkpoint it, register one campaign-scoped reviewer and generator, run both through Temporal, verify evaluation/revision/run/cost records in the UI and Lakebase, and archive the generated artifact. The cleanup assertion must fail if any disposable registration remains enabled, any working copy remains, or any disposable artifact is not archived. Immutable revisions, completed run evidence, action decisions, and cost records remain in the audit ledger by design. This certification creates GTM records but no prospect send and no Databricks infrastructure resource.

The bounded model path also has an opt-in protected-workspace test in agents/tests/live.e2e.test.ts. It executes all three built-in agent definitions through the native Databricks SDK, validates their typed results, requires non-zero model usage, and evaluates the outputs against the frozen Fabric Experiments contract dataset at a 1.0 release threshold. It does not publish, deliver, or create workspace infrastructure. Run it with GTM_AGENT_LIVE_TEST=1, set GTM_AGENT_EVIDENCE_PATH=reports/agent-quality-scorecard.json, and supply the standard DATABRICKS_HOST, DATABRICKS_CONFIG_PROFILE, DATABRICKS_MODEL, and DATABRICKS_WAREHOUSE_ID settings. The scorecard contains hashes and aggregate metrics, never generated text.

For CI, store the native experiment ID as the protected DATABRICKS_MLFLOW_EXPERIMENT_ID repository secret. Grant the dedicated certification service principal CAN_EDIT on that experiment only; it does not need SQL entitlement, a GTM registration, or delivery authority. The workflow derives a stable request ID from the GitHub run and attempt, publishes the compact scorecard with:

pnpm --filter @repo/gtm-agents quality:publish-mlflow

For the production Databricks App, bind DATABRICKS_MLFLOW_EXPERIMENT_ID from the declared gtm-mlflow-experiment App resource. The worker uses the App service identity to verify the run at the Platform adapter boundary. Do not add DATABRICKS_TOKEN to the App, reuse the CI service principal, or give Databricks credentials to Hermes or another external agent.

Successful output is a managed-MLflow run URL plus reports/agent-quality-mlflow-link.json. Verify that the run is FINISHED, its scorecard SHA-256 and source revision match the CI artifacts, and both release gates equal 1. Repeating the command with the same request ID and scorecard must return the same run. Do not retry a changed scorecard under the same request ID. The publisher does not upload generated text, prompts, artifacts, tenant records, or credentials.

For a UI canary, use Workspace → Operations → Agent operations → Install built-in agents. Candidates install as disabled. Open administrator-only Release assurance, choose Verify evaluation for the current registered version, and paste the runUrl or runId from agent-quality-mlflow-link.json. The server derives the remaining protected evidence and confirms the worker-created attestation is current before you configure one registration as enabled with a disposable campaign scope. The form never accepts a Databricks credential or asks the operator to transcribe hashes. Verify that:

1. enabling before certification fails and creates no agent work;
2. the active card shows the immutable version's model, cost ceiling, and current exact certification;
3. the deterministic Temporal run starts, and its first activity records queued before any model work;
4. the Temporal run records the same agentVersionId and registration-bound principal;
5. the run detail panel lists only safe governed-action metadata, never parameters or artifact bodies;
6. reviewer output appears as one hash-bound evaluation, or generator output as one proposed revision;
7. retrying the staged mutation does not create a duplicate evaluation or revision;
8. revoking the certification blocks principal minting and a resumed worker execution;
9. the rolling outcome view suppresses all exact counts until the relevant cohort reaches ten;
10. no publication or delivery action was proposed; and
11. cleanup archives disposable artifacts and changes the registration to disabled.

Also certify one disposable dry-run campaign operator request. Bind independently evaluated and approved brief, audience, sequence, and reviewer-policy revisions, active content policy, and verified proof assets; activate that exact campaign version; start it with a stable idempotency key; and verify the Platform request event precedes the child pass. No delivery is authorized by this canary.

Do not delete immutable versions or run evidence. They are retained audit records. Re-running built-in installation is a safe drift check: it adds missing resources but preserves existing registrations.

Authentication release controls

Google and GitHub social OAuth are disabled by default and in both Databricks application manifests with NEXT_PUBLIC_SOCIAL_OAUTH_ENABLED=0. The login and signup surfaces do not display those providers, Better Auth does not register them, and their absent credentials must not produce startup warnings. Password and passkey login remain available. Magic links, verification messages, and password-reset email require a valid transactional mail provider.

Re-enabling social OAuth is a reviewed release operation, not a secret-only configuration change. Set the public flag to 1 at build and runtime, supply all four Google/GitHub client values, verify canonical callbacks and trusted origins, run the auth and organization-isolation checks, and deploy a new exact SHA. Startup fails closed if the flag is enabled without either complete credential pair.

Temporal searchability and bounded recovery

Before enabling V2 starts or workers, register and verify namespace attributes:

pnpm --filter @repo/gtm temporal:search-attributes
# Production/cloud: run this explicitly with production TEMPORAL_ADDRESS/NAMESPACE credentials.
# Every API process then remotely lists and verifies all six attributes before any V2 start.
# Environment confirmation alone is not accepted. Localhost without TLS/API key may auto-register
# for Aspire/local development; cloud endpoints are never auto-registered.

Registered attributes are gtmContractVersion (Int), gtmTenantId, gtmOperation, gtmEntityId, and gtmStatus (Keyword), plus gtmWaitingSince (Datetime). Values must remain compact IDs/statuses; never put message, approval, evidence, actor, or secret bodies in memo/search attributes or operator audit output.

Recovery is dry-run and bounded by default. Always select with an explicit tenant, operation, status, or ISO older-than filter and a small result/page cap. Every mutation also requires --workflow-family pass|action|reply, which adds an exact V1/V2 workflow-type query filter. before adding explicit confirmation. Failed workflows may be reset only at an exact inspected workflow-task-finished event ID; approval-parked pass workflows must be resolved through their compact decision signal or explicitly cancelled/terminated with a ticket/reason, never blindly restarted. Reset, cancel, terminate, and signal actions require --confirm --reason; reset also requires --reset-event-id. V1 executions can lack custom attributes: inventory them separately by V1 workflow type/task queue and treat missing metadata as V1 fallback, not as proof of absence.

For stuck-workflow reporting, query gtmStatus = 'waiting_approval' with gtmWaitingSince age, and query failed command workflow types/status separately. Alerts should cover approval age, failed activities/workflow tasks, queue backlog, and migration mismatch. Audit JSON contains workflow/run IDs, requested action, dry-run flag, and outcome only—never payloads.

Operational ownership

Schema migrations run as a deployment owner. The App identity validates the migration marker and uses scoped table privileges at request time; it does not rerun owner-level DDL on every API call. This separation is required for a least-privilege production Lakebase binding.