Deploy GTM Brain

Cloudflare is the TechFabric-hosted production target. Databricks remains available when a customer requires the application, data plane, and workspace identity inside its own Databricks boundary.

# Next.js application and app worker
pnpm cloudflare:ship:web

# GTM brain worker and agent runtime
pnpm cloudflare:ship:agents

cd deploy/databricks

databricks bundle validate -t prod -p fabric-harness \
  --var="sql_warehouse_id=<warehouse>,saas_url=<app-url>,mlflow_experiment_id=<experiment>"

databricks bundle deploy -t prod -p fabric-harness --auto-approve \
  --var="sql_warehouse_id=<warehouse>,saas_url=<app-url>,mlflow_experiment_id=<experiment>"

databricks apps deploy fabric-gtm-brain-prod -p fabric-harness \
  --mode SNAPSHOT \
  --source-code-path /Workspace/Users/<user>/.bundle/fabric-gtm-brain/prod/files

Databricks architecture reference

Databricks Apps can run a Node.js web application directly. GTM Brain uploads the complete pnpm workspace, lets Databricks build it in an isolated environment, and starts the Next.js server and Temporal worker as child processes of one supervised application command.

Deployment pipeline

The files that matter

Build versus runtime

This distinction is the most important part of deploying Next.js on Databricks Apps:

• Build time: Databricks installs the lockfile and runs next build. Lakebase resource variables may be visible, but endpoint credentials are deliberately not used during static page collection.
• Runtime: the App receives its workspace OAuth identity, explicit non-secret Lakebase coordinates, and non-database resource bindings. The startup script obtains a short-lived Lakebase token, creates a TLS-verified PostgreSQL URL, and starts the server.
• Public URL: Better Auth, CORS, webhook links, and notification links use the runtime App URL, preventing a build-time localhost fallback from becoming the production trusted origin.

The Docker build keeps Next.js standalone output. Databricks Apps uses the regular Next production server because the complete workspace tree is present on App compute.

Identity and secrets

The App never stores a long-lived Lakebase password. The bundle owns a least-privilege OAuth role for the App service principal; Databricks issues endpoint-scoped credentials, and the PostgreSQL pool refreshes them for new physical connections. Lakebase is not attached as an App postgres resource because that resource grants database CREATE; controlled migrations retain schema ownership while the runtime role receives DML only. Application secrets are read from the fabric-gtm-brain secret scope through named App resources. Vendor credentials entered in the UI are encrypted per organization before being stored in Lakebase.

Deployment commands

cd deploy/databricks

databricks bundle validate -t prod -p fabric-harness \
  --var="sql_warehouse_id=<warehouse>,saas_url=<app-url>,mlflow_experiment_id=<experiment>"

databricks bundle deploy -t prod -p fabric-harness --auto-approve \
  --var="sql_warehouse_id=<warehouse>,saas_url=<app-url>,mlflow_experiment_id=<experiment>"

databricks apps deploy fabric-gtm-brain-prod -p fabric-harness \
  --mode SNAPSHOT \
  --source-code-path /Workspace/Users/<user>/.bundle/fabric-gtm-brain/prod/files

Snapshot mode keeps the currently active deployment online while a replacement snapshot is imported and built.

Provision Knowledge AI Search

Semantic Knowledge retrieval is dark by default. Provision it before deploying a bundle that binds the index:

databricks bundle run bootstrap_gtm_analytics -t prod -p fabric-harness --var="..."

GTM_KNOWLEDGE_AI_SEARCH_SOURCE_TABLE=gtm_brain_analytics.gtm.knowledge_chunks \
GTM_KNOWLEDGE_AI_SEARCH_INDEX=gtm_brain_analytics.gtm.knowledge_chunks_index \
GTM_KNOWLEDGE_AI_SEARCH_ENDPOINT=fabric-gtm-brain-knowledge-search \
GTM_KNOWLEDGE_AI_SEARCH_EMBEDDING_MODEL=databricks-gte-large-en \
pnpm --filter @repo/gtm databricks:knowledge-search -- --apply

The command uses the deployment identity, verifies the CDF-enabled Delta source table, creates one standard endpoint and one managed-embedding continuous Delta Sync index when absent, and never deletes or replaces an existing resource. The App bundle then binds the exact index as a Unity Catalog TABLE securable with SELECT; the source table receives only SELECT and MODIFY for idempotent projection maintenance. Do not put Databricks credentials in bundle variables.

Keep knowledge_ai_search_enabled=0 until source-table writes, tenant isolation, stale-revision rejection, citations, and a clean-state human and external-agent canary pass. Enabling sets GTM_CHAT_VECTOR_SEARCH=1; it does not change canonical Knowledge storage or grant an agent access. That flag controls the legacy Databricks knowledge reader only. Canonical company-document retrieval uses the derived Cloudflare company-memory bridge configured by GTM_COMPANY_MEMORY_ORIGIN; PostgreSQL remains authoritative. If that origin is unset or unavailable, Cloudflare retrieval fails closed rather than falling back to the legacy reader. External runtimes additionally need the explicit gtm.search_workspace_knowledge registration grant.

Backfill existing canonical Knowledge only through the internal maintenance action. The command is plan-only unless --apply is present:

pnpm --filter @repo/gtm knowledge:reindex -- --tenant <organization-id>
pnpm --filter @repo/gtm knowledge:reindex -- --tenant <organization-id> --apply

It submits one bounded gtm.reindex_knowledge invocation per exact Lakebase revision. The action runs as system:knowledge-index-maintenance, emits immutable rebuild evidence before its adapter executes, and is idempotent for the tenant, Knowledge ID, and revision. Generic system actors and external agents cannot invoke it.

Production workers register under the Temporal Worker Deployment gtm-brain-prod with a unique TEMPORAL_WORKER_BUILD_ID. After the replacement snapshot is healthy, promote that exact build with the supported Worker Deployment API. Deprecated Version Set routing is not used.