Databricks
Databricks features in use
Inventory of Databricks capabilities used by Fabric GTM Brain and the role each one plays.
This page documents the optional customer-managed Databricks target. The hosted GTM Brain product runs on Cloudflare with PostgreSQL and Temporal; operators do not need Databricks to connect an agent or run the standard product journey.
When an organization selects this optional target, GTM Brain uses Databricks as an application, operational data, AI, analytics, governance, and observability platform—not merely as a model endpoint.
Feature inventory
| Databricks feature | How GTM Brain uses it | Why it matters |
|---|---|---|
| Databricks Apps | Hosts Next.js, the Hono/oRPC API, Better Auth, and Temporal worker | Managed application compute, OAuth boundary, resource bindings, deployment history |
| Asset Bundles / direct engine | Declares App, Lakebase, UC, jobs, Genie, permissions, and targets | Repeatable infrastructure and releases from one resource graph |
| Lakebase Autoscaling | PostgreSQL 17 operational store for product and GTM state | Low-latency transactions with Databricks identity and UC integration |
| Lakebase branches/endpoints | Protected production branch, 0.5–4 CU endpoint, two readable secondaries | Production isolation, autoscaling, and HA |
| Endpoint-scoped OAuth credentials | App token is exchanged for short-lived database credentials | No static database password in code or configuration |
| Lakebase Unity Catalog catalog | Exposes operational gtm tables to Databricks governance and jobs | Governed projection without a bespoke extraction service |
| Unity Catalog external locations | ADLS Gen2 location backed by an Azure Databricks Access Connector | Centralized storage identity and least-privilege access |
| Managed UC catalog/schema | gtm_brain_analytics.gtm holds events, evaluations, funnel, telemetry, traces | Discoverability, lineage, grants, and shared analytics |
| SQL warehouse + Statements API | Bootstrap, readiness checks, live queries, gateway usage and analytics | Serverless SQL plane usable by the App, operators, and CI |
| Serverless Jobs environment v2 | Hourly Lakebase-to-Delta event projection | Scheduled, idempotent analytics without cluster management |
| AI Gateway v2 | Native Anthropic protocol for judge, drafter, classifier, and chat models | Workspace OAuth, centralized model access, usage visibility, governance |
| Unity Catalog model services | Discovers system.ai.* Claude, GPT, Gemini, and other services | Workspace-governed availability instead of hard-coded endpoints |
| Model Provider Services | Optional governed Anthropic or Amazon Bedrock route | Bring-your-own provider behind a UC securable and common contract |
| MLflow experiments and tracing | UC-backed experiment plus OpenTelemetry-compatible trace tables | Prompt/model observability and future outcome evaluation |
| App telemetry export | Logs, metrics, and spans flow into UC tables | Application observability can be queried and governed like data |
| Genie | Natural-language analytics over event, funnel, and evaluation tables | Governed business questions without SQL |
| Fabric Harness on native SDKs | Runs bounded reviewer/generator agents with Databricks model, SQL, and retrieval tools | Adds typed outputs, budgets, tenant grants, durable runs, artifact revisions, and approval routing without replacing Databricks APIs |
| Secret scopes + App secret resources | Better Auth, Temporal, credential encryption, webhook secrets | Secrets are referenced by name and injected only at runtime |
| App service principal and grants | Separate identity receives only required resource access | Workload identity is distinct from the deploying administrator |
Data destinations
| Dataset | Destination |
|---|---|
| Operational state | Lakebase database gtm, schemas public and gtm |
| GTM event projection | gtm_brain_analytics.gtm.events |
| Model evaluations | gtm_brain_analytics.gtm.llm_evaluations |
| Funnel aggregates | gtm_brain_analytics.gtm.funnel_daily |
| App OTel | gtm_otel_logs, gtm_otel_metrics, gtm_otel_spans |
| MLflow/OTel traces | gtm_brain_traces_* UC tables |
Security boundaries
Rendering diagram...
There are two intentional authentication layers. Databricks controls who can reach the App at all; Better Auth controls product users, organization membership, admin roles, and application sessions. Every GTM query is then scoped to the organization ID.