Agents and runtime status

GTM Brain deliberately separates two kinds of agent. Agent operations explains how the built-in Sense, Remember, Judge, Draft, approval, and learning stages are operating. Team & agents → Agents shows outside runtimes such as Claude, Cursor, Grok Bot, Hermes, and background workers paired with a tenant. Built-in specialists are installed and configured; outside agents connect and request authority. Their counts are never combined.

Governed Harness agents

The first bounded agents are:

GTM Temporal is the only durable outer orchestrator. Fabric Harness runs inside a bounded model activity with strict finite execution; it does not own a second approval or retry workflow. The model result is then staged immutably in Lakebase before separate, retryable mutation activities apply it through Fabric Platform. Full artifact content stays in Lakebase. Temporal receives only revision IDs, hashes, stage IDs, usage totals, and governed action references.

Every execution principal is minted server-side from an enabled tenant registration. It binds tenant, definition/version, run, workflow, capability grant, correlation, and causation. A caller cannot submit an arbitrary actor ID. Run rows report model tokens from Harness result events. When a provider returns an inline USD amount, the ledger displays that attributed cost. Databricks model services currently report token usage without an inline charge, so those runs display Cost pending instead of the misleading $0.0000; final USD attribution requires reconciliation with Databricks billing usage.

Agent registration, version publication, grant changes, and every run-stage transition are governed mutations. Starting a run uses a deterministic idempotency key and one Temporal lifecycle. Its first activity persists the queued record before reasoning or model work, then advances through loading inputs, generating, staging output, applying mutations, and a terminal stage. Open a run in the UI to inspect its Temporal identity, artifact bindings, budget, failures, and policy-safe action records. Raw action parameters and staged artifact bodies are never exposed in that view. After successful body cleanup, a compact immutable attestation remains with input/output hashes, exact campaign/artifact bindings, runtime/model/prompt identities, tool and skill manifests, provider request IDs, usage, applied action IDs, policy outcome, and final artifact or evaluation references.

Organization administrators use Team & agents for the combined view of human roles, agent registrations, emergency pause controls, registration-bound credentials, tenant guardrails, and immutable authority events. See Team & agents.

Install and activate the built-in agents

Open Workspace → Operations → Agent operations and select Install built-in agents. Installation runs the definition, version, and tenant-registration mutations through Fabric Platform and leaves all three registrations disabled. Repeating installation is idempotent and never disables or replaces an existing registration.

Use Configure on each card to select:

• disabled, enabled, or suspended status;
• the built-in's fixed, least-privilege autonomy mode;
• the Databricks Unity AI Gateway model;
• source-input tokens, per-model-call output tokens, cumulative run/retry tokens, and the per-run USD ceiling; and
• zero or more campaign bindings for the reviewer and generator, where zero means all tenant campaigns.

The Tenant skill runner registration remains tenant-unscoped because each request is bound instead to one exact active skill version and its current knowledge hashes. It cannot be launched from the artifact-run form; the authenticated skill-run endpoint or a properly scoped external registration creates its canonical admission.

Saving a changed model or budget publishes a new immutable AgentVersion and rebinds the tenant registration through governed actions. It does not mutate an existing version. Campaign IDs must resolve to tenant campaign definitions, and built-in cost ceilings cannot be expanded beyond the reviewer or generator safety envelope.

New built-in versions also bind the reviewed agent source digest and Fabric Harness runtime version. The worker refuses a registration whose stored executable identity does not match the deployed built-in. The run detail panel exposes both values for deployment and audit verification. Older custom versions remain readable, but do not satisfy the built-in executable check.

The UI exposes each capability grant for inspection, but built-in grants cannot be expanded—even through the generic control-plane mutation API. Executable code, prompt version, Databricks tool set, autonomy, and allowed proposal actions remain a certified envelope; admins can narrow campaign scope or disable/suspend the registration.

Use Connect an agent to pair Hermes, Grok Bot, or another out-of-process runtime. The guided flow asks for a recognizable name, a plain-language role, and explicit active campaign scope. Campaign operator requests the complete reviewed external action profile; Lead researcher, Content and messaging, Campaign reviewer, and CRM operator request narrower subsets. The administrator sees a short capability summary first and can open the individual action list before approval.

Once approved and claimed, the external runtime appears with its own name under Agents. It uses a registration-bound gateway key and starts every session from the live catalog. Every permitted mutation uses the same Fabric Platform action, policy, durable invocation, audit, and recovery path as the product UI. The role is a convenience preset, not blanket authority: campaign scope, grant lifetime, prospect- visible approvals, and action-specific guards still apply. See External agent gateway.

After enabling a registration, select Run agent. The reviewer accepts any campaign-bound immutable artifact revision. The generator accepts a campaign brief and creates a proposed artifact revision. The UI sends the exact artifact ID, revision, and content hash; the server mints the execution principal and Temporal workflow identity. Full artifact bodies never enter the browser mutation or Temporal input.

The agent definitions use the native Databricks SDK credential chain and schema-validated JSON extraction. They do not require a model-specific result-tool protocol, so the same bounded definitions work across Unity AI Gateway model services while retaining typed validation and bounded retries. AI Search/RAG is enabled only when a deployment supplies a real index, text column, and Unity Catalog permissions; it is never represented as an available tool without those resources.

The worker resolves the immutable registered version before model execution. That version—not a UI label—selects the Databricks model and governs the configured budget. Source bodies are rejected before the model when they exceed the configured input envelope. The per-call output envelope is applied as a hard max_tokens clamp on every Databricks model request and checked against the largest provider-reported call. A separate cumulative ceiling counts input and output tokens across structured-result retries for the whole run. This lets a bounded retry succeed when each response is within its call limit while still stopping runaway aggregate usage before evaluation or artifact mutation. The failed run and budget evidence remain visible in the run ledger.

Token and output ceilings are enforced synchronously during the run. A configured USD ceiling is also enforced whenever the provider reports cost inline. Until delayed Databricks billing reconciliation is available, Cost pending means “not yet attributed,” not “free” or “zero cost.”

Exact retries use the same command identity. Reusing a command ID with changed parameters fails closed, while replaying the same staged output cannot create another revision or evaluation. A worker crash after staging therefore resumes mutation without another model call; a crash before staging may repeat model inference, but cannot repeat a domain mutation.

Governed tenant skill contracts

The guided Create skill flow now stores a versioned gtm-skill-io-v1 contract with the Markdown instructions. Every draft declares bounded input and output fields, required active knowledge IDs, an explicit maximum list of allowed Fabric Platform actions, a risk tier, approval behavior, fixtures, and finite iteration, time, token, cost, and concurrency limits. Additional undeclared I/O fields are denied and the runtime failure mode is always fail-closed.

To create one, ask the authenticated GTM assistant to create a skill. It reads the system authoring guide and current tenant workspace, interviews for the contract, runs the isolated fixture evaluator, shows the exact proposal, then requests approval before invoking gtm.upsert_skill. The saved version remains a draft. The manual editor preserves the same contract when editing the Markdown or fixtures.

Promotion to active is schema- and policy-gated. An active version needs at least two fixtures; high or critical risk must retain approval; every required knowledge record must currently be active in the same tenant; and every allowed mutation must be in the reviewed external-action catalog. An allowed-action entry never grants authority. At execution time the actor or agent registration must still possess the current action grant and resource scope, and the action still traverses Platform admission, policy, durable invocation, adapter, event, and recovery handling.

This means tenant instructions may narrow product behavior but cannot add a tool, authorize a mutation, remove approval, exceed a runtime budget, or weaken the platform system policy.

An active contract runs only through gtm.request_skill_run. Platform first creates the durable, hash-bound admission; Temporal then carries only the opaque admission and staging IDs plus hashes into the compiled finite skill-runner. The runner has no tools, commands, or network access. It returns schema-validated private output and optional inert proposals. Any proposed business change must be submitted as a separate canonical Platform action under the requesting human's or external registration's current grant.

Private input, knowledge material, and output remain in tenant-isolated Lakebase staging and expire after 30 days. Long-lived GtmSkillRunRequested and AgentRun evidence contains only identities, hashes, budget/usage, policy outcome, and output references. Both the authenticated SaaS endpoint and the external-agent gateway reject credential-shaped input before dispatch; secrets belong in tenant-sealed storage and are never skill parameters.

Hermes uses its External campaign operator registration, an expiring gtm.request_skill_run execution grant, and exact skillIds. It does not receive a Databricks service principal. See External agents for installation, MCP request, polling, output, replay, and recovery details.

Phase 4 quality certification

Agent promotion uses a frozen Fabric Experiments dataset rather than treating a successful model call as release evidence. The protected certification executes the reviewer, generator, and skill runner against synthetic, hash-bound material, requires typed output plus non-zero usage, and applies the versioned gtm-finite-agent-output-contract@1.0.0 evaluator. Every example must pass; the release threshold is 1.0.

The resulting gtm-agent-quality-scorecard-v2 artifact contains the source revision, Databricks model endpoint, immutable dataset version/hash, evaluator version, observed gate, token/cost totals, all three output hashes, and the exact version/configuration/implementation/runtime/prompt/model-policy binding for every built-in agent. Generated text, artifact bodies, credentials, and tenant production data are excluded. The Phase 4 agent quality certification workflow runs weekly or by explicit dispatch. A passing local scorecard is published to the protected native managed-MLflow experiment as compact longitudinal metrics and hash-bound tags. The workflow then reads that finished run through Fabric Experiments and requires both contract_pass_rate>=1 and certification_passed==1; a missing metric, mismatched scorecard binding, failed write, or failed read-back blocks the release. The redacted scorecard and MLflow deep link are retained as CI evidence for 90 days.

The publisher requires a stable certification request ID. Retrying the exact scorecard converges on one MLflow run; a conflicting scorecard fails closed, and an ambiguous create response is resolved by the stable binding before another run can be created. The dedicated certification principal has edit access to only the evidence experiment and no GTM tenant or delivery authority.

Passing CI does not enable an agent. In the administrator-only Release assurance section, an organization owner or administrator chooses Verify evaluation for the currently registered version and pastes one managed MLflow run URL or ID from agent-quality-mlflow-link.json. The server reads the configured experiment and derives the scorecard, source, dataset, evaluator, and evidence bindings; the browser never asks an operator to transcribe hashes. The same gtm.record_agent_version_certification Platform action then uses an after_adapters managed-MLflow verification step, so GtmAgentVersionCertified exists only after the worker verifies the exact immutable binding and both release gates.

Ordinary GTM members see only business-facing readiness—Release approval needed, Ready to enable, Ready, or Paused. Release assurance shows currently registered versions first and keeps superseded immutable candidates in collapsed history. gtm.revoke_agent_version_certification retains the audit history and immediately blocks enablement, principal minting, final authorization, and worker execution for that version.

The Agents page also shows gtm-agent-outcome-evidence-v1, a privacy-safe rolling 90-day projection. It reports exact-version run reliability, reviewer classifications, and independently decided generator proposals. Cohorts below ten expose no counts or rates. The report contains no tenant, campaign, account, contact, actor, artifact, prompt, generated content, provider request, reason, or failure detail. Replies and meetings are not attributed to an agent without a separately governed prospective Fabric Experiments assignment.

This scorecard supplements the canonical mutation ledger. Agent registration, run transitions, evaluations, and proposed revisions still execute through Fabric Platform actions; the evaluation workflow cannot grant tools, approve artifacts, publish, send, or enable live delivery. MLflow evidence is a release gate, not an alternative mutation authority.

Deploy the agent artifact with Runway

GTM Brain no longer treats the monorepo as the agent deployment unit. The agents package is a Fabric Harness workspace that emits a detached Databricks App artifact:

pnpm agents:build:databricks-app

The command creates agents/.fabricharness/build/databricks-app/ with app.yaml, databricks.yml, the Harness server, all three compiled job definitions, and a verification manifest. Shared schemas from @repo/gtm-agent-contracts and pnpm catalog dependencies are bundled into the definitions. The artifact contains no private workspace imports or workspace:/catalog: install requirements.

agents/runway.json points Fabric Runway at this directory. Runway can therefore digest, stage, register, gate, and deploy the immutable artifact without checking out GTM Brain or resolving sibling packages. The artifact verifier fails the build if a private import survives, a required file is missing, or any built-in job is absent.

The deployed App exposes only the three explicitly webhook-enabled job routes under /api/jobs/. Databricks App identity, the Harness API token, tenant scoping, model endpoint access, and production resource bindings remain mandatory; packaging does not weaken those runtime controls.

Teams can also create a separate finite Harness agent without changing this repository. It connects to the same registration-bound MCP gateway used by Hermes and receives the same governed actions, current authorization checks, approvals, status, and audit evidence. Follow Build a Fabric Harness agent for the scaffold-to-Databricks procedure. External agents never receive GTM Brain database, Temporal, Platform Host, or delivery provider credentials.

The six stages

The webhook status appears alongside the stages because replies are required to close the learning loop.

The page intentionally separates two histories:

• Agent executions contains bounded Fabric Harness runs. It is collapsed by default and shows the registered definition, stage, model, campaign scope, token/cost budget, terminal status, and retained immutable execution attestation.
• Campaign pass history contains Temporal campaign sweeps and approval parking. A pass is not an agent run, even when it invokes bounded reasoning activities.

This distinction prevents generic rows labelled only “Pass” from being mistaken for autonomous-agent work. Technical workflow and run identifiers remain available for audit, while the primary cards explain what ran and what consequence it had.

Sense

When an organization has stored Apollo credentials—or deployment fallback credentials exist—the screen reports Apollo. Otherwise it reports stub fixtures. Stub mode is useful for product testing but must never be mistaken for a live market scan.

Judge

GTM_OFFLINE=1 selects deterministic heuristic judgment. Otherwise the judge uses the configured model through the active AI provider. Heuristic mode keeps the workflow testable without a model but is not the production intelligence path.

Draft

The card displays GTM_DRAFT_MODEL, defaulting to claude-sonnet-4-6. Model names are configuration, not proof that the endpoint is reachable; use the Databricks doctor and end-to-end tests for live validation.

Approve gate

This is a workflow boundary, not a model persona. Every prospect-touching action parks on an exact draft. It can be resolved by a person or by an enabled, campaign-scoped agent registration with a non-expired outreach.approver delegation. Prompt language cannot grant that capability, and there is no raw send bypass.

Send

The screen reports Instantly only when both usable credentials and campaign configuration resolve. It also shows live sends today against GTM_DAILY_SEND_CAP, which must be a plain integer from 1 to 500 and fails startup closed when invalid. Inbox ramp policy can impose stricter limits.

Learn

Initial bucket weights are compared with stored weights. If any weight has moved from its prior, the screen reports that learning has occurred and identifies the strongest current bucket.

Run a pass

Select Run a pass now to start the same dry-run Temporal workflow available from Overview. Watch the recent-pass list to see the current phase and pending approval count.

Example: verify production configuration

Before the first live campaign, Agents should report:

Sense       Apollo · connected
Judge       LLM mode
Draft       intended production model
Approve     0 or an intentionally reviewed queue
Send        Instantly · connected · cap understood
Webhook     configured
Learn       weights readable from durable storage

If a card reports stub or heuristic mode unexpectedly, stop and resolve configuration before approving any live campaign.

Next: Governed artifacts, AI models and gateways, Integrations, and External agents.