How to manage AI agent configs across environments

Last February, a team at a mid-sized SaaS company deployed their AI support agent's Valentine's Day persona to production three weeks late, in early March. It had been sitting in staging, configured through a dashboard, waiting for a promotion workflow that didn't exist. Someone finally clicked "copy to production" -- except by then the promotion was for a different holiday. Customers interacted with an agent that called them "sweetheart" and mentioned gift guides until someone noticed the logs.

The config lived in a dashboard with no review process, no history, and no rollback. The incident took four minutes to cause, and forty minutes to diagnose.

Agent configuration has the same blast radius as code -- arguably larger, because a config change can silently alter behavior across thousands of conversations before anyone notices. But most teams still manage it through dashboards and copy-paste. This article shows you how to build a config management system that treats agent config as the first-class artifact it is.

What agent config actually includes

Agent configuration is everything that controls how your agent reasons and acts, distinct from the runtime data it produces. Teams starting out usually think of config as "the system prompt." That's true, but it's only one piece.

A complete agent config includes the system prompt and any prompt templates for sub-tasks, the model identifier and generation parameters (temperature, max tokens, top-p), the full list of available tools and their permission scopes, memory configuration (what gets stored, retention policy, embedding model), RAG pipeline settings (chunk size, retrieval k, similarity threshold), fallback behaviors (what to do when a tool fails, when the model hits a rate limit, when confidence is low), escalation rules (when to hand off to a human, what triggers immediate escalation), and rate limits both for the agent itself and for individual tools.

export interface AgentToolConfig {
  name: string;
  server: string;           // MCP server URL or identifier
  permissions: string[];    // Which operations this tool can perform
  rateLimit?: {
    maxCallsPerMinute: number;
    maxCallsPerConversation: number;
  };
}
 
export interface AgentMemoryConfig {
  enabled: boolean;
  retentionDays: number;
  embeddingModel: string;
  similarityThreshold: number;
  maxEntriesPerUser: number;
}
 
export interface AgentRagConfig {
  enabled: boolean;
  knowledgeBaseId: string;
  chunkSize: number;
  retrievalK: number;
  similarityThreshold: number;
}
 
export interface AgentConfig {
  version: string;          // Semantic version: "2.1.0"
  name: string;
  description: string;
  model: {
    provider: "anthropic" | "openai" | "google";
    modelId: string;
    temperature: number;
    maxTokens: number;
  };
  systemPrompt: string;
  tools: AgentToolConfig[];
  memory: AgentMemoryConfig;
  rag: AgentRagConfig;
  escalation: {
    triggers: string[];     // Keywords or conditions
    handoffMessage: string;
  };
  fallbacks: {
    toolError: "retry" | "graceful-message" | "escalate";
    lowConfidence: "clarify" | "escalate" | "attempt";
    modelTimeout: "retry" | "escalate";
  };
}

This schema becomes your single definition of what an agent is. Everything else -- dashboards, deployment scripts, monitoring systems -- reads from this schema.

The dashboard problem

Dashboard-based config management fails as soon as you have more than one environment, more than one agent, or more than one engineer making changes. The failure mode arrives gradually: a QA engineer adjusts staging to test something and forgets to revert, a product manager tweaks the production prompt through the UI and doesn't tell engineering, two developers make conflicting changes to different environments without knowing it.

The underlying problem is that dashboards optimize for single-user, synchronous edits. They don't offer code review, approvals, or a diff view that shows why something changed alongside what changed. When your production agent starts behaving differently on a Tuesday, the dashboard activity log shows "System prompt updated" with no context.

Compare this to a git commit with a message like: "fix: tighten escalation trigger to prevent false handoffs on price questions -- was causing 12% of pricing questions to route to human agents incorrectly." The fix is the same change. The git approach makes the reasoning retrievable.

The other failure mode is environment drift. Without a single source of truth, staging and production configs diverge silently. You test a behavior in staging, promote it to production, and discover that production has a different memory config you forgot to update, so the behavior doesn't match.

Designing the config file structure

The config-as-code pattern uses a file hierarchy with three layers: a base config that applies everywhere, environment-specific overlays, and agent-specific overrides.

config/
  agents/
    support-agent/
      base.json           # The canonical config for this agent
      dev.json            # Dev overrides (different model, debug logging)
      staging.json        # Staging overrides (real model, test tools)
      production.json     # Prod overrides (prod tool endpoints)
    sales-agent/
      base.json
      dev.json
      staging.json
      production.json
  shared/
    models.json           # Common model configs
    tools.json            # Shared tool definitions
    rate-limits.json      # Organization-wide rate limits

The merge order at deploy time is: base config + shared overrides + environment overlay. Environment overlays only need to specify what changes -- everything else inherits from base.

{
  "version": "2.1.0",
  "name": "support-agent",
  "model": {
    "provider": "anthropic",
    "modelId": "claude-sonnet-4-5",
    "temperature": 0.3,
    "maxTokens": 2048
  },
  "memory": {
    "enabled": true,
    "retentionDays": 90,
    "embeddingModel": "text-embedding-3-small",
    "similarityThreshold": 0.35,
    "maxEntriesPerUser": 500
  },
  "escalation": {
    "triggers": ["legal action", "regulatory complaint", "speak to manager"],
    "handoffMessage": "I'm connecting you with a specialist who can help."
  }
}

{
  "model": {
    "modelId": "claude-haiku-4-5-20251001"
  },
  "memory": {
    "retentionDays": 7
  },
  "tools": [
    {
      "name": "getCustomerProfile",
      "server": "http://localhost:4000/mcp",
      "permissions": ["read"]
    }
  ]
}

The dev overlay uses a cheaper model and points tools at a local MCP server instead of the production endpoint. Staging uses the production model but points at staging tool endpoints. Production uses production everything.

Merging configs at deploy time

You need a merge function that deep-merges environment overlays onto base config, validates the result against the schema, and fails the deploy if validation fails.

import { readFileSync, writeFileSync } from 'fs';
import { join } from 'path';
import { AgentConfig } from '../types/agent-config';
import { validateConfig } from './validate-config';
 
function deepMerge<T>(base: T, overlay: Partial<T>): T {
  const result = { ...base };
  for (const key in overlay) {
    const baseVal = base[key];
    const overlayVal = overlay[key];
    if (
      overlayVal !== null &&
      typeof overlayVal === 'object' &&
      !Array.isArray(overlayVal) &&
      typeof baseVal === 'object'
    ) {
      result[key] = deepMerge(baseVal as object, overlayVal as object) as T[typeof key];
    } else {
      result[key] = overlayVal as T[typeof key];
    }
  }
  return result;
}
 
export function buildAgentConfig(
  agentName: string,
  environment: 'dev' | 'staging' | 'production'
): AgentConfig {
  const configDir = join('config', 'agents', agentName);
 
  const base = JSON.parse(readFileSync(join(configDir, 'base.json'), 'utf-8'));
  const overlay = JSON.parse(
    readFileSync(join(configDir, `${environment}.json`), 'utf-8')
  );
 
  const merged = deepMerge(base, overlay) as AgentConfig;
 
  const errors = validateConfig(merged);
  if (errors.length > 0) {
    throw new Error(`Config validation failed for ${agentName}/${environment}:\n${errors.join('\n')}`);
  }
 
  return merged;
}

Run this at deploy time, not at runtime. The deploy process should produce a validated, merged config artifact and pass that artifact to the agent runtime. If merge or validation fails, the deploy fails before anything touches production.

Schema validation that catches real problems

A JSON Schema or TypeScript-based validator catches structural errors. But agent config has semantic constraints that schema validation can't catch automatically -- constraints about the relationship between fields.

Write explicit validation rules for the constraints that matter in your system:

import { AgentConfig } from '../types/agent-config';
 
export function validateConfig(config: AgentConfig): string[] {
  const errors: string[] = [];
 
  // Version must be semantic
  if (!/^\d+\.\d+\.\d+$/.test(config.version)) {
    errors.push(`Invalid version format: ${config.version}. Must be semver (e.g. 2.1.0)`);
  }
 
  // Temperature constraints
  if (config.model.temperature < 0 || config.model.temperature > 1) {
    errors.push(`Temperature must be between 0 and 1, got ${config.model.temperature}`);
  }
 
  // Escalation must have triggers
  if (config.escalation.triggers.length === 0) {
    errors.push('escalation.triggers cannot be empty — agent has no escalation path');
  }
 
  // Memory threshold must be meaningful
  if (config.memory.enabled && config.memory.similarityThreshold < 0.2) {
    errors.push(`Memory similarity threshold ${config.memory.similarityThreshold} is too low — will retrieve irrelevant memories`);
  }
 
  // Max tokens must fit model context
  const maxTokensByModel: Record<string, number> = {
    'claude-sonnet-4-5': 8192,
    'claude-haiku-4-5-20251001': 4096,
    'gpt-4o': 4096,
  };
  const modelMax = maxTokensByModel[config.model.modelId];
  if (modelMax && config.model.maxTokens > modelMax) {
    errors.push(`maxTokens ${config.model.maxTokens} exceeds model max ${modelMax} for ${config.model.modelId}`);
  }
 
  // Tools must have at least one permission
  config.tools.forEach((tool, i) => {
    if (tool.permissions.length === 0) {
      errors.push(`Tool at index ${i} (${tool.name}) has no permissions — agent can't use it`);
    }
  });
 
  return errors;
}

Run validation in CI on every pull request. A config PR that would deploy a broken agent fails before merge.

Versioning and promotion workflow

Every config change goes through the same review process as code. The PR contains the config diff, a description of what changed and why, and a link to the scenario results that validated the change (more on scenarios below).

Versioning convention:

• Major (2.0.0 → 3.0.0): Changes that meaningfully alter agent behavior -- system prompt rewrites, adding or removing tools, model upgrades. Requires shadow mode validation before production.
• Minor (2.0.0 → 2.1.0): Non-breaking additions -- new fallback behaviors, updated descriptions, new escalation triggers. Requires staging validation.
• Patch (2.0.0 → 2.0.1): Low-risk fixes -- typo corrections, threshold adjustments, log format changes. Staging validation recommended but promotion can be expedited.

The version in config is a contract with your monitoring system. When a production incident is traced to a config version, you want to know exactly which version introduced the change and why.

Validating with scenarios before promotion

Scenario test results are a more reliable promotion gate than human review alone. A config change that passes schema validation and looks correct in a PR review can still break agent behavior in ways that only show up in realistic conversations.

Write scenario definitions that cover the key behaviors your config affects:

{
  "name": "Escalation trigger: legal threat",
  "description": "Agent must escalate when customer mentions legal action",
  "turns": [
    {
      "user": "I've been waiting three weeks for my refund. I'm going to contact my lawyer.",
      "expectedBehavior": {
        "mustEscalate": true,
        "mustNotResolveOnOwn": true
      }
    }
  ],
  "scoringCriteria": {
    "escalationTriggered": { "weight": 1.0, "required": true },
    "appropriateHandoffMessage": { "weight": 0.5 }
  }
}

Run scenarios against the proposed config in a sandboxed environment before any promotion. If scenario pass rate drops below your threshold, block the promotion and surface the failing scenarios in the PR.

This connects to Chanl's scenario testing feature, which lets you define expected behaviors and grade agent responses automatically against real or synthetic conversations. When you adopt config-as-code, your scenario results become part of the config's PR review artifact. For teams managing agent tools and MCP integrations, scenario coverage is especially important -- tool config changes are harder to review visually because the impact only shows in behavior.

Secrets management

The config file is in git. API keys, MCP server credentials, and database connection strings are not -- they live in environment variables or a secrets manager and are injected at runtime.

The pattern: config files reference secrets by name, not value.

{
  "tools": [
    {
      "name": "getCustomerProfile",
      "server": "${MCP_CUSTOMER_PROFILE_URL}",
      "apiKeyRef": "CUSTOMER_PROFILE_MCP_KEY"
    }
  ]
}

At deploy time, a config loader resolves the references against the environment:

function resolveSecrets(config: AgentConfig): AgentConfig {
  const resolved = JSON.parse(JSON.stringify(config)); // deep clone
  
  function resolveValue(value: unknown): unknown {
    if (typeof value === 'string' && value.startsWith('${') && value.endsWith('}')) {
      const envKey = value.slice(2, -1);
      const envVal = process.env[envKey];
      if (!envVal) throw new Error(`Missing required env var: ${envKey}`);
      return envVal;
    }
    if (typeof value === 'object' && value !== null) {
      return Object.fromEntries(
        Object.entries(value as Record<string, unknown>).map(([k, v]) => [k, resolveValue(v)])
      );
    }
    return value;
  }
  
  return resolveValue(resolved) as AgentConfig;
}

The config schema documents which fields are secret references. New team members know to check their environment variables when local dev fails, not to look for hardcoded values in config files.

Loading config at runtime

Once the config is validated, merged, and secret-resolved, it becomes the single input to the agent runtime. If you're using Chanl, you can push the resolved config to a deployed agent and verify the active config matches your expected version:

import Chanl from '@chanl/sdk';
 
const chanl = new Chanl({ apiKey: process.env.CHANL_API_KEY });
 
async function deployAgentConfig(agentId: string, config: AgentConfig) {
  // Verify tools are reachable before committing
  const toolCheck = await chanl.tools.list({ agentId });
  const expectedToolNames = config.tools.map(t => t.name);
  const missingTools = expectedToolNames.filter(
    name => !toolCheck.tools.some(t => t.name === name)
  );
  if (missingTools.length > 0) {
    throw new Error(`Tools not reachable: ${missingTools.join(', ')}`);
  }
 
  // Push the new config
  await chanl.agents.updateConfig(agentId, {
    version: config.version,
    systemPrompt: config.systemPrompt,
    model: config.model,
    tools: config.tools,
    memory: config.memory,
  });
 
  console.log(`Deployed config v${config.version} to agent ${agentId}`);
}

After deploy, your monitoring dashboard should show the active config version alongside your quality metrics. When a quality dip occurs, correlating it to a config version change becomes a two-second lookup instead of a half-hour investigation. Tracking which config version was running for each conversation is also the first step toward building the feedback loop where prompt performance data drives the next config improvement.

Handling rollbacks

A config-as-code system makes rollbacks straightforward: redeploy the previous version. This assumes your config versions are immutable once deployed -- never edit a deployed version, always create a new version.

The rollback procedure is the same as a forward deploy, with a previous version as the target:

# Rollback support-agent to version 2.0.3
CONFIG_VERSION=2.0.3 ENV=production ./scripts/deploy.sh support-agent

Keep a production deployment log that records which config version is running, when it was deployed, and who approved it. This log is your audit trail and your rollback history.

Common pitfalls to avoid

Treating config as deployment-time input, not runtime state. Config should be determined before the runtime starts, not read from the database during a conversation. If your agent reads its config from a database on every call, a database change during a live conversation can alter behavior mid-conversation.

Putting behavioral logic in feature flags. Feature flags handle instant on/off switches for features. They're not a config management system. If you're running more than two or three feature flags that affect agent behavior, you've outgrown feature flags and need proper config versioning.

Skipping validation for "small" changes. Validation is fast. The cost of running schema validation on a two-line change is milliseconds. The cost of a typo in a threshold value causing subtle misbehavior for a week is much higher.

Not pinning configs to deployments. Logs need to know which config version was running at the time of a conversation to be useful for debugging. Emit the config version as structured metadata on every conversation and every log line.

Managing AI agent configs with the same discipline as application code isn't optional when you're running agents in production. The blast radius of a config mistake is a conversation -- or a thousand of them. The investment in a config-as-code workflow pays back the first time you need to roll back a bad system prompt at 2am and it takes four minutes instead of forty.

The pattern here -- schema, environment layering, semantic versioning, scenario-gated promotion -- works whether you're running one agent or fifty. Start with the schema. Add validation. Add versioning. The promotion workflow falls out naturally once the foundation is in place.

El briefing de Signal

Un email por semana. Cómo los equipos líderes de CS, ingresos e IA están convirtiendo conversaciones en decisiones. Benchmarks, playbooks y lo que funciona en producción.

Subscribe

500+ líderes de CS e ingresos suscritos