Cost Engineering

Cost engineering is not a late optimization pass. In GenAI systems, the architecture itself determines the cost shape: prompt length, retrieval depth, tool calls, retries, fallback chains, and model choice all compound.

A practical cost model looks like this:

total_request_cost =
  model_input_cost +
  model_output_cost +
  retrieval_cost +
  tool_execution_cost +
  retry_cost +
  observability_overhead

If you only track the final API bill, you are already too late. You need to attribute spend to the request path that created it.

The highest-leverage cost control is usually routing, not prompt trimming. Use small, cheap models for classification, extraction, and low-risk drafting. Escalate to larger models only when the task actually requires them.

def route_request(task_type: str, risk_level: str):
    if task_type in {"classification", "simple_extraction"}:
        return "small-fast-model"
    if risk_level == "high":
        return "large-reliable-model"
    return "mid-tier-model"

Good routing decisions should consider:

• Task complexity — reasoning depth, retrieval needs, and output precision
• Risk — financial, legal, customer-facing, or irreversible actions
• Latency target — user-facing chat flows tolerate less delay than background jobs
• Fallback policy — whether cheaper models are allowed to fail open to a stronger model

Not every request should hit a model. Reuse work aggressively where the product allows it.

What to cache

• Stable prompt templates and system messages
• Repeated retrieval results for popular documents or queries
• Deterministic tool outputs such as internal metadata lookups
• Low-volatility generation tasks like FAQ answers or summaries

What not to cache blindly

• User-specific answers that depend on private state
• Compliance-sensitive content that must reflect current policy
• Outputs from prompts that change frequently

cache_key = hash(
    model_name,
    prompt_version,
    retrieval_snapshot_id,
    user_scope,
    normalized_input
)

For offline or async workloads, batch processing can reduce cost substantially. Summaries, tagging jobs, enrichment pipelines, and background classification should not run through an interactive path if you can avoid it.

Every production GenAI system should have explicit budget controls. Otherwise the cheapest success path in development becomes the most expensive production incident.

Enforce limits at several levels:

• Per request — max prompt size, max tool calls, max retries, max output tokens
• Per session — rolling spend cap for a chat or workflow run
• Per tenant — monthly quota, alert thresholds, and overage policy
• Per feature — separate budgets for internal copilots, customer chat, batch pipelines, and evaluators

{
  "tenant_id": "acme-co",
  "feature": "support-copilot",
  "request_id": "req_123",
  "model_cost_usd": 0.018,
  "tool_cost_usd": 0.004,
  "retry_cost_usd": 0.002,
  "total_cost_usd": 0.024
}

If finance or product teams cannot answer “which customer workflow costs the most?” you do not yet have cost observability.

Cost guardrails belong in the runtime, not just in dashboards. Use live controls to stop runaway requests.

• Token ceilings on every call
• Tool-call caps per run
• Fallback suppression when the system is already over budget
• Concurrency limits for expensive workflows
• Alerts on sudden spend spikes by tenant, workflow, or model

The goal is not “lowest possible cost.” The goal is predictable cost for acceptable quality.