Dataset & Prompt Versioning

Teams often say they “version prompts,” but the real behavior of a GenAI system depends on much more:

• System prompt and tool instructions
• Few-shot examples
• Output schema
• Retrieval chunking and ranking logic
• Evaluation datasets and scoring rules
• Tool behavior and fallback policy

If any of those change, the system changed.

Prompts should have stable identifiers and a visible change history. Avoid unnamed prompt edits inside application code with no traceability.

prompts/
  support/
    triage_v1.md
    triage_v2.md
  sales/
    quote_review_v3.md

Each version should record:

• Why the prompt changed
• Which failure mode it addresses
• Which evaluations should improve
• Who approved the rollout

Evaluation datasets are not static forever, but they cannot be edited casually. If tests change at the same time as the system, you can easily hide regressions.

Use separate dataset operations for:

• Additions — new failure cases discovered in production
• Corrections — fixes to labels or expected outputs
• Retirements — removing obsolete scenarios with a documented reason

evals/
  support-triage/
    dataset_v4.jsonl
    rubric_v4.md
    changelog.md

The dataset should tell a story of what the team learned, not just provide a pile of examples.

A release should point to a complete behavior snapshot. That means linking prompt versions, dataset versions, retrieval settings, and model choices to the deployed build.

{
  "release": "2026.04.12",
  "prompt_version": "support_triage_v7",
  "dataset_version": "support_eval_v4",
  "retrieval_config": "kb_chunking_v3",
  "model_policy": "routing_policy_v2"
}

This is what makes rollback possible. If a change hurts quality, you need to know exactly which bundle of changes went live.

Prompt and dataset changes should follow a review path similar to application code:

1. Propose a prompt, dataset, or retrieval change
2. Run offline evaluation against the current baseline
3. Review differences, not just average scores
4. Ship gradually if the change is user-visible or policy-sensitive
5. Record approval and keep rollback instructions nearby

Versioning only matters if it supports safer change management.