Quality gates for every agent release
Define evaluation criteria once. Run them automatically against every trace, deployment, or on-demand. Block bad releases before users see them.
LLM-as-judge
Use a judge model to score agent outputs on custom rubrics — accuracy, helpfulness, safety, and tone.
Deterministic
Regex, JSON schema, and semantic similarity checks. Fast and cheap; great for structured outputs.
Human-in-loop
Route borderline traces to human reviewers. Label and feed back into the scoring pipeline.
Behavioral Regression
Compare new deployments against golden baselines. Alert when output distribution shifts.
Tool-call validation
Inspect every tool invocation. Catch malformed JSON args, hallucinated tools, and out-of-policy function calls.
Adversarial probes
Run a library of 600+ red-team prompts on every release. Track jailbreak resistance and PII leakage rates over time.
Ship with 40+ scorers tuned across RAG, agent, brand-voice, and safety axes. Each one can be invoked from the SDK, the dashboard, or a CI job.
Author scorers
Compose pre-built scorers or write your own rubric in 5 lines of YAML. Versioned alongside your agent code.
Curate a dataset
Sample production traces, import golden Q&A pairs, or synthesize examples with Vektor's seed generator.
Run an eval suite
Replay 500 traces in parallel across 4 candidate prompts. Cost-capped and rate-limit aware.
Block or ship
Suites fail the build below your threshold. Successful runs are signed and attached to the release.
| Capability | Vektor | Generic LLM tracing | Spreadsheets & vibes |
|---|---|---|---|
| LLM-as-judge scoring | ✓ | ✓ | — |
| CI pass/fail gates | ✓ | — | — |
| Behavioral regression detection | ✓ | — | — |
| Trace replay with prompt swap | ✓ | — | — |
| Adversarial probe library | ✓ | — | — |
| Eval cost per 1k traces (typ.) | $0.40 | $2.10 | n/a |
Evaluations integrate natively with GitHub Actions, GitLab CI, and any webhook-compatible pipeline. Deployments are automatically blocked if eval pass-rate drops below your configured threshold.
# .github/workflows/eval.yml
- name: Run Vektor evals
uses: vektor-ai/eval-action@v2
with:
suite: production-baseline
threshold: 0.95
fail_on_regression: true“We replaced three notebooks, a Looker dashboard, and a Slack channel of vibes with one Vektor eval suite. Catching faithfulness regressions now takes 90 seconds instead of three days.”