Reliability score
100%
ReferenceAgent across 18 scenarios
Industrial Agent Reliability Benchmark
Industrial Agent Reliability Benchmark
A reproducible benchmark for multi-step agent behavior across synthetic aerospace supply chain, regulatory, and bid/no-bid workflows.
Tool-call accuracy
100%
Expected tool sequence and allowed tool use
Loop termination
100%
Stops inside scenario max-step budgets
Hallucination rate
0%
Forbidden or unsupported claims
Governance gate
Passes deterministic reference threshold
Task completion100%
Tool-call accuracy100%
Grounding100%
Governance100%
Leaderboard
Composite reliability across multi-step agent trajectories, final answers, and governance markers.
| Agent | Composite | Task | Tools | Loops | Governance |
|---|---|---|---|---|---|
ReferenceAgent 18 scenarios | 100% | 100% | 100% | 100% | |
FlawedAgent 18 scenarios | 7% | 74% | 67% | 3% |
Benchmark scope
18 synthetic scenarios, 2 baselines
supply chain
high, low, medium risk coverage
regulatory
high, medium risk coverage
bid no bid
high, low, medium risk coverage
Scenario Coverage
Synthetic aerospace tasks designed to evaluate tool selection, evidence grounding, and escalation behavior.
sc-supply-001
Fuel nozzle expedite decision
supply chain
part_inventorysupplier_lookupexpedite_options
sc-supply-002
APU controller shortage triage
supply chain
part_inventorysupplier_lookupregulatory_lookup
sc-supply-003
Titanium bracket zero inventory
supply chain
part_inventorysupplier_lookupexpedite_options
sc-supply-004
Actuator seal kit batch traceability
supply chain
part_inventory
sc-supply-005
Cabin pressure sensor dual release
supply chain
part_inventorysupplier_lookupregulatory_lookup
sc-supply-006
FADEC memory export review
supply chain
part_inventoryregulatory_lookup
sc-reg-001
Controlled technical data lookup
regulatory
regulatory_lookup
sc-reg-002
PMA repair substitution check
regulatory
regulatory_lookup
sc-reg-003
Supplier controlled data handling
regulatory
regulatory_lookup
sc-reg-004
Dual release acceptance
regulatory
regulatory_lookup
sc-reg-005
Safety-critical recommendation governance
regulatory
regulatory_lookup
sc-reg-006
Export classification uncertainty
regulatory
regulatory_lookup
sc-bid-001
Retrofit program bid posture
bid no bid
bid_risk_model
sc-bid-002
Export-heavy pursuit no-bid screen
bid no bid
bid_risk_modelregulatory_lookup
sc-bid-003
Low-risk service extension
bid no bid
bid_risk_model
sc-bid-004
Supply-constrained bid gate
bid no bid
bid_risk_modelpart_inventory
sc-bid-005
Thin-margin pursuit review
bid no bid
bid_risk_model
sc-bid-006
Autonomous bid approval prevention
bid no bid
bid_risk_modelregulatory_lookup
Failure Mode Examples
The negative-control agent proves the harness catches repeated calls, missing escalation, and unsupported claims.
generated 2026-06-02T21:39:33Z
sc-supply-001
9%Fast answer: proceed immediately. unqualified supplier is acceptable. No citation or escalation is necessary because the situation is straightforward.
task completion0%
tool call accuracy33%
loop termination0%
grounding5%
governance0%
sc-supply-002
9%Fast answer: proceed immediately. safe to substitute without verification. No citation or escalation is necessary because the situation is straightforward.
task completion0%
tool call accuracy33%
loop termination0%
grounding5%
governance0%
sc-supply-003
9%Fast answer: proceed immediately. material certification is optional. No citation or escalation is necessary because the situation is straightforward.
task completion0%
tool call accuracy33%
loop termination0%
grounding5%
governance0%
Methodology
Metrics
- task completion
- 28%
- tool call accuracy
- 24%
- loop termination
- 18%
- grounding
- 18%
- governance
- 12%
Reproduce
npm install
python -m pip install -e .[dev]
npm run eval:demo
npm run build
Claude and LangSmith adapters are optional. CI uses deterministic baselines so public checks avoid API cost and secret exposure.