Simulator Evaluation Scaffold
Synthetic baseline comparison
1,000 synthetic scenarios comparing ASBOS multi-agent architecture against monolithic-agent and rule-based baselines. These results support the prospective evaluation protocol described in the manuscript — they are not production pilot evidence.
Simulator benchmark profile
Synthetic scenario results across resolution, safety, auditability, and escalation metrics.
Resolution Rate
Coverage
Share of synthetic scenarios resolved in the simulator harness.
Compliance Safety
Safety
Guards assigned only when credentials, availability, and site rules pass.
Audit Trail Depth
Auditability
Average typed events per scenario preserving parent-event lineage.
Autonomous Handling
Autonomy
Share of synthetic scenarios completed without human escalation.
Agent Separation
Architecture
Degree of bounded responsibility across specialized agents.
Escalation Discipline
Escalation
Balance of safe autonomous action and appropriate human handoff.
| System | Scenarios | Resolution Rate | Escalation Rate | Invalid Assignments | Avg Events |
|---|
Interpretation
Zero invalid assignments in simulation. ASBOS multi-agent achieved 0% invalid assignment rate through compliance filtering and typed event validation — compared to 12.6% for both monolithic and rule-based baselines in this synthetic harness.
Higher event count reflects audit design. ASBOS averages 10.23 events per scenario vs. 3–4 for baselines, reflecting typed event propagation and parent-event lineage — a deliberate auditability tradeoff.
Next step is field evidence. The manuscript defines proposed primary pilot outcomes including administrative minutes per scheduled guard-hour, median and 90th percentile fill time, payroll exception rate, human escalation rate, and false-positive exception rate. No measured fill time, administrative savings, or validated ROI is claimed today.