LLMSheriff
Research prototype for intent-aware monitoring of autonomous AI agents.
Hypothesis: execution traces can be interpreted to infer behavioral states — distinguishing a healthy running agent from one that has quietly abandoned its goal.
Preset scenarios
Load a representative execution trace for analysis.
Task information
Use this button for immediate results. Use "Replay Trace & Analyze" below for a step-by-step demo.
Agent execution trace
Step-by-step record of agent actions.
Live execution demo
Replay the trace step by step, then auto-run analysis.
Task
Build a landing page for a SaaS startup.
Press "Replay Trace & Analyze" to run the full guided flow.
Intervention recommendation
Run analysis to see a recommendation.
Execution timeline
Step index vs event duration (seconds).
Intent timeline
Behavioral state inferred per execution step.
15:20
Planning
PLANNING · 1.2s
15:20
Executing
LLM_CALL · 2.8s
15:20
Executing
TOOL_CALL · 3.1s
Behavior graph
Activity distribution and confidence trajectory.
Run analysis to see metrics.
Behavior score
Run analysis to see scores.
Analyzer output
Rule engine
Deterministic thresholds
Run analysis to see prediction.
Nimotron LLM
NVIDIA Nemotron judge
Run analysis to see prediction.
Research contribution
LLMSheriff explores whether execution traces can be transformed into higher-level behavioral states using hybrid symbolic and LLM-based inference. Rather than replacing observability tools, it investigates how explainable behavioral monitoring can support debugging and human intervention — particularly distinguishing an agent still working toward a goal from one that has effectively given up.
Prototype limitations
- Uses heuristic thresholds rather than learned models.
- Evaluates preset scenarios rather than live autonomous agents.
- Behavioral inference is exploratory and not validated at scale.
- No formal human evaluation or ground-truth labeling has been conducted.
- LLM-based analysis depends on external API availability.
Recent runs
No runs yet.