Observability

Observability is built into the core. State automatically writes trace.jsonl to the session directory whenever you use SessionStore. There is nothing to enable or configure — tracing is always on.

trace.jsonl format¶

Every significant event during a session is appended as a JSON line to trace.jsonl. Each line has a type field and a timestamp.

Event types¶

Type	Description
`run_start`	A new `session.run()` call begins
`run_end`	A `session.run()` call completes (includes status)
`message`	A message added to conversation (role: user, assistant, tool, system)
`llm_start`	LLM API call initiated
`llm_end`	LLM API call completed (includes usage)
`tool_start`	Tool execution begins (includes tool name and args)
`tool_end`	Tool execution completes (includes result)
`tool_blocked`	Tool call denied by permissions or approval
`compact`	Context compaction occurred (includes tokens before/after)
`usage`	Cumulative usage snapshot

Sample trace¶

{"type": "run_start", "run_id": 1, "prompt": "List all tables.", "timestamp": "2026-02-21T10:00:00Z"}
{"type": "message", "role": "user", "content": "List all tables.", "timestamp": "2026-02-21T10:00:00Z"}
{"type": "llm_start", "model": "gpt-5-mini", "context_size": 150, "timestamp": "2026-02-21T10:00:00Z"}
{"type": "llm_end", "model": "gpt-5-mini", "input_tokens": 150, "output_tokens": 45, "cache_read_tokens": 0, "cost_usd": 0.0003, "timestamp": "2026-02-21T10:00:01Z"}
{"type": "tool_start", "tool_name": "bash", "tool_call_id": "call_abc", "tool_args": {"command": "psql -c '\\dt'"}, "timestamp": "2026-02-21T10:00:01Z"}
{"type": "tool_end", "tool_name": "bash", "tool_call_id": "call_abc", "success": true, "timestamp": "2026-02-21T10:00:01Z"}
{"type": "llm_start", "model": "gpt-5-mini", "context_size": 280, "timestamp": "2026-02-21T10:00:02Z"}
{"type": "llm_end", "model": "gpt-5-mini", "input_tokens": 280, "output_tokens": 90, "cache_read_tokens": 0, "cost_usd": 0.0006, "timestamp": "2026-02-21T10:00:03Z"}
{"type": "message", "role": "assistant", "content": "The database has 12 tables...", "timestamp": "2026-02-21T10:00:03Z"}
{"type": "run_end", "run_id": 1, "status": "completed", "timestamp": "2026-02-21T10:00:03Z"}

Session directory layout¶

Each session directory at ~/.config/rho-agent/sessions/<session_id>/ contains:

File	Purpose
`config.yaml`	AgentConfig snapshot (model, profile, system prompt, etc.)
`trace.jsonl`	Append-only event log — source of truth for the session
`meta.json`	Session metadata (id, status, timestamps, model, profile, first prompt)
`cancel`	Sentinel file — presence signals cancellation request
`pause`	Sentinel file — presence signals pause request
`directives.jsonl`	Operator directives queued for the agent (JSON lines)

Monitor¶

For live observation and control of running agents, see the Monitor guide.

Offline inspection¶

trace.jsonl files are plain JSON lines and can be inspected with standard tools.

Using jq¶

# Count tool calls in a session
jq -s '[.[] | select(.type == "tool_start")] | length' trace.jsonl

# Show all LLM usage events
jq 'select(.type == "llm_end") | .usage' trace.jsonl

# List distinct tool names used
jq -r 'select(.type == "tool_start") | .tool_name' trace.jsonl | sort -u

# Get total input tokens across all LLM calls
jq -s '[.[] | select(.type == "llm_end") | .usage.input_tokens] | add' trace.jsonl

Using State.from_jsonl¶

Replay a trace file to restore full conversation state in Python:

from pathlib import Path
from rho_agent import State

trace = Path("~/.config/rho-agent/sessions/abc123/trace.jsonl").expanduser()
state = State.from_jsonl(trace.read_bytes())
print(f"Messages: {len(state.messages)}")
print(f"Usage: {state.usage}")
print(f"Status: {state.status}")
print(f"Runs completed: {state.run_count}")

Observers¶

The StateObserver protocol lets you attach custom side channels to receive events in real time. Any object with an on_event(event: dict) method satisfies the protocol.

from rho_agent import Agent, AgentConfig, Session

class MetricsCollector:
    def on_event(self, event: dict) -> None:
        if event["type"] == "llm_end":
            usage = event.get("usage", {})
            print(f"LLM call: {usage.get('input_tokens', 0)} in, {usage.get('output_tokens', 0)} out")
        elif event["type"] == "tool_end":
            print(f"Tool {event['tool_name']} completed in {event.get('duration_ms', 0)}ms")

agent = Agent(AgentConfig(profile="developer"))
session = Session(agent)
session.state.add_observer(MetricsCollector())

result = await session.run(prompt="Analyze the codebase.")

Observers receive every event that is written to trace.jsonl, making them suitable for live dashboards, metrics collection, alerting, or custom logging pipelines.

ATIF export¶

Session traces can be exported to ATIF (Agent Trajectory Interchange Format), a structured JSON schema for agent trajectories designed for SFT/RL training data.

# Export a session to stdout
rho-agent export <session-id> --dir ~/.config/rho-agent/sessions

# Write to a file
rho-agent export <session-id> -d ~/.config/rho-agent/sessions -o trajectory.json

from rho_agent.export.atif import trace_to_atif

trajectory = trace_to_atif(
    "path/to/trace.jsonl",
    session_id="abc123",
    model_name="gpt-5-mini",
)

The export maps trace events to ATIF steps, groups tool calls with their results, and populates per-step and session-level metrics. See rho_agent/export/ for details and the CLI reference for all flags.