Consult benchmark: native SendMessage versus murmur¶
Three Claude Code sessions (Implementer, Peer A, Peer B), model haiku, Claude Code 2.1.304 (Claude Code), run on 2026-10-20, cwd = this repository. Arm native has no murmur MCP and no murmur hooks; arm native-told is arm native with a prompt that also tells each peer to send its answer back with SendMessage (below); arm murmur has what murmur install claude writes. Sessions are renamed first (/rename Implementer, /rename Peer A, /rename Peer B), then this prompt (arms native and murmur) is typed into the Implementer only:
Consult Peer A and Peer B: ask each of them to check whether HANDLE_RE in src/protocol.ts allows handles of up to 48 characters, and to answer AGREE or DISAGREE with the line they checked. Wait for both answers, then tell me the outcome in one line.
Arm native-told gets:
Consult Peer A and Peer B: ask each of them to check whether HANDLE_RE in src/protocol.ts allows handles of up to 48 characters, and to send their answer back to you with SendMessage, AGREE or DISAGREE with the line they checked. Wait for both answers, then tell me the outcome in one line.
Numbers come from each session's own transcript, counted from the task prompt onward and deduplicated by message id; arm murmur also checks the broker deliveries table. Regenerate with make bench-consult.
Arm native¶
| run | send calls by Implementer | copies A / B | A saw B reply / B saw A reply | replies that reached Implementer | wall s | tokens Implementer | tokens Peer A | tokens Peer B | API calls I / A / B (all) | tokens all (in / cache write / cache read / out) | ToolSearch calls I / A / B | completed | timed out |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | SendMessage x2 (2) | 1 / 1 | no / no | none | 5.7 | 115327 | 76320 | 76347 | 3 / 2 / 2 (7) | 267994 (18 / 15772 / 250918 / 1286) | 0 / 0 / 0 | no | no |
| 2 | SendMessage x2 (2) | 1 / 1 | no / no | Peer A | 13.5 | 199990 | 159727 | 76449 | 5 / 4 / 2 (11) | 436166 (28 / 21650 / 411998 / 2490) | 1 / 1 / 0 | no | no |
| 3 | SendMessage x2 (2) | 1 / 1 | no / no | none | 5.0 | 115384 | 76472 | 76416 | 3 / 2 / 2 (7) | 268272 (18 / 15758 / 250956 / 1540) | 0 / 0 / 0 | no | no |
| median | 2 | 1 / 1 | 5.7 | 115384 | 7 | 268272 (18 / 15772 / 250956 / 1540) | 0 / 0 / 0 | 0 of 3 |
Median per session (tokens / API calls): Implementer 115384 / 3, Peer A 76472 / 2, Peer B 76416 / 2.
Implementer tool use per run: #1 ListAgents x1, SendMessage x2; #2 ListAgents x1, ToolSearch x1, SendMessage x2; #3 ListAgents x1, SendMessage x2.
Arm native-told¶
| run | send calls by Implementer | copies A / B | A saw B reply / B saw A reply | replies that reached Implementer | wall s | tokens Implementer | tokens Peer A | tokens Peer B | API calls I / A / B (all) | tokens all (in / cache write / cache read / out) | ToolSearch calls I / A / B | completed | timed out |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | SendMessage x2 (2) | 1 / 1 | no / no | Peer A, Peer B | 15.0 | 195523 | 157708 | 157726 | 5 / 4 / 4 (13) | 510957 (32 / 22050 / 486597 / 2278) | 0 / 1 / 1 | yes | no |
| 2 | SendMessage x2 (2) | 1 / 1 | no / no | Peer A, Peer B | 13.8 | 195287 | 157841 | 157803 | 5 / 4 / 4 (13) | 510931 (32 / 22223 / 486330 / 2346) | 0 / 1 / 1 | yes | no |
| 3 | SendMessage x2 (2) | 1 / 1 | no / no | Peer A, Peer B | 11.4 | 195237 | 157765 | 157803 | 5 / 4 / 4 (13) | 510805 (34 / 21986 / 486468 / 2317) | 0 / 1 / 1 | yes | no |
| median | 2 | 1 / 1 | 13.8 | 195287 | 13 | 510931 (32 / 22050 / 486468 / 2317) | 0 / 1 / 1 | 3 of 3 |
Median per session (tokens / API calls): Implementer 195287 / 5, Peer A 157765 / 4, Peer B 157803 / 4.
Implementer tool use per run: #1 ListAgents x1, SendMessage x2, Bash x1; #2 ListAgents x1, SendMessage x2, Bash x1; #3 ListAgents x1, SendMessage x2.
Arm murmur¶
| run | send calls by Implementer | copies A / B | A saw B reply / B saw A reply | replies that reached Implementer | wall s | tokens Implementer | tokens Peer A | tokens Peer B | API calls I / A / B (all) | tokens all (in / cache write / cache read / out) | ToolSearch calls I / A / B | completed | timed out |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | mcp__murmur__murmur_send x1 (1) | 1 / 1 | no (db no) / no (db no) | none | 8.7 | 80421 | 120803 | 120698 | 2 / 3 / 3 (8) | 321922 (20 / 16777 / 303586 / 1539) | 0 / 0 / 0 | yes | no |
| 2 | mcp__murmur__murmur_send x1 (1) | 1 / 1 | no (db no) / no (db no) | none | 10.1 | 121109 | 120855 | 120915 | 3 / 3 / 3 (9) | 362879 (22 / 17365 / 343788 / 1704) | 0 / 0 / 0 | yes | no |
| 3 | mcp__murmur__murmur_send x1 (1) | 1 / 1 | no (db no) / no (db no) | none | 11.5 | 121290 | 121172 | 163109 | 3 / 3 / 4 (10) | 405571 (24 / 18541 / 384811 / 2195) | 0 / 0 / 0 | yes | no |
| median | 1 | 1 / 1 | 10.1 | 121109 | 9 | 362879 (22 / 17365 / 343788 / 1704) | 0 / 0 / 0 | 3 of 3 |
Median per session (tokens / API calls): Implementer 121109 / 3, Peer A 120855 / 3, Peer B 120915 / 3.
Implementer tool use per run: #1 mcp__murmur__murmur_send x1; #2 mcp__murmur__murmur_status x1, mcp__murmur__murmur_send x1; #3 mcp__murmur__murmur_status x1, mcp__murmur__murmur_send x1.
Arm native: completed 0 of 3, median tokens 268272, median API calls 7, median wall time 5.7 s.
Arm native-told: completed 3 of 3, median tokens 510931, median API calls 13, median wall time 13.8 s.
Arm murmur: completed 3 of 3, median tokens 362879, median API calls 9, median wall time 10.1 s.