Script
Here is today's Pi Project Update for Thursday July 30th. The lead story is a major step forward for Pi’s evaluation workflow. A merged change today adds a comparative Pi eval harness for the coding agent. It supports seeded, repeated comparisons across harnesses, reports score lift, and separates token, latency, and cost deltas. It also persists run artifacts and session snapshots, which should make it much easier to judge whether a prompt, provider, or extension change actually improves agent behavior instead of just feeling better in one test run. [1]
Provider correctness also improved. Yesterday’s fixes preserve original provider stop reasons across Anthropic, Bedrock, Google, Mistral, OpenAI-compatible, and Responses adapters. That gives Pi better raw diagnostic information when a model stops for a provider-specific reason. Another OpenAI-compatible fix preserves valid function-call arguments when a provider also emits an empty custom payload, preventing tool calls from being finalized with a blank `{ input: "" }` shape. Together, these are small but important reliability wins for tool-heavy sessions. [2]
On the interactive coding-agent side, Pi merged a cleanup today for `setToolsExpanded(false)`. Extensions that enforce collapsed tool output at startup will no longer produce a redundant “Tool output: collapsed” status when nothing changed. Yesterday, Pi also improved startup context display so file-backed `SYSTEM.md` and `APPEND_SYSTEM.md` entries are visible in the interactive Context section, making it clearer what instructions are actually in play. [3]
The issue queue shows where attention is going next: terminal stability, scrollback behavior, paste handling, extension failure isolation, provider retries, and safer defaults. Reports covered Windows Terminal scroll jumps, Termux multiline paste, broken extensions aborting startup, transient model catalog refresh failures, and requests for installable destructive-command guards. [4]
The broader trend is that Pi is moving from feature growth into operational hardening: better measurement, clearer provider errors, less noisy UI behavior, and more resilience around terminals and extensions. Thank you for listening to Pi Project Update from The Daily FM. See you tomorrow!
- earendil-works/pi: development activity
...feat(coding-agent): add comparative Pi eval harness (merged 2026-07-30) ## Summary - add seeded, repeated multi-harness comparisons with score lift and separate token, latency, and cost deltas - snapshot native Pi sessions before temporary cleanup, register artifacts against the explicit Vitest test task, and index persisted runs in `runs.jsonl` - support per-harness model selection and system-prompt treatments - compare extension authoring with and without the full default Pi prompt using a non-gating deterministic judge - load extension dependencies correctly from TypeScript source, built Node, and Bun runtimes ## Testing - `npm run check` passes, with two unrelated existing TUI style infos - `npm test --workspace=@earendil-works/pi-evals` passes: 28 tests - coding-agent extension suites pass: 66 tests - `./test.sh` reaches the eval suite successfully, but exits...
- earendil-works/pi: development activity
...: preserve function arguments with empty custom payloads (merged 2026-07-29) Prefer valid function tool-call payloads when an OpenAI-compatible provider also emits an empty `custom` object. This prevents custom-tool finalization from replacing parsed function arguments with `{ "input": "" }`. Patch by @sunnyyoung. Fixes #7160 Files: packages/ai/src/api/openai-completions.ts (+3/-4), packages/ai/test/openai-completions-tool-choice.test.ts (+49/-0) #7272 preserve providers raw stop reason (merged 2026-07-29) Adds `AssistantMessage.rawStopReason` and preserves provider original stop reasons. Improves certain cases of error messages. For mistral, change unmapped finish reasons to be "error" instead of "stop". fixes #7255 Files: packages/ai/src/api/anthropic-messages.ts (+2/-1), packages/ai/src/api/bedrock-converse-stream.ts (+4/-1), packages/ai/src/api/google-gene... - earendil-works/pi: development activity
...d. Reproduced. Fixed by making `setToolsExpanded()` return early when the requested state matches the current state, so startup extensions can safely enforce collapsed tools without noisy UI updates. The repro/regression test now passes. Files: packages/coding-agent/CHANGELOG.md (+4/-0), packages/coding-agent/src/modes/interactive/interactive-mode.ts (+2/-0) #7289 feat(coding-agent): add comparative Pi eval harness (merged 2026-07-30) ## Summary - add seeded, repeated multi-harness comparisons with score lift and separate token, latency, and cost deltas - snapshot native Pi sessions before temporary cleanup, register artifacts against the explicit Vitest test task, and index persisted runs in `runs.jsonl` - support per-harness model selection and system-prompt treatments - compare extension authoring with and without the full default Pi prompt using a non-gati...
- earendil-works/pi: development activity
...dels` succeeded 3/3 times in about three seconds. ### Steps to reproduce 1. Use a network path where an individual `pi.dev/api/models/providers/*` request can transiently stall. 2. Run `pi update --models`. 3. One stalled request causes the global 15-second refresh timeout. A direct curl test showed 4/10 requests succeeding and 6/10 stalling after TLS, demonstrating that retrying is necessary on this network path. ### Expected behavior Retry transient catalog request failures within the existing outer refresh deadline. Aborting the outer signal should immediately stop all attempts. ### Version Pi 0.83.0, Node 26.5.0, macOS 26.6 arm64 #7322 Single broken extension aborts pi startup with process.exit(1) instead of degrading gracefully (closed 2026-07-30) ## Summary When a single extension fails to load (syntax error, parse error, runtime error during factory ex...