Context compaction, ported from pi's local summary method
(pi-agent-core/src/compaction/compaction.ts; omp writes it as
method: "soft"). When a request nears the model's context window, the
older part of the branch is summarized by the model into a compaction
entry (Rho.Session.compaction/4), and Rho.Session.context/1
sends that summary plus the recent tail from then on.
Pure functions; Rho.Loop makes the summary call.
- how big:
context_tokens/2is pi'scompactionContextTokens: the larger of the last reply's reported usage (plus an estimate of what came after it) and an estimate of the whole request. Estimates are bytes / 4 (pi's chars / 4, on ASCII), a picture 1,200 tokens (pi's), a PDF page 2,000. - when:
should_compact?/3, over the window minus pi's reserve (resolveThresholdTokensunder pi's default settings). - where:
prepare/2is pi'sprepareCompaction/findCutPoint: keep the newest whole messages that fitkeep_recent_tokens(the newest one even when it alone is bigger), cutting only before a user, assistant or notice message, never before a tool result, so a tool call is never separated from its result. - what:
summary_request/4is one call with pi's summarization prompts. After an earlier compaction its summary goes in as<previous-summary>with pi's update prompt, so the new summary carries it forward. pi's split-turn prefix (a second call) is folded into this one: everything before the cut is summarized together. overflow?/1: pi-ai's context-overflow error patterns.
Summary
Types
What one compaction summarizes: the message entries before the cut (oldest first), the id of the first entry kept, and the latest earlier compaction's summary, if any.
Functions
The context size to decide on (pi's compactionContextTokens): the
larger of the provider's count and the local estimate of request. The
provider's count is the last reply's usage (pi's calculateContextTokens)
plus an estimate of the entries after it, and only counts a reply after
the latest compaction: earlier ones measured a context that is gone.
The model's context window, from a small table: 200k for Claude models, 128k for anything else (low for most current models, so compaction comes early rather than too late). Not req_llm's catalog: its first lookup loads llm_db (1.4 s on a laptop), which would block the loop process.
Estimated tokens of text: bytes / 4, rounded up.
Did the provider reject the request for being over the context window?
pi-ai's overflow patterns (error/flags.ts) on the error's message.
What to compact (pi's prepareCompaction), or nil when nothing would
be summarized. Considers the message entries the context is built from
(after an earlier compaction: from its first kept entry on) and keeps
the newest ones that fit keep_recent_tokens. When the last reply
reports more prompt tokens than the local estimate, the budget shrinks
by that ratio, as pi does.
Estimated tokens of a whole request: system prompt, tool declarations, messages.
pi's shouldCompact: is a context of tokens over the threshold?
The summary call for plan (pi's generateSummary, one window): the
summarized messages serialized as pi does ([User]:, [Assistant]:,
[Tool Call]:, [Tool Result]: cut to 2,000 bytes) in
<conversation>, the previous summary in <previous-summary>, then pi's
prompt. A conversation too big for the summarizer's window keeps its
head (pi's clampConversationToBudget).
pi's resolveThresholdTokens under its default settings: the context
may grow to window minus a reserve for the next prompt and reply. The
reserve is the larger of 15% of the window and reserve_tokens (pi's
compaction.reserveTokens; nil means pi's default, 16,384); when that
default would leave no room (small windows) it falls back to the 15%.
Types
@type plan() :: %{ entries: [Rho.Session.entry()], first_kept_id: String.t(), previous_summary: String.t() | nil }
What one compaction summarizes: the message entries before the cut (oldest first), the id of the first entry kept, and the latest earlier compaction's summary, if any.
Functions
@spec context_tokens([Rho.Session.entry()], Rho.LLM.request()) :: non_neg_integer()
The context size to decide on (pi's compactionContextTokens): the
larger of the provider's count and the local estimate of request. The
provider's count is the last reply's usage (pi's calculateContextTokens)
plus an estimate of the entries after it, and only counts a reply after
the latest compaction: earlier ones measured a context that is gone.
@spec context_window(String.t()) :: pos_integer()
The model's context window, from a small table: 200k for Claude models, 128k for anything else (low for most current models, so compaction comes early rather than too late). Not req_llm's catalog: its first lookup loads llm_db (1.4 s on a laptop), which would block the loop process.
@spec estimate(String.t()) :: non_neg_integer()
Estimated tokens of text: bytes / 4, rounded up.
@spec overflow?(Rho.LLM.error()) :: boolean()
Did the provider reject the request for being over the context window?
pi-ai's overflow patterns (error/flags.ts) on the error's message.
@spec prepare([Rho.Session.entry()], pos_integer()) :: plan() | nil
What to compact (pi's prepareCompaction), or nil when nothing would
be summarized. Considers the message entries the context is built from
(after an earlier compaction: from its first kept entry on) and keeps
the newest ones that fit keep_recent_tokens. When the last reply
reports more prompt tokens than the local estimate, the budget shrinks
by that ratio, as pi does.
@spec request_tokens(Rho.LLM.request()) :: non_neg_integer()
Estimated tokens of a whole request: system prompt, tool declarations, messages.
@spec should_compact?(non_neg_integer(), non_neg_integer(), pos_integer() | nil) :: boolean()
pi's shouldCompact: is a context of tokens over the threshold?
@spec summary_request(plan(), String.t(), pos_integer(), pos_integer()) :: Rho.LLM.request()
The summary call for plan (pi's generateSummary, one window): the
summarized messages serialized as pi does ([User]:, [Assistant]:,
[Tool Call]:, [Tool Result]: cut to 2,000 bytes) in
<conversation>, the previous summary in <previous-summary>, then pi's
prompt. A conversation too big for the summarizer's window keeps its
head (pi's clampConversationToBudget).
@spec threshold(pos_integer(), pos_integer() | nil) :: non_neg_integer()
pi's resolveThresholdTokens under its default settings: the context
may grow to window minus a reserve for the next prompt and reply. The
reserve is the larger of 15% of the window and reserve_tokens (pi's
compaction.reserveTokens; nil means pi's default, 16,384); when that
default would leave no room (small windows) it falls back to the 15%.