Rho.LLM.ReqLLM (rho v0.1.0)

Copy Markdown View Source

The model calls through req_llm's providers, streamed, with the provider's credential from Rho.Auth (fetched, and for a subscription refreshed when due, per call):

  • anthropic: signed in to a subscription (Claude Pro/Max) → auth_mode: :oauth, access_token, with_claude_subscription: true (Claude Code's betas and identity); with an API key → api_key. Either way prompt caching (anthropic_prompt_cache: true, anthropic_cache_messages: -1): req_llm puts a cache_control breakpoint on the last tool, the system prompt and the last message, so each call reads the previous one's prefix from the cache instead of paying (and spending rate limit on) the whole prompt again
  • openai_codex: (ChatGPT Plus/Pro) → auth_mode: :oauth, access_token, chatgpt_account_id, and the session id as session_id (the prompt cache key)
  • openai: (an API key) → api_key, and the session id as prompt_cache_key
  • google: (a Gemini API key) → api_key, sent in the x-goog-api-key header (google_auth_header: true) rather than the URL, so it stays out of logged URLs

A provider without a credential fails with {:signed_out, provider}; any other model prefix with {:other, _}. max_tokens is always explicit (req_llm's default is the model's whole output limit).

Pictures the model makes come back as the reply's :images: the :image content parts (bytes and media_type) of the response's message. That message already holds the streamed :content_part chunks (req_llm's response builder materializes them; preview frames, stream_only?, it leaves out), so each picture counts once. An :image_url part (a link, not bytes) is skipped: rho doesn't fetch it.

Every call that reached the provider is recorded with Rho.Usage.record/3: its tokens and cost, the subscription windows its response headers report, and a 429's reset.

A 429 (or a 529 overloaded) comes back as {:rate_limited, limit} (Rho.LLM.limit/0): its retry-after, and when a usage window is used up, which one and until when: from the response's headers, a Codex usage-limit body, else the windows Rho.Usage recorded from earlier calls (an Anthropic 5-hour window at 100 % answers with the same message as a short throttle).