Rho.LLM behaviour (rho v0.1.0)

Copy Markdown View Source

The model client the loop talks to. Rho.LLM.ReqLLM is the real one (Anthropic, OpenAI, OpenAI Codex and Google through req_llm, with the credential from Rho.Auth); tests inject a scripted fake.

stream/3 runs in a process the loop owns (and kills on Stop). It calls sink for every streamed delta and returns the whole reply:

  • events: {:text, delta}, {:thinking, delta}
  • {:ok, %{text:, thinking:, tool_calls: [%{"id", "name", "arguments"}], usage: map | nil, finish_reason: atom}}, and optionally images: [{mime_type, bytes}], pictures the model made (Rho.Session.assistant/2 keeps them in the session; they aren't sent back to the model)
  • {:error, error} where error is normalized: {:http, status, message}, {:rate_limited, limit} (a 429, or a 529 overloaded, with what the response and Rho.Usage say about when it lifts: see limit/0), {:transport, reason}, {:signed_out, provider} (no credential for the model's provider), {:auth, provider, message} (its token couldn't be refreshed) or {:other, message}

describe/1's remedies say where the user signs in or switches model as the host words it (Rho.Config.hint/1).

Two kinds of 429 need telling apart (Rho.Loop does): a short throttle ("would exceed your account's rate limit"), worth waiting out, and a subscription window used up for hours (a window at 100 %, a Codex "usage limit has been reached"), where waiting is pointless: the limit's resets_at says which.

Summary

Types

error()
limit()

A rate limit: retry_after_ms from the response's retry-after(-ms), and resets_at (unix s) / window (its label, e.g. "5 hours", or nil) when a usage window is used up; provider is nil when unknown.

reply()
request()

Callbacks

stream(request, opts, sink)

Functions

clock(at, now \\ System.os_time(:second))

Unix time at on the device's clock: HH:MM today, Mon D HH:MM on another day. now (unix s) is for tests.

describe(arg)

What the user sees for a failed call, with the remedy where there is one.

rate_limit(arg1)

The rate limit an error is (a 429 or a 529 overloaded, with what is known about it), or nil for any other error.

retryable?(arg1)

Worth retrying: rate limits, provider 5xx, dropped connections.

usable(model, dir \\ Rho.Config.data_dir(), now \\ System.os_time(:second))

Can model take over a run another model's limit stopped: its provider signed in, and none of its usage windows used up for more than a minute (Rho.Usage)? :ok or {:error, why}.

Types

error()

@type error() ::
  {:http, integer(), String.t()}
  | {:rate_limited, limit()}
  | {:transport, term()}
  | {:signed_out, Rho.Auth.provider()}
  | {:auth, Rho.Auth.provider(), String.t()}
  | {:other, String.t()}

limit()

@type limit() :: %{
  status: integer(),
  message: String.t(),
  provider: Rho.Auth.provider() | nil,
  retry_after_ms: non_neg_integer() | nil,
  resets_at: integer() | nil,
  window: String.t() | nil
}

A rate limit: retry_after_ms from the response's retry-after(-ms), and resets_at (unix s) / window (its label, e.g. "5 hours", or nil) when a usage window is used up; provider is nil when unknown.

reply()

@type reply() :: %{
  :text => String.t(),
  :thinking => String.t(),
  :tool_calls => [map()],
  :usage => map() | nil,
  :finish_reason => atom() | nil,
  optional(:images) => [{String.t(), binary()}]
}

request()

@type request() :: %{
  :model => String.t(),
  :system_prompt => String.t(),
  :messages => [ReqLLM.Message.t()],
  :tools => [ReqLLM.Tool.t()],
  :max_tokens => pos_integer(),
  optional(:session_id) => String.t()
}

Callbacks

stream(request, opts, sink)

@callback stream(request(), opts :: term(), sink :: (term() -> any())) ::
  {:ok, reply()} | {:error, error()}

Functions

clock(at, now \\ System.os_time(:second))

@spec clock(integer(), integer()) :: String.t()

Unix time at on the device's clock: HH:MM today, Mon D HH:MM on another day. now (unix s) is for tests.

describe(arg)

@spec describe(error()) :: String.t()

What the user sees for a failed call, with the remedy where there is one.

rate_limit(arg1)

@spec rate_limit(error()) :: limit() | nil

The rate limit an error is (a 429 or a 529 overloaded, with what is known about it), or nil for any other error.

retryable?(arg1)

@spec retryable?(error()) :: boolean()

Worth retrying: rate limits, provider 5xx, dropped connections.

usable(model, dir \\ Rho.Config.data_dir(), now \\ System.os_time(:second))

@spec usable(String.t(), Path.t(), integer()) :: :ok | {:error, String.t()}

Can model take over a run another model's limit stopped: its provider signed in, and none of its usage windows used up for more than a minute (Rho.Usage)? :ok or {:error, why}.