Rho.Dyn.Compiler (rho v0.1.0)

Copy Markdown View Source

Compiles a source set into generation n (part of the pipeline's step 2, see Rho.Dyn) and loads, unloads and purges generations' modules.

Versioned names. The agent writes Dyn.Foo (the namespace, Rho.Dyn.Config); rewrite/2 turns every Dyn.* alias in the AST (aliases, module references, template sigils such as ~MOB) into Dyn.G<n>.*, so generation n loads next to the running one and never over it: loading a new version of a module the BEAM runs, then loading again, would kill every process still in the old code. __MODULE__ is the versioned name, as it should be.

Containment. The compile runs in its own process with a timeout and a max_heap_size (module bodies run at compile time). If it fails, every module of generation n it loaded is purged: nothing runs them. Its errors are formatted in that process (Rho.Untrusted.format/4) and its diagnostics come back as plain strings; if it dies, its exit reason is only described (Rho.Untrusted.exit_text/2).

Screens report to the Keeper. Each screen module (as the host's Rho.Dyn.Screens tells them: mob's use Mob.Screen with a mount/3) gets a wrapping entry function (mount/3) that registers the screen process with Rho.Dyn.Keeper.watch/3 before calling the original, so the Keeper counts its crashes and knows the generation is in use.

Unloading an old generation (unload/1) happens only when no process runs its code: no module of it is anywhere on a process's stack or its initial call; then :code.delete/1, and :code.soft_purge/1 (which refuses while :erlang.check_process_code/2 finds a process in the old code); anything still in it is purged later (purge_old/1).

Summary

Types

A compiled generation: its binaries, how long the compile took, its warnings, its compile dependencies (Rho.Dyn.Reuse) and the files reused from the parent rather than compiled.

Functions

Checks sources and compiles them into generation n. Options: :keeper (the Keeper screens report to), :src_dir (absolute dir the sources live in, for file names and diagnostics), :compile_timeout_ms, :compile_max_heap_mb, :reuse (%{n: parent, deps: parent's deps, beams: %{"Elixir.Dyn.G<parent>.X" => binary}, unchanged: MapSet of files whose source is the parent's}: compile incrementally, Rho.Dyn.Reuse).

Compiles checked files (Rho.Dyn.Check.run/1's result) into generation n.

A digest of the Core's code as this VM runs it: every module below a Core prefix (Rho.* and the host's, Rho.Dyn.Config's :core) except the Dyn generations' (the namespace's), by the MD5 of its loaded code, or of its .beam on the code path while it isn't loaded (what will load). The modules are the :rho and host (:app) applications' plus every loaded one below a Core prefix (a delivered update can add modules); without an application's .app, every module below a Core prefix on the code path. A delivered Core (mob_deliver loads it at launch, before the Dyn layer boots) counts with its delivered code.

The generation a versioned module belongs to, or nil for any other module. A loaded protocol implementation for a Dyn module, which Elixir names <Protocol>.<module> (Inspect.Dyn.G3.Card, from a defimpl or a library's use), belongs to its module's generation: what says so is its own __impl__ attribute (protocol and target, and the name they make), not the name alone. The build only lets one through as that (Rho.Dyn.Check.beam/2).

Loads binaries; one already loaded with identical code is skipped.

Loads generation gen from its stored binaries (checked against the manifest's hashes). Binaries built by another runtime (an app update since, stale?/1) are rebuilt from the sources and selftested again before they load (rebuild/3, then store_rebuild/4). Modules already loaded with the same code are left alone.

Every loaded module of generation n, its protocol implementations (Inspect.Dyn.G<n>.Card) included.

The name the agent wrote: Dyn.G3.Notes → "Dyn.Notes", a loaded Inspect.Dyn.G3.Card (an implementation, see generation_of/1) → "Inspect.Dyn.Card".

Fully unloads modules nothing runs (a candidate's): purge, delete, purge; what the loop kept of their tool declarations goes too.

Purges the old code of mods that no process executes (:code.soft_purge/1, which asks :erlang.check_process_code/2); returns the rest.

Rebuilds generation gen from its stored sources for this runtime and selftests it: its modules are loaded, nothing is stored (that's store_rebuild/4), and a failure leaves none of them loaded. Touches only the code server, so it can run outside the Keeper (it does at launch: Rho.Dyn.Keeper).

Rewrites Dyn.* (the namespace) to Dyn.G<n>.* throughout ast.

Identifies the runtime binaries were built by: OTP, Elixir, the host app's version (Rho.Dyn.Config's :app, else rho's) and the Core's code (core_digest/0), fixed for the VM's lifetime. A Core release, by cable or over the air, changes it from the next launch on, so every generation built before is rebuilt and selftested against the new Core when it loads.

Were gen's stored binaries built by another runtime (runtime/0)?

Stores rebuild/3's binaries and selftest results for gen: the manifest gets the new hashes and this runtime, and a proven generation goes back on probation (its code is new). The caller unloads the modules on an error.

Unloads an old generation's modules if no process runs them. :in_use leaves everything loaded; {:ok, stuck} deleted them all and purged all but stuck, whose old code a process is still executing (retry with purge_old/1).

Types

build()

@type build() :: %{
  modules: [{module(), binary()}],
  compile_ms: non_neg_integer(),
  warnings: [String.t()],
  deps: Rho.Dyn.Reuse.deps(),
  reused: [String.t()]
}

A compiled generation: its binaries, how long the compile took, its warnings, its compile dependencies (Rho.Dyn.Reuse) and the files reused from the parent rather than compiled.

failure()

@type failure() :: {:check, [Rho.Dyn.Check.violation()]} | {:compile, String.t()}

Functions

build(sources, n, opts)

@spec build(%{required(String.t()) => String.t()}, pos_integer(), keyword()) ::
  {:ok, build()} | {:error, failure()}

Checks sources and compiles them into generation n. Options: :keeper (the Keeper screens report to), :src_dir (absolute dir the sources live in, for file names and diagnostics), :compile_timeout_ms, :compile_max_heap_mb, :reuse (%{n: parent, deps: parent's deps, beams: %{"Elixir.Dyn.G<parent>.X" => binary}, unchanged: MapSet of files whose source is the parent's}: compile incrementally, Rho.Dyn.Reuse).

compile(parsed, n, opts)

@spec compile([{String.t(), Macro.t()}], pos_integer(), keyword()) ::
  {:ok, build()} | {:error, {:compile, String.t()}}

Compiles checked files (Rho.Dyn.Check.run/1's result) into generation n.

Compiling loads what it compiles at once, VM-wide, before any check of the result can run. So the compile process's tracer refuses a module outside the generation as its definition starts (Rho.Dyn.Trace.guard/3), and reports every module this compile defines, with what was loaded under that name before. Afterwards, whatever the outcome, each one other than the generation's modules and its own protocol implementations (Rho.Dyn.Check.generation_module?/4) is undone: a new one unloaded, a replaced one reloaded from its file. A module the compile returned that the tracer never saw (Module.create/3 with a keyword list, not an env, traces nothing) and that isn't the generation's is undone too: unloaded, and reloaded from the code path when a file for it is there. Modules other processes define meanwhile (an eval, a subagent) are left alone. Rho.Dyn.Check.beam/2 then rejects a build that had any other.

core_digest()

@spec core_digest() :: String.t()

A digest of the Core's code as this VM runs it: every module below a Core prefix (Rho.* and the host's, Rho.Dyn.Config's :core) except the Dyn generations' (the namespace's), by the MD5 of its loaded code, or of its .beam on the code path while it isn't loaded (what will load). The modules are the :rho and host (:app) applications' plus every loaded one below a Core prefix (a delivered update can add modules); without an application's .app, every module below a Core prefix on the code path. A delivered Core (mob_deliver loads it at launch, before the Dyn layer boots) counts with its delivered code.

generation_of(mod)

@spec generation_of(module()) :: pos_integer() | nil

The generation a versioned module belongs to, or nil for any other module. A loaded protocol implementation for a Dyn module, which Elixir names <Protocol>.<module> (Inspect.Dyn.G3.Card, from a defimpl or a library's use), belongs to its module's generation: what says so is its own __impl__ attribute (protocol and target, and the name they make), not the name alone. The build only lets one through as that (Rho.Dyn.Check.beam/2).

load(beams)

@spec load([{module(), binary()}]) :: :ok | {:error, String.t()}

Loads binaries; one already loaded with identical code is skipped.

load_generation(root, gen, opts)

@spec load_generation(Path.t(), Rho.Dyn.Generation.t(), keyword()) ::
  {:ok, [module()], Rho.Dyn.Generation.t()} | {:error, String.t()}

Loads generation gen from its stored binaries (checked against the manifest's hashes). Binaries built by another runtime (an app update since, stale?/1) are rebuilt from the sources and selftested again before they load (rebuild/3, then store_rebuild/4). Modules already loaded with the same code are left alone.

loaded(n)

@spec loaded(pos_integer()) :: [module()]

Every loaded module of generation n, its protocol implementations (Inspect.Dyn.G<n>.Card) included.

logical(mod)

@spec logical(module()) :: String.t()

The name the agent wrote: Dyn.G3.Notes → "Dyn.Notes", a loaded Inspect.Dyn.G3.Card (an implementation, see generation_of/1) → "Inspect.Dyn.Card".

purge(mods)

@spec purge([module()]) :: :ok

Fully unloads modules nothing runs (a candidate's): purge, delete, purge; what the loop kept of their tool declarations goes too.

purge_old(mods)

@spec purge_old([module()]) :: [module()]

Purges the old code of mods that no process executes (:code.soft_purge/1, which asks :erlang.check_process_code/2); returns the rest.

rebuild(root, gen, opts)

@spec rebuild(Path.t(), Rho.Dyn.Generation.t(), keyword()) ::
  {:ok, build(), [Rho.Dyn.Generation.selftest()]} | {:error, String.t()}

Rebuilds generation gen from its stored sources for this runtime and selftests it: its modules are loaded, nothing is stored (that's store_rebuild/4), and a failure leaves none of them loaded. Touches only the code server, so it can run outside the Keeper (it does at launch: Rho.Dyn.Keeper).

rewrite(ast, n)

@spec rewrite(Macro.t(), pos_integer()) :: Macro.t()

Rewrites Dyn.* (the namespace) to Dyn.G<n>.* throughout ast.

runtime()

@spec runtime() :: String.t()

Identifies the runtime binaries were built by: OTP, Elixir, the host app's version (Rho.Dyn.Config's :app, else rho's) and the Core's code (core_digest/0), fixed for the VM's lifetime. A Core release, by cable or over the air, changes it from the next launch on, so every generation built before is rebuilt and selftested against the new Core when it loads.

stale?(generation)

@spec stale?(Rho.Dyn.Generation.t()) :: boolean()

Were gen's stored binaries built by another runtime (runtime/0)?

store_rebuild(root, gen, build, tests)

@spec store_rebuild(Path.t(), Rho.Dyn.Generation.t(), build(), [
  Rho.Dyn.Generation.selftest()
]) ::
  {:ok, Rho.Dyn.Generation.t()} | {:error, String.t()}

Stores rebuild/3's binaries and selftest results for gen: the manifest gets the new hashes and this runtime, and a proven generation goes back on probation (its code is new). The caller unloads the modules on an error.

unload(mods)

@spec unload([module()]) :: :in_use | {:ok, [module()]}

Unloads an old generation's modules if no process runs them. :in_use leaves everything loaded; {:ok, stuck} deleted them all and purged all but stuck, whose old code a process is still executing (retry with purge_old/1).

"Runs them" looks at each process's whole stack (its backtrace), not only the function it is in: a process sleeping in Process.sleep/1, called from a Dyn function, returns into that function.