Skip to main content

RLM Query

Category: AI & ML
Node id: rlm-query
Maturity: beta
Risk tier: B

Executes Recursive Language Model queries: context is offloaded into a persistent Python REPL and the root model iteratively probes, chunks, and delegates it to sub-LLM calls instead of receiving it in the prompt

What it does​

  • Near-arbitrary context length via REPL offloading (context never enters the root prompt)
  • Multi-turn orchestration with in-REPL sub-LLM calls (llm_query, llm_query_batched)
  • Dual-model architecture (root orchestrator + cheaper sub-call model)
  • Answer-dict sentinel termination with partial answers on turn/time/budget exhaustion
  • FHIR navigation helpers inside the REPL for structured clinical contexts
  • Optional per-turn execution trace for auditability

Use cases​

  • Whole-bundle FHIR analysis that exceeds a single context window
  • Cross-document aggregation over large batches (e.g. many patient summaries)
  • Complex question answering requiring programmatic decomposition and delegation

Runtime​

  • Performance tier: slow

Input ports​

NameTypeRequiredDescription
querytextNoQuery to execute (overrides config query)
contextanyNoContext data offloaded into the REPL (text or JSON; JSON stays structured)

Output ports​

NameTypeRequiredDescription
resultjsonNoRLM execution result: answer, status (completed | max_turns | timeout | budget_exhausted | error), turns, sub_calls, tokens_in, tokens_out, duration_ms, and trace when include_trace is set
successbooleanNoTrue when the model submitted an answer via the answer-dict sentinel (status completed)

Settings​

Basic​

SettingNameTypeRequiredDefaultAllowed valuesDescription
ProviderproviderenumNoopenaiopenai, anthropic, vllm, localModel provider for in-process execution
Root Modelroot_modelstringNogpt-4oOrchestrating model that writes REPL code
Sub Modelsub_modelstringNogpt-4o-miniModel used by llm_query sub-calls (defaults to root_model when empty)
QueryquerytextNoStatic query (overridden by input)

Auth​

SettingNameTypeRequiredDefaultAllowed valuesDescription
API Keyapi_keysecretNoAPI key for the provider (or the external service in legacy mode)
API Base URLapi_base_urlstringNoOpenAI-compatible base URL; required for vllm/local providers

Advanced​

SettingNameTypeRequiredDefaultAllowed valuesDescription
Context Typecontext_typestringNotextHow the context input is loaded: text or json
Max Turnsmax_turnsnumberNo20min 1; max 100Maximum root-model reasoning turns
Max Execution Time (s)max_execution_timenumberNo300min 10; max 600Wall-clock budget for the whole run
TemperaturetemperaturenumberNo0.7min 0; max 2Root model sampling temperature
Include Traceinclude_tracebooleanNofalseAttach the per-turn execution trace (code + truncated outputs; never the raw context) to the result
External RLM Endpoint (legacy)llm_endpointstringNoWhen set, delegates execution to POST {llm_endpoint}/rlm/execute instead of running in-process
Prompt Type (legacy)prompt_typestringNogeneralForwarded to the external service in legacy mode; the in-process executor uses the baseline orchestrator prompt

Errors​

CodeDescription
QUERY_REQUIREDNo query provided via input or config
CLIENT_BUILD_FAILEDModel client could not be constructed from provider/model/api settings
ENV_START_FAILEDThe Python REPL environment could not be started (no python3/python on PATH)
RLM_EXECUTION_FAILEDThe RLM run failed before producing a result
HTTP_ERRORLegacy mode: external RLM service returned an HTTP error status

Runtime notes​

  • In-process mode (default) runs a persistent local Python subprocess as the REPL: model-generated code executes with OS-process isolation only. Do not point it at PHI-bearing contexts until the sandboxed environment ships; see docs/architecture/rlm-on-hydra-design.md
  • Setting llm_endpoint switches to legacy external delegation (POST {llm_endpoint}/rlm/execute) and ignores provider/api_base_url
  • Termination is the REPL answer-dict sentinel, never string parsing of model output; non-completed statuses still return partial answers
  • Sub-LLM calls are capped per turn and bounded by max_execution_time overall
  • The query input takes precedence over the static query in config

See Build any workflow for how to put nodes together and Ports, types and data flow for how port types connect.

Something unclear? Send documentation feedback by email.