The analysis pipeline
Which LLM calls a clause goes through on its way to a reviewable IR, which code checks run after each step, and how failures are repaired.
LLM analysis stages
Analysis works at two levels. First the whole document is split into tasks; a task is one part of the text that can be judged on its own. Then each task runs the stages below, with up to 4 tasks of a document in flight at once. The analysis progress shows which stage each task has reached.
Once per document
- 1Split into tasks 拆解任務
extract_outlineThe model returns only the first 10–20 characters of each task; code then cuts the source at those points, so every task’s text is verbatim. With no LLM configured, or if this step fails, the text is split by its article headings instead.
Once per task
- 2Triage 分類
triageLabels the clause simple or complex, with a confidence from 0 to 1 and a one-line reason.
- 3Extract quotes 擷取引文
ir_unitsCopies each supporting passage verbatim from the source, numbered S1, S2 and so on.
- 4Extract conditions 擷取條件
ir_nodesDerives atomic conditions C1, C2 and so on from the quotes. Each condition is one yes/no question a physician would ask, and cites at least one quote.
- 5Build the logic 建立邏輯
ir_groupsJoins the conditions with AND, OR and NOT groups and picks the root group. This step only arranges conditions; it neither adds nor rewrites them.
- 6Audit omissions and grouping 覆核（遺漏與群組）
ir_pass2One look at the whole IR: conditions the text states but nobody extracted, polarity, and the group structure. Atomic conditions that are already correct are not reworded.
- 7Per-condition cross-check 逐條件覆核
ir_node_reviewOne small call per condition against the text it cites. See per-condition cross-check.
- 8Round-trip check 反向驗證
roundtripWrites the generated CQL back as a plain-language summary and compares its meaning with the source. If the call fails, the result is only recorded as undetermined; the analysis is unaffected.
- 9Compile and ablation checkno LLM call
cql-engine compiles the CQL and runs the ablation check, overriding the parameters one at a time. The result is the executability evidence on the review page.
Triage is used in two places: a clause judged simple with confidence ≥ 0.8 skips the whole-IR audit (ir_pass2), still gets the per-condition cross-check, and is labeled LLM 自動通過 · 可推翻 (LLM auto-pass, can be overridden) on the review page; every other clause is labeled 需醫師簽核 (physician sign-off required). Neither label changes the clause status; sign-off is still done by people. If the triage call fails, the clause is treated as complex, so an LLM error never removes an audit.
Code checks and targeted repair
A code check runs after every stage, and the next stage starts only once it passes. When a check fails, only the failing part goes back to the model; nothing else re-runs:
| Check | Repair call | What it repairs |
|---|---|---|
| Every quote is found in the source | ir_fix_units | Rewrites only the quotes not found |
| Each condition, checked on its own, has a valid definition | ir_fix_node | Fixes only the failing condition |
| Groups and the root are valid, and every condition and group is reachable from the root | ir_fix_groups | Rearranges only the groups |
| Errors that need restructuring, such as an AND or OR hidden inside one condition | ir_fix | Repairs the whole IR once |
Some slips are fixed in code without asking the model: group operators written as and or or, or a group written G2 instead of GROUP:G2. When one condition packs “A and B” or “A or B” and every part is a complete atomic condition, code splits it into two conditions joined by a group. NOT is never split automatically; the scope of a negation is left to the model.
Each task gets at most 4 repair calls in total, shared by all stages. If the structure is still invalid after that, the task is marked failed with its error and the other tasks finish normally. If only quotes are still unmatched, the result is kept; see quote verification.
When a response doesn’t parse as JSON, the parser’s error message and the text around the failure point go back to the model with a request to output it again, up to 2 times by default (LLM_JSON_RETRIES). Code itself only applies repairs that can’t change the content: removing markdown fences, trailing commas, and unescaped line breaks inside strings. Truncated output is never closed off with extra brackets: a half-parsed IR can silently drop conditions and still pass validation. Instead the model is asked for shorter, complete JSON.
If the input is gibberish, empty or unrelated to clinical criteria, the model reports that it can’t be decomposed and says why. That is a valid answer, not a parse failure, so it is not retried.
Quote verification
Quotes are what tie a condition to the source: each condition’s source_ids point to one or more quotes, and each quote’s exact_quote must appear verbatim in the clause text. This is how reviewers can confirm a condition wasn’t made up by the model.
Before matching, the quote and the source get the same treatment: all whitespace is removed, full-width letters, digits and symbols become half-width, the Chinese full stop and enumeration comma count as a period and a comma, and letter case is ignored. One extra period, comma or semicolon at the end of a quote is also accepted. Otherwise the text must match exactly.
Take the example clause from the CQL primer:
本品限內科專科醫師處方，病人需為成人（年滿十八歲），且 eGFR ≥ 30 mL/min/1.73m²；未達標準者，須經腎臟科會診同意後使用。Translation: Must be prescribed by an internal medicine specialist. The patient must be an adult (18 or older) with eGFR ≥ 30 mL/min/1.73m². Below that threshold, use requires nephrology consultation approval.
| Quote | Result | Why |
|---|---|---|
| eGFR≥30mL/min/1.73m² | Pass | Whitespace doesn’t count. |
| 病人需為成人(年滿十八歲) | Pass | The source uses full-width parentheses and the quote half-width; they count as the same. |
| 限內科專科醫師處方。 | Pass | One extra full stop at the end is ignored. |
| 須由內科醫師開立 | Pending | The source never says this; the model paraphrased it. |
Quotes are matched right after extraction, and the ones not found go to ir_fix_units. Every later round of checks matches them again.
If a quote still doesn’t match once the repair budget is spent, the analysis doesn’t fail: the logic has passed its checks, so the IR and CQL are kept and only those quotes are flagged. The workbench task list shows N 筆引文待核 (N quotes pending review), and the review page shows the quote under the condition’s source text with 出處 S5 在原文中找不到逐字對應 (source S5 has no verbatim match in the text). That warning means the condition may rest on text the model supplied itself.
Human edits follow a stricter rule: a quote a reviewer adds or rewrites that isn’t in the source is rejected on save. See editing logic.
Per-condition cross-check
The per-condition cross-check (ir_node_review) is the last LLM stage that changes the IR. Once the whole IR is settled, each condition gets its own small call, up to 3 at a time per task. Each call carries only three things: the clause text, the quotes this condition cites, and where it sits in the AND, OR and NOT structure.
The model checks four things for each condition:
- Faithful
- The label, clinical question and definition say exactly what the quote says, no stricter and no looser. Anything the quote doesn’t say is removed.
- Atomic
- A condition asks one thing. If it is really two, the definition is left as is and a note is added.
- Normalized
- Thresholds become numbers with their unit spelled out, enumerations become lists, durations and doses carry units, and drug names stay as written.
- Evidence
- It fills in the FHIR R4 resource and element that would prove or refute the condition, for use by the evidence contract.
It runs last so that what it adds isn’t overwritten by a whole-IR rewrite. It runs per condition because this job used to be one whole-IR pass, which was the slowest call in the pipeline and tended to merge atomic conditions back into compound ones. Small calls run in parallel, and when one condition is in doubt, only that one has to be re-run.
The model can’t change a condition’s ID or the sources it cites. The revised condition is checked again on its own: if the model tries to turn it into a compound condition, or the revision fails the check, the original is kept and a note is recorded. If the call for one condition fails, that too is only a note, and the other conditions are still checked.
Doubts the model raises, such as ambiguous wording or a threshold it had to interpret, are stored as notes and shown under 模型覆核註記 (model review notes) for that condition on the review page. Notes are for the reviewer’s information and don’t affect the verdict.
If a condition looks doubtful during review, choose 用 LLM 重新分析此條件 (re-analyze this condition with the LLM) from its menu to run the cross-check again for that condition alone. The result is saved as a logic edit by you, and the condition’s review decision is invalidated only if it actually changed. See sign-off and invalidation.
This is the model checking itself, not review: physicians do the per-condition review on the review page, where a person accepts, rejects or marks each condition not applicable.
Call log
Every LLM call is logged: the stage, the profile and model used, how long it took, time spent queued, and the prompt and response verbatim. Admins can open a document analysis under 管理 → 背景作業 (Admin → Background jobs) and step through the calls by task and stage. Full prompt and response text is kept for 14 days; after that only the statistics remain.
Every stage’s prompt can be edited under 管理 → Prompt 範本 (Admin → Prompt templates); a saved version replaces the built-in one. For setting up model connections, see the admin console.