Consultation: full text screening by a single language model reviewer (LivingGLP1)

Paths relative to /Users/choxos/Documents/GitHub/LivingGLP1. Do not edit, create or delete any file. Answer in at most 900 words, numbered, most important first.

Owner decision (not open to debate): the 6,150 records with an open full text (5,629 JATS XML, 523 PDF) are screened at full text now, skipping title and abstract screening, by one language model reviewer (the coordinating model, recorded as opus-5.5), who checks every record one by one. The owner will supply paywalled full texts later. This deviates from the protocol (protocol/protocol.qmd, sections Eligibility criteria, Selection process and Use of automation and large language models), and will be logged as an amendment before the first decision.

Planned design:

1. For each record the reviewer reads a compact extract: article type, title, journal, year, abstract, the first part of the methods section (design, participants, interventions, comparators), and registry numbers; the full text is opened when the extract does not settle the decision.
2. One decision per record: include_nma (randomized trial eligible for the network meta-analysis), include_harms (comparative nonrandomized study of an eligible design reporting an outcome in protocol/harms_nrs.csv), include_review (systematic review or meta-analysis of GLP-1 receptor agonist containing drugs), inventory_only (in scope for the inventory but not for a synthesis), or exclude (out of scope), with a reason code from a fixed list and, for include_nma, candidate population codes from protocol/populations.csv.
3. Draft fixed reason list: E1 not humans; E2 no GLP-1 receptor agonist containing drug given or compared (including native GLP-1 infusion, antagonists); E3 not a primary study or systematic review (narrative review, editorial, guideline, modeling or economic study); E4 erratum, retraction notice or duplicate; N1 single arm or uncontrolled; N2 pharmacokinetic or mechanistic study in healthy volunteers; N3 randomized contrast does not change the GLP-1 receptor agonist containing regimen; N4 quasi randomized; N5 nonrandomized design not eligible for the harms synthesis (case report, case series, cross sectional, disproportionality, uncontrolled cohort); N6 comparative nonrandomized study without an outcome in harms_nrs.csv; N7 protocol, registration or design paper without results; N8 secondary analysis of a trial (linked to its study, not a separate trial).
4. Decisions go to data/screening/fulltext_decisions.csv (record, stage, reviewer, time with zone, decision, reason, populations, a short rationale), append only, published; no full text content is published.

Questions: A. Is the decision set and reason list complete and mutually exclusive for this protocol's eligibility rules? Give the corrected list. B. Which decision should secondary analyses, extensions, pooled analyses of several trials, conference abstracts and trial protocols receive, so that they are linked to their study and not counted as separate trials? C. What safeguards make a single language model reviewer's decisions auditable and as reliable as possible (for example a random sample for later human audit, re-screening a sample blind to the first decision, rules for uncertain records)? D. What must the amendment text say?

Full text screening by one language model reviewer (reviewer fable-5.1, 2026-10-13)

1 (D, A). Register before the first decision. The repository has no tag or release; the protocol says registration precedes screening and no stage begins until the human reviewers are named. Tag protocol-registered on version 0.22.0 containing A021. If a decision precedes registration, the amendment must say so, rewrite the status callout ("written before any screening") and the registration sentence, as a pre-registration revision. The section field must also name Administrative information, Study records, the "Human gates" paragraph and Timeline.

2 (A). The decision set has no path to a human and no study identity. Add deferred (D1 unreadable or wrong file; D2 awaits an unretrieved report; D3 judgment uncertain) and linked for records that are not the primary report of a study (item 4). Add columns study_id, role, confidence, basis (extract, full text, abstract only), flags, supersedes. Without study_id, three reports of one trial count as three trials.

3 (A). Corrected list. Codes are tested in order; the first that applies is recorded, so they are exclusive by construction. E means exclude, N inventory_only. E1 not humans. E2 no therapeutic GLP-1 receptor agonist containing drug given as an intervention or studied as an exposure (native GLP-1, antagonists, utilization, prescribing, cost, qualitative studies). E3 not a primary study or systematic review (narrative review, editorial, guideline, modeling, economic study). A systematic review needs a named database search and eligibility criteria; multi class reviews qualify when a synthesized contrast involves a GLP-1 containing regimen; umbrella reviews qualify, flagged. E4 erratum, retraction notice or commentary on a record outside scope; on an inventoried study it is linked with that role. N1 interventional study without a concurrent comparison group (single arm, before after). N2 pharmacokinetic or mechanistic study reporting no outcome in outcomes.csv within a window in windows.csv; when one is reported, include_nma with flag mechanistic. N3 randomized contrast does not change the GLP-1 receptor agonist containing regimen. N4 quasi randomized. N5 observational design listed for no outcome in harms_nrs.csv (case report, case series, cross sectional, disproportionality, cohort without comparison group), listed as a signal; self controlled case series is eligible for O34, not N5. N6 comparative nonrandomized study reporting no outcome in harms_nrs.csv with a design listed for that outcome; include_harms records harms_outcome_ids. N7 retracted primary report (excluded from syntheses, listed with the notice). Drop the draft N7 and N8 (roles, not reasons; item 4). Duplicates are a logged inventory merge, not a decision.

4 (B). Eligibility belongs to the study; each record gets one of the protocol's ten study_record roles plus one new role, pooled analysis, linked to every constituent study. Same registry number and same randomization: same study, role secondary report (secondary analyses, longer follow-up, baseline subgroups that place the trial in another population, preprints, open label extensions). New randomization: new study. Pooled analysis of sponsor trials without a systematic search: linked to each trial; with one: include_review. Conference abstract: linked when the study has another report; otherwise the primary report, include_* flagged abstract_only, rescreened when a full report arrives. Protocol, analysis plan or design paper: linked, role protocol; a study without results stays "awaiting results". The first screened report is the provisional primary; later re-roling is an appended row. PRISMA counts distinct study_id per decision, not rows. This stage fills the 3,030 unclassified links in study_records.csv; uncertain links go to a human.

5 (C). Validity needs humans and known positives; a blind re-screen by the same model measures stability only. (a) Every include_* confirmed by a named human before extraction. (b) A random sample of exclude and inventory_only, stratified by design_class and reason, 300 records, kappa against the protocol's 0.8 threshold (zero errors in 300 bound the false exclusion rate below 1 percent); any false exclusion of a trial rescreens every record with that reason. (c) Every trial in the search benchmark set must be include_nma. (d) A blind duplicate screen of the sample and of all low confidence records by the second model in use (GPT-6 Astra). (e) deferred and low confidence records go to a human, never back to the model. (f) Full text opened for every include_* and whenever the decision contradicts design_class or a PubMed publication type. (g) Fresh context per record in fixed record_id order; per row the instructions hash, model identifier, date, extract hash and fulltext_opened; output validated against the fixed lists; rationale paraphrased with a section locator. (h) Inclusive default when uncertain, alternative recorded. (i) Agreement statistics in every release note; 20 first batch records rescreened last as a drift check.

6 (D). A021, 2026-10-13, version 0.22.0, before results, rationale "owner decision". State: the 6,150 records with an open full text go directly to full text screening without title and abstract screening, calibration or duplicate screening; one language model reviewer, opus-5.5 with exact version string and instructions hash, reading the brief's extract and the full text when needed; item 3's lists and the schema of data/screening/fulltext_decisions.csv (append only, in datapackage.json, no full text content); eligibility criteria unchanged; every decision provisional until item 5's verification, with sample size, threshold and failure rule, or, if the owner declines verification, that the human gate sentences are deleted and reported as a limitation; the process for the remaining 23,739 records and the paywalled texts; a full text only PRISMA stage for this subset; the human reviewers named before verification; this consultation cited.

Full text screening by a single language model reviewer (consultant gpt-6-astra, 2026-10-13)

1. A. Five decisions need clearer meanings. include_nma and include_harms should mean eligible for that review pathway, subject to later assessment for quantitative synthesis. Add awaiting_classification; handle duplicate_record administratively. For reports covering multiple designs, keep one primary decision using precedence NMA > harms > review > inventory, while preserving every applicable pathway on study/phase links. Otherwise, mutually exclusive record labels lose eligible evidence. Protocol already requires many-to-many record–study links.

2. A. Corrected fixed reasons. Use one primary reason, selecting first applicable code within its disposition; retain other relevant facts as flags. Apply scope and synthesis disqualifiers before inclusion codes.

   Disposition	Codes
   Inclusion	A1 eligible randomized study/report; A2 eligible nonrandomized harms study/report; A3 eligible systematic review/meta-analysis.
   Exclude	E1 no eligible human investigation, actual, planned or reviewed; E2 no therapeutic GLP-1 receptor agonist containing regimen, actual, planned or reviewed, including native GLP-1 infusion/antagonists; E3 neither eligible primary study, systematic review nor associated supporting document.
   Inventory only	I1 retracted study/results; I2 protocol, registration, SAP or design paper without results; I3 linked ancillary document without independently eligible results, including correction/retraction notice; I4 pharmacokinetic study in healthy volunteers, or mechanistic study in any population; I5 single-arm interventional study; I6 quasi-randomized study not otherwise eligible for harms; I7 randomized contrast leaves GLP-1-containing regimen unchanged; I8 nonrandomized study reporting no specified harm; I9 nonrandomized design ineligible for relevant harm, including case reports, case series, cross-sectional, uncontrolled cohort and disproportionality studies; I10 other harm-specific population/exposure/comparator incompatibility.
   Administrative/pending	D1 duplicate bibliographic record of same report, with canonical record ID; U1 unresolved eligibility; U2 unavailable, incomplete or unreadable source.

   Replace E4 with D1/I1/I3. Remove N8: secondary-report status is a link role, not grounds for exclusion. Define “mechanistic study” operationally before screening; phase, duration or mechanistic outcomes alone cannot settle that label. Eligibility rules.

3. B. Classify evidence, then link reports. Secondary analyses reporting eligible randomized comparisons receive include_nma, linked as secondary reports; baseline/descriptive companion reports receive inventory_only. Extensions retaining randomized comparisons can receive include_nma; uncontrolled extensions receive inventory_only; comparative observational extensions receive include_harms only when qualifying. Record phase and any new randomization explicitly.

   Pooled trial analyses link to every constituent study. Use include_nma when reporting eligible randomized evidence; never create an extra “pooled trial” or combine pooled estimates with overlapping constituent estimates. Pooling alone does not make a systematic review.

   Conference abstracts receive whichever substantive decision their evidence supports, with abstract-only flag. Trial protocols receive inventory_only/I2, linked to their study; missing results never mean no eligible trial exists. Corrections supplying usable corrected results remain attached to the eligible evidence. [Cochrane likewise distinguishes studies from reports.](

https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-05)

4. A. Preserve protocol-specific boundaries. Harms qualification must use the relevant CSV row, not a generic design whitelist: procedure cohorts, pregnancy cohorts and self-controlled designs have different permissions. Assess confounder adjustment, latency, estimand and usable estimates at extraction; distinguish descriptive eligibility from pooling restrictions.

   NMA population codes remain candidates until baseline 80%/subgroup rules are verified. Preserve P01 strata, P02W, heart-failure and kidney splits; route new conditions through P99 amendment. Missing usable outcomes, crossover statistics or network connectivity does not itself justify scope exclusion.

5. C. Make evidence inspection reproducible. Compact extract can establish clear positive facts; absence from an extract cannot establish absence from the paper. Before I8 or another absence-based decision, inspect relevant full-text sections, tables and supplements. Verify extraction completeness, OCR, translation and article identity; retrieve full methods when truncated. Unresolved records remain pending, with missing fact and next action recorded.

   Freeze protocol, rubric and prompt versions. Log actual model/provider identifier alongside opus-5.5, settings, source/extract hashes, inspected sections and precise evidence locators. Keep excerpts private; publish paraphrased rationales. Append corrections using decision IDs and supersedes; derive one current decision per record.

   Prespecify, for example, a seeded 300-record random sample for later blinded human audit, supplemented by targeted exclusion/inventory and rare-design checks. Blindly re-screen a separate random sample using fresh context. Report decision-specific errors, missed eligibility and uncertainty intervals; self-agreement measures repeatability, not accuracy. Any missed eligible study triggers review of its error category and affected decisions. Never invent human validation before it occurs.

6. D. Amendment must state date/version, owner rationale, implementation before first decision, and actual prior exposure to results; direct full-text-stage screening by one coordinating model; skipped title/abstract stage; extract-first inspection and escalation rules; rubric, linkage, pending/retrieval handling, audit plan and correction process. Explicitly replace duplicate-human screening, third-reviewer adjudication, human calibration/kappa prerequisite and advisory-only LLM restrictions, including conflicting administrative text. Preserve other stages unless separately amended.

   Record 6,150 unique records: 5,629 XML + 523 PDF − two overlaps, verified from manifest. Paywalled records remain awaiting retrieval. Disclose incomplete coverage and single-model/extract-related limitations; report records, reports and studies separately. PRISMA items 8, 16a and 24c support these disclosures.

   No files changed.

Screening design: decisions taken from the consultation

Both consultants accepted the owner's decision and proposed safeguards. The coordinating reviewer adopted the following (amendment A021, data/screening/README.md, protocol/exclusion_reasons.csv):

• Registration before the first decision (fable-5.1): the protocol is registered as a GitHub release before any screening decision; the four-source harvest becomes the baseline search after registration.
• Decisions (both): include_nma, include_harms, include_review, inventory_only, exclude, and deferred for unreadable files and unresolved eligibility. Secondary reports, extensions, pooled analyses and abstracts take the decision their evidence supports, with a role that links them to their study (both), so they are never counted as separate studies.
• Reasons (fable-5.1's ordered list, with gpt-6-astra's protocol, ancillary and pending categories): E1 to E4, N1 to N9, D1 and D2, tested in order. Mechanistic studies are inventory only unless they report a review outcome, in which case they are included with a flag (fable-5.1), so the label never removes eligible evidence (gpt-6-astra).
• Harms eligibility follows the harms_nrs.csv row for each outcome (both); new user design and confounder adjustment are judged at extraction.
• Evidence rules (both): an absence based decision requires the full text; so does a decision that contradicts the automatic class.
• Oversight: every deferred and low confidence decision is re-examined; a stratified random sample of 300 is screened blind by the coordinator and by a second model (fable-5.1's item 5d); a seeded sample of 300 is kept for human audit (both); every trial of the search benchmark set must be included (fable-5.1).
• Not adopted: human confirmation of every inclusion before extraction (fable-5.1 5a) and a human audit as a precondition (both), because no human reviewer is available; the owner chose language model screening, and the absence of human verification is reported as a limitation, with the seeded sample ready for a human audit.
• Not adopted: fresh context per record (fable-5.1 5g); each reviewer screens its shard sequentially and records a decision after every batch of eight, and the blind re-screen in a fresh context measures the effect.