LivingGLP1
2026-10-14

Review of the extraction manual

Brief

Consultation: data extraction manual for LivingGLP1

You are a methodological reviewer for the LivingGLP1 living systematic review and network meta-analysis of GLP-1 receptor agonist containing drugs (repository root; read only).

Full text screening is complete (996 records include_nma, 209 include_harms, 1,177 include_review). Extraction starts tonight with many language model extractor agents (Opus 5.5), each study extracted twice independently (sets A and B), then compared by the coordinator. Records are first linked to studies.

Please review the draft manual data/extraction/README.md against the registered protocol protocol/protocol.qmd (read the sections: Study records > Data collection; Data items; Outcomes and prioritization, including time windows, composites, event outcomes and harms, competing risks and estimands; Risk of bias in individual studies; Data synthesis > Networks and nodes) and the tables protocol/outcomes.csv, protocol/windows.csv, protocol/effect_modifiers.csv, protocol/nodes.csv, protocol/harms_nrs.csv, protocol/populations.csv.

Questions:

  1. Which data items the protocol requires are missing from the manual or cannot be captured by its fields (give the protocol passage and a concrete field to add)? Prioritize items whose absence would make a planned analysis impossible or would force re-extraction later.
  2. Which rules in the manual contradict the protocol?
  3. Is the key=value line format with the listed statistics sufficient for the planned likelihoods (normal on mean differences and log ratios; complementary log log binomial with offset log t; known status at t; cause specific versus subdistribution hazards; recorded harms analysis Q)? What exact fields are missing?
  4. Practical risks with language model extractors (ambiguity that will produce discrepant double extractions, workload per study) and how to reduce them.
  5. Anything to defer safely to a later pass (for example curve digitization, RoB 2) without re-reading articles twice.

Answer as a numbered list of concrete changes, most important first, each with the exact text or field to add or change. End with a verdict line: READY, READY WITH CHANGES, or NOT READY. Be concise. Do not write any files.

Brief round2

Consultation round 2: data extraction manual for LivingGLP1

You reviewed the draft extraction manual in round 1 (your review: protocol/peer-review/extraction-2026-10-14/ROUND1_FILE). Read the response protocol/peer-review/extraction-2026-10-14/response.md, then the revised manual data/extraction/README.md, the recorder in R/extraction.R (functions extraction_enums, extraction_fields, extraction_identity, validate_extraction_line, record_extraction) and pipeline/08_extract.R, and amendment A023: protocol/amendments.csv (last row) and protocol/protocol.qmd (Administrative information, Data collection, Use of automation and large language models, Timeline).

Questions:

  1. Are your round 1 items resolved? For each item not resolved, say what exact change remains.
  2. Does amendment A023 resolve the protocol's human extraction requirement transparently and consistently across the protocol text?
  3. Any defect in the recorder that would accept wrong data or reject correct data for a randomized trial?
  4. Anything that must change before a pilot of two extractors on three trials and then a full launch?

Answer as a numbered list of concrete remaining changes, most important first. End with a verdict line: READY, READY WITH CHANGES, or NOT READY. Be concise. Do not write any files.

Fable 5.1 round1

Review of data/extraction/README.md against the protocol (fable-5.1, round 1)

Key finding before the list. The manual documents seven Step 2 commands (next, packet, text, registry, record, done, status) and promises "the recorder rejects a line that breaks a rule below". None of this exists: pipeline/08_extract.R and R/extraction.R implement only the linking commands. Extraction cannot start tonight under this manual regardless of its text. Put every controlled vocabulary below in one machine readable file (for example data/extraction/fields.csv) so the recorder enforces it and the README is derived from it; otherwise two language model extractors will each invent values.

Ordering rule: an item comes first when its absence forces re-reading an already extracted article or makes a protocol likelihood unfittable.

  1. Add unit to result lines (Q1; forces re-extraction). Protocol "Missing statistics and conversions": HbA1c mmol/mol to percent, WOMAC rescaled to 0 to 100; outcomes.csv O06 "converted using baseline weight", O24B/O24C "direction aligned per instrument". The result table has no unit field. Add: unit required on change, percent_change, final, md, slope; instrument required for O24A to O24C, O41, O80 (SF-36, SF-12, IWQOL-Lite-CT, KCCQ, WOMAC, other name); scale_range (0-20, 0-50, 0-100) for O80.

  2. Known status as counts, and separate the three meanings of t_weeks (Q1, Q2, Q3; contradiction). Protocol Data items: "number with known status at t by arm; observation period". Event outcomes: counts enter the cloglog likelihood "only when status is known for every randomized participant at t, the longest follow-up at which that holds" (95 percent in sensitivity); recorded harms use "the nominal observation period in years"; Rates: "median follow-up [is] not [a] substitute". The manual's known_status (complete, partial, unknown) cannot feed any of these, and "the median follow up as t_weeks" is exactly what the protocol forbids. Replace on events lines with: status_weeks (the t), n_unknown (randomized participants lost or withdrawn from follow up before t without a target event or documented competing death; 0 when all are accounted for), observation_weeks (nominal adverse event observation period of the arm: treatment period plus any protocol follow up window). Allow several events lines per arm at different status_weeks. Move follow up to the arm table: median_follow_up_weeks, follow_up_type (median, mean, max, planned); the analyzed horizon is "the median follow-up of the network's reference arms".

  3. Competing event inputs (Q1, Q3). Protocol Data items: "competing deaths without a preceding target event and person years at risk by arm ... any reported hazard ratio for the competing event"; Rates: "treatment exposure and median follow-up are not substitutes". Add on events and rate lines: competing_deaths (optional count), person_years_type (at_risk_first_event, exposure, total_follow_up, unclear; required whenever person_years is given), optional week_from for interval specific counts (piecewise route). Add contrast statistic competing_hr (estimate, lower, upper, estimator) carrying the target outcome_id.

  4. Estimand per intercurrent event with a sponsor term mapping (Q1, Q4). Protocol Data items: "estimands and the strategy for each intercurrent event (treatment discontinuation, rescue medication, death)"; Estimands: "classified by its strategy for every intercurrent event". One value cannot encode "on treatment without rescue". Add to study and result lines ie_discontinuation, ie_rescue, ie_death (same vocabulary as estimand); estimand=treatment_policy only when discontinuation and rescue are both treatment policy. Mapping table in the manual: "treatment regimen", "treatment policy" to treatment_policy; "trial product", "efficacy estimand", "on treatment without rescue", "hypothetical" to hypothetical (EM12 already says "hypothetical or trial product"); "on treatment", "while on treatment" to while_on_treatment; completers or per protocol analyses to population=completers|per_protocol with ie_discontinuation=while_on_treatment.

  5. Nonrandomized studies: estimand=not_stated contradicts harms_nrs.csv (Q1, Q2, Q5). That table requires "(a) initial treatment (intention to treat analogue) and (b) on treatment; never combined", non users "in a separate secondary stratum", latency, a checkable minimum confounder set and effect measures pooled separately. Replace with: design controlled (new_user_cohort, active_comparator_new_user, target_trial_emulation, nested_case_control, case_control, self_controlled, procedure_cohort, pregnancy_cohort); on result lines nrs_estimand (initial_treatment, on_treatment, not_stated), comparator_type (active, non_user), latency_weeks, risk_window (short); confounders as a ; list from a vocabulary mirroring harms_nrs.csv; several comparators as arms C1, C2 with comparator_class (SGLT2I, DPP4I, other_glucose_lowering, non_user). Recommendation: do not start the 209 include_harms records tonight; run them as a second pass under these fields.

  6. Cluster and crossover fields (Q1). Protocol sec-designs needs the design effect 1 + (m − 1)ρ and the paired contrast or first period rule. Study line: clusters_n, mean_cluster_size, icc, cluster_adjusted (yes, no); periods_n, period_weeks, washout_weeks, carryover_addressed (washout, analysis, both, no). Result line: period (all_paired, first, all_unpaired).

  7. subgroup on result lines; documentation of the 80 percent rule (Q1). Protocol Participants: a trial enters a population "when results are reported for a subgroup defined at baseline that meets it"; Meta-regression: "Subgroup results reported within trials are extracted". Add subgroup=<variable>:<level> (for example egfr:<60, established_cvd:yes), absent for the whole trial; extract now only subgroups that place the trial in another population, for that population's primary outcomes; defer modifier subgroups. Study line: population_percent, stratum_percent (share meeting the definition or stratum), eligibility_key (age, BMI, HbA1c, eGFR, LVEF ranges in a few words), run_in_enrichment for P02W (populations.csv: "recorded separately").

  8. Define the replacement key (Q4). "A later line with the same key fields replaces an earlier one" names no keys; two O22D composites or two subgroups overwrite each other silently. Result key: study_id, outcome_id, definition_stratum, subgroup, week, ie_discontinuation, ie_rescue, ie_death, population, arm_id, vs_arm_id, statistic, source. Baseline key: study_id, arm_id, variable, statistic, subgroup. Arm key: study_id, arm_id.

  9. Coverage lines (Q2, Q4, Q5). Add one table=coverage|outcome_id=O21|status=reported|not_reported|figure_only|registry_only line per protocol outcome applicable to the study's populations. Without it the coordinator cannot tell an extractor's omission from true absence when sets A and B differ, except by re-reading. figure_only also resolves the contradiction between the manual's "not figures only" and the protocol's "Values read from figures are digitized twice independently and flagged".

  10. Cut the scope of result lines (Q4). "Every time point ... every estimand ... every source" is combinatorial (a CVOT: 20 outcomes, 3 sources, 2 estimands). Replace with: time points inside any windows.csv window for the outcome and population, plus end of randomized treatment and longest follow up; every estimand at those; journal and supplement always; registry, abstract and regulatory lines only when the value is absent from the journal or differs from it (the protocol's "all values are kept" governs disagreement, not duplication).

  11. Decide responders versus events by model_scale; derive hazard_type (Q2, Q3, Q4). Rule: responders for every outcome with model_scale=log_odds (not only O03 and O11A to O11E; O60, O61, O90A to O90C, O110B, O111, O111B, O112, O113 too); events for log_hazard_cloglog and time_to_event; rate and rate_ratio only for O04B. Drop hazard_type from extraction: the protocol says "the hazard type follows the estimator" and not_applicable is O20 only, so extractors offered a free choice will disagree; derive it at analysis from estimator and outcome. Add on hr lines ph_test (not_reported, not_rejected, rejected) and ph_p (protocol: "reported proportionality assessments are extracted").

  12. Controlled component vocabulary and flags (Q1, Q4). definition_stratum as free text will never match between extractors. Tokens: cv_death; mi; stroke; ua_hosp; hf_hosp; hf_urgent; revasc; all_cause_death; kidney_failure; dialysis; transplant; egfr_decline_50; egfr_decline_40; creatinine_doubling; egfr_lt15; kidney_death; macroalbuminuria. Add mapped=pre2017 on O04 lines mapped under the outcome's mapping rule (protocol: "mapped trials are flagged and excluded in a sensitivity analysis") and adjudicated (yes, no) on O21, O21B, O29 lines (O29 "investigator reported in a sensitivity analysis").

  13. Second tier statistics (Q3, Q4). On change and final: allow median, q1, q3, min, max in place of estimate plus dispersion, and optional p or t (protocol: SDs "derived from ... t statistics or P values; medians ... converted and flagged"). cuminc: optional se. Make ci_level an explicit optional field on every line with lower and upper. Numeric rules: values as printed, no rounding, sign kept, percent as a number, p_lt=0.001 for inequalities.

  14. Denominator rule (Q4). n on result lines is the denominator printed in the source; population says which; the analysis substitutes n_randomized from the arm table (protocol: "Denominators are the number randomized; the safety population is used only when the randomized number is not reported, and those trials are flagged"). Never let the extractor substitute.

  15. RoB 2: capture the design facts now, defer the judgments (Q5). Study line: sequence_generation (computer, other_adequate, not_reported, inadequate), allocation_concealment (central, pharmacy_or_sealed, not_reported, inadequate), outcome_adjudication (yes, no, not_reported), sap_available (yes, no). These sit in the methods section the extractor reads anyway. Note the conflict for the later pass: the manual's "Never copy sentences" versus the protocol's "signaling question answers and supporting quotations are stored as data"; resolve by allowing quotations in a non public RoB table.

  16. Figure index lines so digitization can be deferred without re-reading (Q5). table=figure|outcome_id=|arm_ids=|source=R0001234:Figure 2|curve=km_survival|km_cuminc|aalen_johansen|mean_over_time|numbers_at_risk=yes|no|competing_curve=yes|no. Curves, numbers at risk, proportionality and constant hazard judgments are then a later reviewer pass (protocol: "made by two reviewers before relative effects are modeled").

  17. State what is not an extractor item (Q5). Registration versus start date and registered but unreported outcomes (ROB-MEN) come from the registry packet and inventory; the coordinator derives them. Minor additions to baseline: add waist to the always list (protocol names it) and insulin_use, metformin_use, sglt2i_use (percent) to document S0 to S3.

  18. Clarifications, not contradictions. Double extraction of study and arm characteristics is stricter than the protocol's "extracted by one reviewer and verified by a second"; say so in one clause. Add one worked example block (study, arms, baseline, results, coverage, done) per table; it does more for agreement between two Opus extractors than any rule.

Verdict: NOT READY. Items 1, 2 and 4 would force re-extraction of every study done before they are added, item 5 means NRS extraction under the current section would be redone, and the recorder the manual relies on does not exist. With items 1 to 10 in the manual and a recorder that enforces the vocabularies, the randomized trial part is READY WITH CHANGES; NRS waits for item 5.

Fable 5.1 round2

Review round 2 (fable-5.1)

Review done against the files as they stood at the end of this session: protocol/protocol.qmd (00:11), data/extraction/README.md (00:12), R/extraction.R (00:13), pipeline/08_extract.R (00:13), protocol/amendments.csv (00:04). The files changed while I was reading; everything below was re-tested on the latest versions by loading the package in R and calling validate_extraction_line(), extraction_compare() and extraction_done_check() directly (nothing was written to the project).

Q1, round 1 items. All 18 are adopted in the manual text; the worked example validates. Items 2, 3, 4, 6, 8, 9, 10, 12, 13, 14, 15, 16, 18 are resolved. Residual gaps are enforcement, not text: item 1 (instrument not required on O24A to O24C, O41, O80, O100A, O100B, O101 although it is in the identity), item 7 (population_percent format mismatch, point 3 below; stratum_percent was not adopted, and I accept population_percent carrying P01:<percent> per stratum only if the manual says so), item 11 (statistic by model_scale stated but unenforced, point 2 below). Subgroup scope was widened beyond my item 7 to effect modifier subgroups for primary outcomes; the owner's call, but it raises load and is unenforced.

Q2, amendment A023. Transparent in the status note, Administrative information, Data collection, Human gates and Timeline. Four inconsistencies remain (points 9 to 12).

Q3, recorder. Fixed since round 1 and confirmed by test: multi value enums, vs_arm_id format and self comparison, final with CI, template node IDs, estimator and competing_handling on hazard ratios, figure identity, ie_death in identity, non negative and integer checks. Open defects are points 1 to 8.

Changes, most important first

Before the pilot continues:

  1. The pilot cannot be evaluated as set up. extraction_compare() takes the role from substr(set, 1, 1), so sets PA and PB both become P and are filtered out; I ran it on the 154 rows already in rows_PA.csv and got zero rows. Derive the role from the A or B in the set name (or name the pilot sets A00, B00). Add compare <study_id> and resolve <study_id> commands to 08_extract.R (none exists) with a discrepancy log holding the coordinator's value and basis per identity and field, and a "Published tables" section in the README (final rows, discrepancy log, agreement) mirroring the screening README. Without it the protocol's "resolves every discrepancy against the source and logs it; agreement is reported" has no artifact.
  2. Statistic and time field by outcome scale are unenforced. Accepted today: responders on O25, change and md on O03, or on O21, events on O25 with week instead of status_weeks; hr lines accept either time field and the manual gives no rule; the pilot rows carry week on or lines. Both fields are in the identity, so A and B will produce missing lines rather than value differences. Rule by model_scale from outcomes.csv: MD, SMD and log_odds take week; log_hazard, log_hazard_cloglog and log_rate take status_weeks (with week_from for intervals); check the statistic against the scale in the same place (extraction_reference() already reads the file).
  3. population_percent rejects what the manual asks for. README says P08A:45, ; separated; the recorder lists it in extraction_numeric and rejects it ("must be a finite number"). Tested.
  4. The worked example contradicts the pilot's own output for the same trial and value. Example: population=randomized, no ie_discontinuation; the PA extractor wrote population=modified_itt|ie_discontinuation=treatment_policy for the same SURMOUNT-3 line. population is in the identity, so this divergence alone splits every result line. Correct the example to what the rule yields from the printed label and add one sentence: randomized only when the printed denominator is the number randomized.

Before full launch:

  1. parse_kv is silent on malformed input. A token without = (|adjusted|) is dropped and the first of duplicate keys wins (estimate=-1|estimate=-2 records -1); both tested. Reject both.
  2. definition_stratum token order changes the identity (cv_death;mi;stroke and mi;stroke;cv_death produce different keys; tested). Sort tokens when building the identity.
  3. Unchecked formats, all tested as accepted: arm_ids=tirz;pbo on figure lines; registration_date=March 2021; subgroup=eGFR < 60 and analysis_label=tipping point (both in the identity, so enforce ^[a-z0-9_]+ patterns); source=registry:outcomes with source_type=journal; instrument absent on PRO outcomes; arm n_randomized optional although the denominator rule depends on it.
  4. extraction_compare() will be flooded by free text. It skips only note and source; the comment says definitions are skipped but definition is compared, as are eligibility_key, label, sponsor, countries, titration, missing_data, time_reported. Compare free text fields for presence only and report agreement on controlled fields.
  5. Study records (lines 396 to 397) still says uncertain links "are adjudicated by two reviewers." A023 uses one language model reviewer, and the amendment row's section list omits Study records. Add the section to the row and a pointer sentence in the paragraph.
  6. Use of automation still says models "may also act as an advisory, non binding additional reader during extraction" two lines before the sentence that makes them the extractors. Delete the clause.
  7. The amendment row is thinner than the protocol text it logs. The Data collection paragraph adds "or all when fewer", the 5 percent error rate re-extraction trigger, and the "until a human reviewer is named" release statement; the row says only "a seeded 10 percent sample (at least 50 trials)". Sync the row, and state when the sample is drawn and where the seed is recorded.
  8. The coordinating reviewer's nature is unstated. Human gates says the guarantor's coordinating reviewer adjudicates every disagreement, the only gate between the two extractions and the final values; nowhere does the protocol say whether that is the human guarantor or a model session, and if a model, which. Say so in the A023 paragraph and the row. Also add a bracketed pointer to A023 in the first Data collection paragraph, which still reads as two human reviewers plus a verifier.
  9. Pilot design. The three pilot trials (SURMOUNT-3, SUSTAIN 7, LixiLan JP-O1) are all continuous and responder trials; none exercises status_weeks, n_unknown, competing_handling, cuminc, KM figure lines or the hypoglycemia mapping, which were the center of round 1. Swap in or add a CVOT or kidney trial. The PA rows began at 00:10 under a README and recorder that changed at 00:12 and 00:13; discard or re-record them under the final manual, and log any manual change after launch in amendments.csv, since A023 names the README as the governing document. The README documents neither sets, set naming, the pooled only rule, nor the comparison step; the coordinator's half of the workflow is invisible to a reader of the registered manual.

Verdict: READY WITH CHANGES. Points 1 to 4 before the pilot continues (3 rejects correct data; 1 means the pilot yields no agreement figure); 5 to 13 before full launch.

GPT-6 Astra round1

  1. Fix event times, completeness, and denominators before extraction. Protocol: known status and recorded harms requires different inputs for hazard analysis and Q. Replace manual’s “median follow up as t_weeks” with: “Record status_time_weeks, nominal_observation_weeks, and median_followup_weeks separately.” Add arm/outcome/time fields n_randomized, n_known_status, n_unknown_status, n_incomplete_followup, and corresponding event counts. Categories complete|partial|unknown cannot support 95%, 20%, or 10-percentage-point thresholds. State: “Known status includes target events, competing deaths before target events, and confirmed event-free observation through t; numbers at risk are not known-status counts.” Replace safety-denominator exception with protocol wording: “Use safety denominator only when randomized number is not reported.” Preserve original reported denominator and population.

  2. Capture estimands per analysis; remove forced NRS not_stated. Protocol: Estimands requires “strategy for every intercurrent event.” Add estimand_id, discontinuation_strategy, rescue_strategy, death_strategy, analysis_set_definition, and result-specific missing_data_method. Add: “Do not infer treatment policy from ITT/FAS terminology alone.” Manual’s mandatory NRS estimand=not_stated contradicts separate NRS syntheses. Replace with initial_treatment|on_treatment|risk_per_procedure|risk_per_pregnancy|other|not_reported.

  3. Add competing-event inputs needed for absolute effects. Protocol: Data items requires competing deaths, valid person-time, and competing-event HRs. Add event_role=target_first|competing_first, risk_set_id, interval_start, interval_end, person_years_at_risk, person_time_definition, and linked competing_event_result_id. State: “Person-time ends at first target event, competing event, censoring, or cutoff; treatment exposure and general follow-up are not substitutes. Deaths after target events are not competing-first deaths.” Preserve reported intervals. Without these inputs, planned cause-specific absolute-risk calculations become unavailable.

  4. Make hazard classification explicit. Protocol: Competing risks requires estimator-specific networks. Add: “Complete-status counts enter subdistribution networks for competing-event outcomes; not_applicable applies only to O20. Kaplan–Meier estimates censoring competing deaths yield cause-specific contrasts and cannot supply competing-risk absolute probabilities.” Require hazard_type, estimator, and competing_event_handling for every hazard-scale input, including binary harms. Add proportionality_test, proportionality_p, and proportionality_source; preserve evidence for later constant-hazard judgments.

  5. Keep key=value format, expand likelihood schema. Syntax is sufficient; listed fields are not. Protocol: likelihoods requires normal inputs and multi-arm covariance. Add estimate_scale, se_scale, ci_level, unit, adjusted, adjustment_model_id, and contrast_id; add table=covariance|contrast_id_1=…|contrast_id_2=…|covariance=…|scale=…|source=…. Permit reported log-ratio plus SE without mandatory CI, especially O04B. Extend cuminc with quantity=cumulative_incidence|survival|event_free_survival, se, greenwood_variance, and uncertainty_scale. Retain uncertain estimates descriptively; require qualifying uncertainty before synthesis. Mark assumed CI levels explicitly.

  6. Preserve recoverable continuous summaries and distinguish modeled proportions. Protocol: conversions permits median conversions, dispersion derivation, and imputation. Add result-level median, q1, q3, min, max, t_statistic, df, dispersion_status, instrument, scale_min, scale_max, and higher_is_better. Allow incomplete estimates to remain pending rather than rejecting them. Preserve change/final distinction for SMDs. For binary results add proportion_method=observed|imputed|model_estimated and uncertainty; state: “Never round imputed/model-based percentages into binomial event counts.” Generalize responder capture beyond O03/O11 to applicable fixed-time binary outcomes.

  7. Identify samples and prevent result replacement collisions. Protocol: population assignment permits baseline-defined subgroups and multiple populations. Add sample_id, sample_definition, population_id, population_stratum, assignment_basis, and supporting eligible numerator/denominator. Reference samples from baseline and result rows. Separate clinical population from analysis population. Document replacement keys, including sample, outcome definition, analysis, time/interval, estimand, hazard type, arm/contrast, statistic, and source. Add stable result_id; adjusted and unadjusted results must coexist.

  8. Make NRS metadata analysis-specific. Protocol: NRS synthesis separates latency, risk windows, designs, and adjustment sets. Add nrs_analysis_id, cohort_id, target_population, comparator_type, risk_window, latency, censoring_rule, adjusted_covariates, and minimum_confounders_met per analysis. Preserve database, enrollment dates, and cohort definition for overlap detection. Add denominator_unit=participants|procedures|pregnancies|person_time and design-specific case/control or exposure-period identifiers. Do not force self-controlled studies into independent E/C participant arms.

  9. Complete node-defining fields and enums. Protocol: Networks and nodes requires more than numeric maintenance dose. Add permitted_maintenance_doses, dose_adjustment_algorithm, component-specific agent/dose/unit, component_ratio, permitted_pens, switching_threshold, and strategy_id. Match nodes.csv, including IMPLANT, CONT, Q4W, and titrated_to_target; normalize mtd to canonical vocabulary. Replace “dose, number only” with component-level doses for combinations. State: “Template rows never constitute completed node assignments.”

  10. Capture missing study and special-design facts now. Protocol: study-level data requires registration/start dates, sites, eligibility, and actual duration. Add registration_date, start_date, end_date, sites_n, eligibility_criteria, actual treatment/follow-up durations, and COI details. Add run-in duration, dose, response enrichment, and tolerability enrichment. For cluster/crossover analyses, capture cluster counts, mean cluster size, ICC, cluster adjustment, sequence/period, washout, carryover handling, paired SE, and first-period results.

  11. Make definitions, time, and baseline categories reproducible. Protocol: composites requires identical components and thresholds. Add component_codes, threshold_value, threshold_unit, sustained_duration, adjudication_status, and hypoglycemia_mapping_used. Preserve time_value_reported, time_unit_reported, time_origin, and randomized_phase; derive windows centrally with inclusive lower bounds, exclusive upper bounds, and later-time tie breaking. Procedure/pregnancy outcomes have no week window. Baseline rows need category and category denominator for fibrosis, antipsychotics, and ethnicity. Make waist explicitly mandatory when reported, matching protocol Data items.

  12. Operationalize source priority and field provenance. Protocol: Data collection ranks prespecified analysis first, then regulatory review, registry, and journal. Add prespecified, prespecification_source, analysis_id, source_version/date, and field-specific source locations. Replace “each value is taken from” highest-ranked source with: “Retain all distinct reported values; record selected value, selection reason, and alternatives after reconciliation.” Source type alone cannot identify prespecification or distinguish analyses.

  13. Resolve extraction’s human-review requirement. Protocol: automation permits LLMs as an “advisory, non binding additional reader”; screening exception does not extend to extraction. Add: “LLM A/B outputs remain provisional. Human reviewers verify source values and make extraction decisions, including values on which models agree. Human-model agreement is measured and reported.” Record extractor, model/version, human verifier, and adjudication. Replacing this arrangement requires explicit protocol amendment before launch.

  14. Control omissions, linkage errors, and workload. Add table=availability with outcome_id, sample_id, status=reported|not_reported|figure_only|unavailable|pending|not_applicable, source, and followup_action; omitted fields cannot distinguish missing evidence from unfinished work. Require confirmation that automatically linked registry IDs identify reported participants, not cited trials. Freeze arm IDs and document manifests before independent A/B work. Read each source once per set, extract in outcome blocks, and pilot contrasting study designs before mass dispatch. Deduplicate identical values while retaining all provenance and disagreements.

  15. Defer processing, preserve evidence now. Protocol requires double digitization; replace figure exclusion with digitization_status=pending plus figure/page, arm/sample, outcome, estimator, time range, risk-set linkage, numbers-at-risk, and covariance availability. Preserve both first-event curves or target incidence plus event-free survival. SD conversions, window assignment, and formal RoB judgments can follow. Capture RoB methods/flow/SAP evidence now; add table=rob_evidence|result_id=…|domain=…|supporting_quote=…|source=…. Exempt these quotations from manual’s copying prohibition because protocol explicitly requires them. Deferred tasks remain tracked and must finish before affected synthesis.

NOT READY.

GPT-6 Astra round2

  1. Prevent replacement collisions (round 1: 7, 12). Identity keys omit ie_death, estimator, quantity, instrument and distinct analysis identifiers. Cox/Fine–Gray results, survival/incidence estimates, and multiple sensitivity analyses can replace each other. Add stable result/analysis IDs; preserve full source locations, versions and field provenance. Figure identities also need outcome and curve identifiers.

  2. Compare every substantive field (13, 14). Comparison code skips definitions, background treatment, missing-data methods, eligibility and dose-adjustment rules. These can change eligibility, estimands or nodes. Route differences to adjudication; retain both values, selected value, reason and adjudicator.

  3. Reject invalid records. Validation accepts negative/fractional counts, events > n, infinite numbers, negative SEs, malformed comparator IDs and self-comparisons. Add numeric bounds, integer-count checks, interval ordering, arm/source linkage and outcome–statistic compatibility. Reject duplicate keys, malformed tokens and multiple values in scalar enums.

  4. Preserve valid incomplete summaries (1, 5, 6). Required-field checks reject final means with CIs, means awaiting dispersion recovery, competing HRs reported with log-SE alone, and studies lacking randomized totals. Permit explicit pending/missing states. Add estimate/uncertainty scales, degrees of freedom and CI-level provenance; support reported log-ratios without extractor exponentiation.

  5. Restore covariance capture (5). Result schema lacks contrast covariance and Greenwood variance. Add linked covariance records and uncertainty scales. Arm summaries cannot generally recover covariance between adjusted contrasts. Rarity does not justify discarding reported covariance.

  6. Finish event completeness and risk-set capture (1, 3, 4). Separate status time, observation period and median follow-up are improvements. Still add outcome-specific incomplete-follow-up counts for recorded-harms sensitivity analyses; link target events, competing-first deaths and person-time to identical samples/intervals. Require estimator and competing-event handling before deriving hazard type; cox or other alone is insufficient. Manual

  7. Enforce modeled-proportion handling (6). Require proportion_method; prevent imputed/model-estimated percentages or counts from becoming observed binomial counts. Preserve their uncertainty and mark synthesis eligibility separately. Current responder validation does neither.

  8. Complete analysis-specific estimands (2). Add result-level missing-data methods, analysis-set definitions and adjustment-model identifiers. Remove automatic assignment of while_on_treatment from completers/per-protocol labels; classify intercurrent-event strategies from reported methods, including death. Manual

  9. Restore sample/population mapping (7, 11). Add sample IDs linking results and baseline data to population, stratum and eligibility numerator/denominator. One population_percent cannot describe multiple populations. Capture baseline category denominators and reported effect-modifier subgroups, including those within an existing population. Current restriction conflicts with protocol subgroup meta-regression. Manual

  10. Fix node validation (9). Exclude template nodes from completed assignments. Make NEW requirements depend on dosing strategy: flexible/titrated regimens need not have one numeric maintenance dose. Add component-specific units, permitted pens, switching threshold and strategy identifier. Recorder

  11. Complete remaining definition/design fields (10, 11). Preserve sustained-duration thresholds, original time value/unit/origin, actual treatment duration, COI details, arm-level cluster information and crossover period-effect handling. Controlled component tokens and study-level design flags do not capture these distinctions. Schema

  12. Verify automatic links and enforce RCT-only dispatch (8, 14). One registry number appearing in references does not establish participant identity; verify automatic links before freezing packets and arm IDs. NRS deferral is acceptable, but set creation currently includes NRS studies. Filter to kind == "rct".

  13. Finish A023 consistency (13). Amendment transparently authorizes replacing human extraction, but Human gates still declares extraction decisions human. Add explicit A021/A023 exceptions; reconcile “advisory” extraction wording and clarify coordinator identity. Specify audit timing, sample-size cap when fewer than 50 trials exist, and correction procedure; deferred audit is not completed verification.

  14. Define pilot and launch gates (14, 15). Fix recorder blockers before three-trial pilot. Require complete coverage or explicit pending/unavailable status, frozen source manifests, adjudicated discrepancies and tracked digitization/RoB tasks before full dispatch. Preserve figure event-role, sample and curve linkage; formal digitization and RoB judgments may remain later passes. Current done checks only set membership. Add regression checks for findings above. Runtime validation remains unverified: R failed because read-only permissions prevented temporary-file creation.

NOT READY

Response

Response to the round 1 reviews of the extraction manual

Both reviewers returned NOT READY. Each change below is in data/extraction/README.md and enforced by the recorder in R/extraction.R (validate_extraction_line()), unless stated otherwise.

gpt-6-astra

  1. Event times, completeness, denominators. Adopted: status_weeks, n_unknown (counts, not categories), observation_weeks, and median or mean follow up on the arm line, never as the status time; the definition of known status is in the manual. Denominator rule: n is the printed denominator, population names it, and the analysis substitutes the number randomized.
  2. Estimands per intercurrent event. Adopted: ie_discontinuation, ie_rescue, ie_death, a mapping of sponsor terms, and the rule not to infer treatment policy from "intention to treat" or "full analysis set". Nonrandomized fields are deferred to a second pass (see 8).
  3. Competing event inputs. Adopted in simplified form: competing_deaths, person_years with person_years_type, week_from for interval counts, and the competing_hr statistic. Explicit risk set identifiers are not added; intervals are identified by week_from and status_weeks within an arm and outcome.
  4. Hazard classification. Adopted the estimator vocabulary and proportional hazards fields (ph_test, ph_p). The hazard type is derived at analysis from the estimator and outcome, following the protocol, so extractors do not choose it (as fable-5.1 item 11 proposes).
  5. Likelihood schema. Adopted unit, ci_level, adjusted, se_log for ratios, and cuminc with quantity and se. A covariance table is not added: reported covariances of multi arm contrasts are rare, and arm level data give them at analysis.
  6. Continuous summaries and modeled proportions. Adopted median, q1, q3, min, max, p, t, instrument, scale_range, and proportion_method for responders, with the rule never to round a model based percentage into counts. Responders follow model_scale.
  7. Samples and replacement keys. Adopted subgroup (only subgroups that place a trial in another population), population_percent and eligibility_key, and an explicit identity per table, including adjusted and analysis so that adjusted and unadjusted or primary and sensitivity results coexist.
  8. Nonrandomized metadata. Deferred: nonrandomized studies are extracted in a second pass with fields built from harms_nrs.csv; nothing is extracted from them under this manual.
  9. Node fields. Adopted permitted_doses, adjustment_rule, component_ratio, dose_other, the node vocabularies of nodes.csv (IMPLANT, CONT, Q4W, maximum_tolerated, titrated_to_target), and the rule that a template node is never a completed assignment.
  10. Study and design facts. Adopted registration, start and end dates, sites, eligibility key, run in weeks and enrichment, and the cluster and crossover fields.
  11. Definitions and baseline categories. Adopted controlled composite tokens, adjudicated, mapped=pre2017, and a baseline category; waist is always recorded. Windows are assigned centrally at analysis from the recorded weeks.
  12. Source priority. Adopted analysis and prespecified; all distinct values are kept with their sources and the choice is made at analysis.
  13. Human review requirement. Resolved by the guarantor's amendment A023 (protocol 0.23.0), made before any value was extracted: duplicate model extraction with coordinator adjudication of every discrepancy, and a seeded 10 percent sample kept for human verification.
  14. Omissions, linkage, workload. Adopted coverage lines, the linking step with reviewer links where registry numbers are absent or ambiguous, and frozen arm numbering (order of the main baseline table). Extractors record as they read each report.
  15. Deferred processing. Adopted figure index lines for later double digitization and the design facts for RoB 2 (sequence generation, allocation concealment, outcome adjudication, analysis plan). RoB 2 judgments and supporting quotations are a later pass; quotations will be kept in a non public table because full texts are not redistributed.

fable-5.1

Items 1 to 4, 6 to 16 and 18 are adopted as above (units and instruments; known status counts and the three follow up quantities; competing event inputs; estimand fields and term mapping; cluster and crossover fields; subgroups and population share; replacement keys; coverage lines; reduced scope of time points and sources; responders by model scale and derived hazard type; component tokens and flags; second tier statistics; denominator rule; RoB 2 design facts; figure index lines; a worked example). Item 5 is adopted by deferring nonrandomized studies to a second pass. Item 17: the coordinator derives registration timing and registered but unreported outcomes from the registry and inventory; waist and the background drug percentages are in the baseline list. The commands of Step 2 now exist and the recorder enforces the vocabularies in code rather than in a separate file, which keeps one source of truth.

Response to the round 2 reviews

fable-5.1 (READY WITH CHANGES). All thirteen points are adopted before the full launch: pilot sets are compared by the A or B in their names and a compare command lists one sided lines and value, classification, presence and text differences (the coordinator's final lines go to set C; Step 3 of the manual); the statistic and the time field are checked against each outcome's model_scale; population_percent, dates, subgroup and analysis labels, figure arm lists and permitted doses have enforced formats; repeated keys and fields without = are rejected; composite tokens are sorted in the identity; a registry source requires source_type=registry; patient reported outcomes require instrument; arms require n_randomized; free text is compared for presence only; Study records, Use of automation, the first Data collection paragraph, Human gates and the amendment row now agree with A023 and say that the coordinating reviewer is a language model session acting for the guarantor; the pilot adds the SELECT outcome trial and its early rows are not published. The pilot also showed that a result identity holding population, estimator, adjustment and intercurrent event fields split the two extractions; these fields are now compared, not part of the identity.

gpt-6-astra (NOT READY). Adopted: identity collisions (analysis labels, instrument, quantity in the identity), comparison of every substantive field (free text for presence), bounds on counts, standard errors, intervals and comparator arms, incomplete summaries kept with incomplete=yes, final values with confidence intervals, competing hazard ratios with log standard errors, a covariance table, competing_handling on hazard ratios, required proportion_method for responder percentages without counts, result level missing_data, no automatic intercurrent event strategy for completers, population_percent per population, effect modifier subgroups for primary outcomes, node checks by dosing strategy and template exclusion, sustained duration and reported time fields, the randomized trial filter for sets, and the human gates text. Not adopted before launch: sample identifiers beyond subgroups and population shares, arm level cluster fields, component specific units for every combination, and conflict of interest details; these are recorded by the extractors in notes when they arise and can be added by amendment. Automatic links rest on registry numbers in the record's own bibliographic data or the only number in its full text; extractors close a study with no data when its reports belong to another trial, and the coordinator relinks it.