Conceptual overview: the living review pipeline
Source:vignettes/conceptual_overview.Rmd
conceptual_overview.RmdSystematic reviews go out of date quickly. A living review is updated as new evidence appears. metawoRld keeps the review’s data in plain, validated, traceable files; tools put data in through one door.
- metawoRld owns the review: settings, forms describing what is extracted, the search and screening log, accepted study data with provenance and verification, analyses and the website. It has no AI dependencies.
-
Channels put data in. Each channel produces
submissions that pass the same intake gate
(
submit_study()): the entry form, CSV files filled in with any editor, scripts, or DataFindR, which uses large language models to screen records and extract data. DataFindR depends on metawoRld, never the other way around.
The full format is defined in SPEC.md in the package
repository.
The project
create_metawoRld() creates a folder (by default a Git
repository):
my-review/
├── _metawoRld.yml # settings, criteria, spec_version, license
├── _rules.yml # variable vocabulary and value checks
├── _analyses.yml # planned meta-analyses
├── people.yml # people (with roles) and tools
├── searches.yml # every search that was run
├── records.csv # one row per unique record (search hit)
├── screening.csv # screening decisions, append-only
├── forms/
│ ├── extraction.yml # what is extracted from each study
│ └── screening.yml # what a screening decision looks like
├── prompts/ # LLM instructions (used by DataFindR)
├── submissions/ # extractions waiting to be accepted
│ └── <study_id>/<extraction_id>/
├── data/ # accepted data
│ └── <study_id>/
│ ├── study.yml # identity, reports (DOIs/PMIDs), form version
│ ├── metadata.yml # design, population, methods, groups
│ ├── data.csv # one row per reported statistic
│ ├── provenance.csv # who/what produced each value, from which source
│ ├── verification.csv # reviewers' decisions per value
│ └── datapackage.json # Frictionless descriptor with file hashes
└── index.qmd, _quarto.yml, ... # website
A study (one cohort or trial, id like
hansen2019) can have several reports
(publications, identified by DOI or PMID). Records are
search hits; an included record is linked to its study.
Forms
The default extraction form is generic: studies have
methods and groups, and every reported
statistic is a data point with a method_id,
group_id, variable_name,
timepoint, sample_size,
statistic_type and up to three values. Edit
forms/extraction.yml to fit your review: add fields,
tighten descriptions, or turn free-text fields into enums. Bump the
form’s version when you change it.
_form:
name: extraction
kind: extraction
version: 1.0.0
...
group_id:
_type: string # object, array, string, integer, number, boolean, enum
_description: "Shown to the LLM and in the entry form."
_required: true # default
_ref: groups.group_id # must match an id in metadata$groupsexport_schemas() writes JSON Schemas and Frictionless
Table Schemas, so channels written in other languages can validate their
output.
Groups and subgroups
Groups are recorded as the paper reports them. A subgroup points to
the larger group it is part of (parent_group_id) and
inherits its facts (category, and fields you add such as
onset). Categories in _rules.yml are rules
over these facts
(PE_early: {where: {category: PE, onset: early}}), so a new
category never needs re-extraction, and a group can belong to several
categories.
Every analysis uses one level per study and arm: a study that reports
both a total and its subgroups contributes the total (or, with
levels: pool, its complete subgroups pooled), never both.
Subgroup analyses split a shared reference group between the subgroups.
The analysis result and the dashboard show which groups each study
contributed, so you can check that the comparison is apples to apples.
Refinements during the review (add_group_fact(),
refinement_queue(), amend_study()) go through
the same door as extractions.
One door, then review
-
Set up with
create_metawoRld(),add_person(), and adjust the forms and criteria. -
Identify records:
add_search()andadd_records()(DataFindR’sdf_search_pubmed()does both). -
Screen:
add_screening()stores each person’s (or tool’s) decision;screening_status()shows conflicts;prisma_counts()feeds the PRISMA diagram. -
Extract: any channel calls
submit_study(). Invalid data are refused with every problem listed; plausibility checks from_rules.ymlare recorded as warnings. -
Accept:
accept_submission()for single extraction, orcompare_extractions()andreconcile()for dual extraction.provenance.csvrecords which extraction each value came from and which others agreed. -
Verify values against the papers with
review_values(). Decisions are tied to a hash of the values, so a changed value returns to the queue. -
Check with
check_project(), locally and in CI (use_github_action_check()). It recomputes the file hashes indatapackage.json, so data edited outside metawoRld are caught. -
Analyse and publish with
run_all_meta_analyses()andgenerate_website(); release versions as signed Git tags archived on Zenodo.