LLM multi-agent systems for automated quantitative research
Most of a quantitative paper is process: frame the question, prepare the data, choose an estimator, check that the choice was defensible, find the literature, write it up. Each step is describable, which makes the whole pipeline a reasonable target for a system of language-model agents. Whether the output is any good is an empirical question, and that is the part I care about.
EDM-ARS
I lead a four-person team building EDM-ARS, an open-source, domain-specific multi-agent system for educational data mining research. A state machine coordinates six specialized agents, each with its own prompt and model routing, from problem formulation through to a manuscript draft, with checkpoint-based recovery and code that runs in a sandbox rather than being pasted into the prose.
The agents run in order. A problem formulator searches Semantic Scholar and scopes the question and hypothesis. A data engineer cleans the features and emits the subgroup labels that later fairness analysis needs. An analyst runs the battery appropriate to the study type: machine learning with SHAP and subgroup fairness, causal estimators with balance and overlap diagnostics, or item response models through an R bridge. An outline agent plans the paper section by section against the results that actually came back, and a writer fills a LaTeX template rather than composing free-form prose.
What distinguishes it from a general-purpose research agent is the sixth agent, a gatekeeper. It reviews everything the others have produced and returns one of three verdicts: pass, revise, or abort. On revise, targeted instructions go back to whichever earlier agent needs them, for up to two cycles before the writer is unblocked regardless. Citations are verified against Semantic Scholar rather than trusted to the model’s memory, and the backend is model-agnostic, running locally deployed open-weight models as well as hosted APIs.
We benchmarked EDM-ARS against two general-purpose automated research systems on federal education data, and used the comparison to build a defect taxonomy for AI-generated studies: a catalogue of the specific ways these systems go wrong, which is more useful than a single quality score.
The system is at edmars.ai, with a technical report at arXiv:2603.18273.
Models that explain themselves
A second strand asks a narrower question. A causal machine learning model can estimate an individual treatment effect accurately and explain nothing. So we distilled one into a small open-weight language model, fine-tuned with LoRA, that returns three things at once: the effect estimate, a plain-language explanation of it, and a recommended action.
The interesting problem is not the fine-tuning but the guarantee. An explanation that sounds right and misdescribes the model is worse than no explanation, so a faithfulness audit gates what enters the training corpus, and the explanations are evaluated against the upstream estimator rather than judged on fluency.
Papers and talks
-
EDM-ARS: A Domain-Specific Multi-Agent System for Automated Educational Data Mining Research
Technical report
2026 -
Domain-Specific Multi-Agent Systems for Automated Educational Data Mining Research and Review
SREE 2026, oral presentation
2026 -
Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics
Under review