Causal inference: a methods toolkit

From Data to Bedside · the full write-up

When this applies

Use this write-up when you have a causal question (does treatment X cause outcome Y) and randomization is not in the cards. The decision framework here walks through choosing among the main causal-inference designs (DiD, RDD, IV, synthetic control, propensity-score methods) and stating the identifying assumption each design relies on. It assumes you’ve already worked through study design (the research question is specified in PICO and target-trial terms) and populations and sample size (the study population is defined and the sample is in hand).

The deliverables a careful pass produces:

The causal estimand stated explicitly (ATT, ATE, LATE)
A map of the threats to identification
A design choice that addresses the dominant threat, with rationale documented
The identifying assumption stated as a single defensible sentence
The estimator that operationalizes the design
A defense plan for the identifying assumption (event-study for DiD, bandwidth sensitivity for RDD, instrument-relevance and exclusion-restriction tests for IV, balance diagnostics for propensity scores)
The pre-specified robustness checks (sensitivity and robustness covers these in detail)

If your data come from a randomized trial, most of this write-up doesn’t apply; randomization handles identification, and the methodological work is on study design, populations, and sensitivity analysis. If your data come from an observational source, this write-up is the core.

The decision framework

Choosing and defending a causal-inference design is a sequence of seven decisions.

Step 1. Specify the causal estimand

The ladder names the estimands; the design question is which one to target, and what each commits you to claiming. For most policy evaluations, ATT is the right target: the effect among those who actually received treatment, which is what a program’s realized cost-effectiveness turns on. For most clinical-effectiveness questions, ATE: the population-average effect, which is what a guideline generalizes to patients who weren’t in the study. For IV-based designs, LATE is what you actually estimate whether you wanted it or not, the effect among the compliers whose treatment status the instrument moves, so an honest report names that complier subpopulation rather than implying the whole population. Write the estimand explicitly; it dictates the rest of the design.

Step 2. Map the threats to identification

For each of the standard threats to causal inference, ask whether it plausibly applies:

Confounding. A common cause of treatment and outcome. The most pervasive threat in observational data.
Selection bias. Who gets included in the study correlates with treatment and outcome.
Measurement error. Treatment, outcome, or confounders are mis-measured in a way that biases the estimate.
Reverse causation. The outcome causes the treatment, not the other way.
Spillover or interference. Treating one unit affects untreated units, violating the stable unit treatment value assumption (SUTVA).

Rank them. The dominant threat, the one most likely to bias your estimate, is the design problem Step 3 has to solve; the others become robustness checks rather than design drivers.

Step 3. Pick the design that addresses the dominant threat

The five main causal-inference design families, with their use cases:

Difference-in-differences (DiD). Treatment applied at a known date to a known group; an untreated comparison group exists; pre- and post-treatment data are available. Use when the policy is uniformly applied at a single date (the Part D insulin cap, the Medicaid expansion, an FDA approval). For staggered roll-outs, use Callaway–Sant’Anna or Sun–Abraham estimators rather than plain two-way fixed-effects, which can be biased.
Regression discontinuity (RDD). Treatment is assigned by a cutoff on a continuous score (CKD stage, BMI threshold, eligibility age, test score). Use when there’s a clean threshold-based assignment rule.
Instrumental variables (IV). A variable (the instrument) affects treatment but not the outcome directly. Use when you can argue for a credible instrument: distance-to-treatment, genetic variants (Mendelian randomization), policy changes that affect treatment uptake but not the outcome through other paths.
Synthetic control. One or a few treated units, many untreated units, a long pre-treatment period. Use for state-level policy evaluations (one or two treated states, many untreated states) or rare-event treatments.
Propensity-score methods (matching, weighting, doubly-robust). Many treated and untreated units, with measured confounders rich enough to address the dominant threats. Use when no clean natural experiment is available but the confounders are measured.

If the dominant threat is unmeasured confounding and you have no instrument, no natural experiment, no threshold-based assignment, and no untreated comparison group, you may not have a causal study, only an associational one. That four-part check is the fastest way to find out before you have committed a protocol to a question the data cannot answer.

Step 4. State the identifying assumption explicitly

The ladder names each design’s identifying assumption. The deliverable here is not to re-list them but to write yours as one sentence precise enough to attack, since the Step 6 defense is built against exactly that sentence. The one the ladder omits is synthetic control’s, pre-treatment fit: the donor-weighted combination that matches the treated unit before treatment also approximates its counterfactual after. It is the weakest of the set, because a good pre-period match is necessary but not sufficient for a good counterfactual. The precision bar is the whole point: “parallel trends” is not yet a sentence you can defend, while “absent the cap, insulin and the non-insulin antihyperglycemics would have moved together in log fills through 2024” is.

Step 5. Choose the estimator that fits the design

The design tells you the family; the data tell you which estimator inside it, and why.

Design	Estimator	Why this one
DiD, simultaneous treatment	two-way fixed-effects (TWFE)	unbiased when every unit is treated on the same date
DiD, staggered roll-out	Callaway–Sant’Anna, Sun–Abraham, or de Chaisemartin–D’Haultfœuille	plain TWFE is biased under heterogeneous timing (the “forbidden comparison” of already-treated against newly-treated units); these weight the $2 \times 2$ comparisons correctly
RDD	local linear regression, triangular kernel, optimal bandwidth (Calonico–Cattaneo–Titiunik)	uses only observations near the cutoff, where the continuity assumption actually holds
IV	two-stage least squares (2SLS) or limited-information maximum likelihood (LIML)	LIML is the more robust choice when the instrument is weak
Synthetic control	Abadie–Diamond–Hainmueller weight optimization (or augmented, Ben-Michael–Feller–Rothstein)	builds a donor-weighted counterfactual; the augmented version corrects residual pre-period imbalance
Propensity-score	matching, optimal full matching, IPTW with stabilized weights, or augmented IPW (AIPW)	AIPW is doubly-robust: consistent if either the propensity-score model or the outcome model is right, not both

Step 6. Plan the assumption-defense

For each design, there’s a standard defense move. Pre-specify it before estimating.

DiD: event-study with year-by-treatment interactions, plus a formal F-test on pre-period leads. Sensitivity and robustness covers when and how this defense fails.
RDD: discontinuity tests in covariates at the cutoff (no jump means assignment is locally random), the McCrary density test for manipulation, bandwidth sensitivity.
IV: first-stage F-statistic for relevance, the Hansen J-test for over-identification when multiple instruments are available, a written argument for the exclusion restriction (which cannot be formally tested with a single instrument).
Synthetic control: in-time placebo (test whether the synthetic control “predicts” earlier pseudo-treatments), in-space placebo (assign the treatment to each untreated unit in turn and compare effect sizes).
Propensity-score methods: balance diagnostics (standardized differences before and after matching or weighting), overlap diagnostics (propensity-score distributions across treated and control).

Step 7. Pre-specify the robustness checks

Sensitivity and robustness covers sensitivity analysis in detail. The Step-7 discipline here is parallel to the study-design write-up: while you’re designing the primary causal estimator, list the methodological choices most likely to be challenged at review. Alternative comparator definitions, alternative outcome operationalizations, alternative bandwidths or matching specifications, alternative instruments. Each is a candidate for pre-specified robustness analysis.

Target trial emulation as a framing layer

Across all five design families, the ladder’s target-trial emulation is the framing layer that makes the causal target explicit. The point specific to this rung: it pairs particularly cleanly with propensity-score methods (the emulation specifies the trial, the propensity score handles the assignment), and running it first is what lets the observational analysis name its estimand rather than back into one.

Worked example

The Part D insulin DiD case study walks through the seven-step framework applied to a policy-evaluation question. The IRA’s $35 insulin cap was a simultaneous treatment with a clean control group, plain TWFE is unbiased here, the parallel-trends assumption is the identifying claim, the event-study tests it visually, and the placebo cap-years plus leave-one-out plus drop-GLP-1 robustness checks each earn their place in the protocol. The case study also surfaces the harder methodological points: when the data are pre-filtered on the test statistic, when the empirical null is heavier-tailed than the textbook assumes, and when the natural experiment’s pre-period is partially contaminated by an earlier demonstration (the CMS Senior Savings Model).