← All publications
BETA This research page is an early version. Explanatory text, figure selection, and presentation are still being revised.
TMLR 2023

A Free Lunch with Influence Functions? An Empirical Evaluation of Influence Functions for Average Treatment Effect Estimation

Matthew J. Vowels · Sina Akbari · Necati Cihan Camgoz · Richard Bowden
Transactions on Machine Learning Research · 2023
Paper GitHub TMLR
Keywords.
causal effect estimation influence functions semiparametric statistics machine learning

TL;DR.

We run a broad empirical evaluation of semiparametric average-treatment-effect estimators, including neural-network nuisance models and influence-function updates. We also introduce MultiNet and MultiStep. The central finding is that no single update dominates everywhere, but several combinations are consistently strong and MultiNet performs particularly well in a number of evaluations.

Why this matters.

Causal estimators are often deployed without access to ground truth. A large controlled benchmark therefore provides practical evidence about which semiparametric pipelines are stable across datasets, sample sizes, and data-generating mechanisms—and where new methods such as MultiNet help.

Abstract

The applications of causal inference may be life-critical, including the evaluation of vaccinations, medicine, and social policy. However, when undertaking estimation for causal inference, practitioners rarely have access to what might be called ‘ground-truth’ in a supervised learning setting, meaning the chosen estimation methods cannot be evaluated and must be assumed to be reliable. It is therefore crucial that we have a good understanding of the performance consistency of typical methods available to practitioners. In this work we provide a comprehensive evaluation of recent semiparametric methods (including neural network approaches) for average treatment effect estimation. Such methods have been proposed as a means to derive unbiased causal effect estimates and statistically valid confidence intervals, even when using otherwise non-parametric, data-adaptive machine learning techniques. We also propose a new estimator ‘MultiNet’, and a variation on the semiparametric update step ‘MultiStep’, which we evaluate alongside existing approaches. The performance of both semiparametric and ‘regular’ methods are found to be dataset dependent, indicating an interaction between the methods used, the sample size, and nature of the data generating process. Our experiments highlight the need for practitioners to check the consistency of their findings, potentially by undertaking multiple analyses with different combinations of estimators.

Main message. This paper is an empirical map of modern semiparametric ATE estimation: which combinations work well, how performance changes with the dataset and sample size, and which new neural estimators are especially promising.

The paper at a glance

An informal guide to the problem, the idea, and the main results.

average treatment effect influence functions double robustness empirical evaluation

1. The problem

Influence-function-based estimators are often presented as a standard upgrade over plug-in estimation. In practice, however, the gain depends on how nuisance models are fit, on overlap, and on the structure of the data-generating process.

2. The key idea

We run a careful empirical study rather than relying only on asymptotic folklore. That lets us compare estimation pipelines under realistic finite-sample conditions and see when influence-function corrections are actually worth it.

3. MultiNet and MultiStep

Method contribution

We introduce MultiNet, a neural pseudo-ensemble outcome estimator, and MultiStep, a variation on the semiparametric update step, and evaluate them alongside established semiparametric and non-semiparametric estimators.

4. What drives performance

Empirical finding

Across the benchmark, estimator performance depends strongly on the interaction between method choice, sample size, and the data-generating process. MultiNet is among the strongest outcome models in several evaluations and performs particularly well in smaller-sample settings.

5. How it works

  1. Define a collection of treatment-effect estimation pipelines.
  2. Fit nuisance components with flexible machine-learning models.
  3. Compare plug-in and influence-function-corrected estimators across many settings.
  4. Read off when the correction is genuinely helpful in finite samples.

What the experiments show

Main empirical message.

The large benchmark shows that the best-performing pipeline changes across datasets and sample sizes, so robustness is better assessed across several competitive estimators than through a single default recipe. MultiNet achieves state-of-the-art performance on a number of evaluations, while the Shapley analyses make sample size and dataset structure two of the strongest predictors of estimator behavior.

Where this helps

Good fit

  • You use machine learning for nuisance estimation in treatment-effect problems.
  • You want empirical guidance on when influence-function corrections are worth the extra machinery.
  • You care about finite-sample behavior, not just asymptotic theory.

Keep in mind

  • The paper compares outcome models, propensity models, and update steps as a full estimation pipeline rather than evaluating each component in isolation.
  • MultiNet and MultiStep are provided as concrete new options within a benchmark designed to help practitioners choose stable semiparametric estimators.

Cite this paper

@article{vowels2023free,
  title   = {A Free Lunch with Influence Functions? An Empirical Evaluation of Influence Functions for Average Treatment Effect Estimation},
  author  = {Matthew J. Vowels and Sina Akbari and Necati Cihan Camgoz and Richard Bowden},
  journal = {Transactions on Machine Learning Research},
  year    = {2023},
  url     = {https://openreview.net/forum?id=dQxBRqCjLr}
}