Quantification and Valuation of Uncertainty of Calibrated Parameters in Decision Models
Published in 40th Annual North American Meeting of the Society for Medical Decision Making, 2018
Abstract
Purpose
Decision models parameters often require estimation through model calibration to observed epidemiological data. There is little guidance on how to characterize uncertainty of calibrated parameters to examine their impact on decision uncertainty. We evaluate the implications of different approaches to account for the uncertainty of calibrated parameters and the value of such uncertainty in decision making.
Methods
We used a simulation model of colorectal cancer (CRC) screening to quantify the cost-effectiveness of a 10-year colonoscopy screening vs. no screening. We calibrated the natural history (NH) model of CRC to epidemiological data (i.e., calibration targets) with different degrees of uncertainty. To accurately propagate the uncertainty from the calibration targets to the calibrated parameters, we conducted a Bayesian calibration using an incremental mixture importance sampling (IMIS) algorithm. The Bayesian approach combines both prior knowledge of the NH parameters and the amount of information on the targets. We conducted a probabilistic sensitivity analysis of all the model parameters with different quantifications of uncertainty of the calibrated parameters, including (1) posterior distribution obtained from the IMIS algorithm, (2) a maximum-a-posteriori (MAP) estimate as the best parameter set, and (3) distributions based solely on posterior means, ranges and type of parameters (naïve approach). We estimated the value of uncertainty of calibrated parameters with a value of information (VOI) analysis.
Results
Different quantifications of uncertainty of the calibrated parameters changed the willingness-to-pay threshold (WTP) of USD per quality-adjusted life year (QALY) at which the screening strategy has the highest expected benefit. For both the IMIS posterior and the MAP approaches for quantification of uncertainty, 10-year colonoscopy screening has the highest benefit (compared with no screening) at WTP greater than $51,000/QALY gained and for the naive approach at values greater than $41,000/QALY gained. In the VOI analysis, the naïve approach had the highest expected value of perfect information (EVPI) of $5,700 at a WTP of $41,000/QALY gained, while with the IMIS posterior and MAP estimates EVPI was $630 and $572, respectively, both at a WTP of $51,000/QALY gained (Figure).
Conclusions
Different quantifications of uncertainty of calibrated decision model parameters affect the WTP at which strategies are optimal. Furthermore, the value of such uncertainties varies significantly. Accurate uncertainty quantification of calibrated parameters should be performed in model calibration through a Bayesian approach.