Calibration and Decision Analysis

Why there is a lineup_member column

lineup_example records two further columns, for a reason that applies to any lineup data you collect or simulate yourself.

identification collapses a witness’s decision into three categories. That is what the analysis functions consume, but it is not what should be recorded. The raw datum is which lineup member the witness chose — lineup_member, with 0 for a rejection — together with the suspect’s position, suspect_pos. The label is then derived:

head(classify_lineup_choices(
  lineup_example[, c("lineup_member", "suspect_pos")],
  k = 6
))
  lineup_member suspect_pos identification
1             2           1         filler
2             3           1         filler
3             2           1         filler
4             1           1        suspect
5             1           1        suspect
6             0           1         reject

This is important because foils are not interchangeable. They differ in how plausible they are, and a foil that nobody ever chooses is not a functioning alternative to the suspect at all — it is precisely what makes a lineup unfair. Recording only “filler” throws that information away. In lineup_example the five foils attract very different numbers of choices:

lineup_table(
  lineup_example$lineup_member[lineup_example$lineup_member > 0],
  k = 6, suspect_pos = 1
)
Lineup choices (k = 6, N = 1132)

 pos:  1*    2    3    4   5   6
 cnt: 503  248  203  115  53  10

 * = nominated suspect

lineup_example is simulated, and it is simulated at the level of individual foils: each is given its own mean memory strength rather than being drawn from a common distribution. See ?simulate_lineup_data and its foil_bias argument if you want to generate data of your own on the same basis.

Basic Calibration

# Compute calibration with binned confidence
cal_result <- make_calibration(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100),
  choosers_only = TRUE  # Analyze only suspect identifications
)

print(cal_result)

=== Lineup Calibration Analysis ===

Population: Suspect identifications 
Total N: 503 

Calibration Statistics:
  C (Calibration):         0.0727 
  O/U (Over/Under):        -0.2290 
  NRI (Resolution):        0.0453 

Overall Performance:
  Mean Accuracy:      0.825 
  Mean Confidence:    0.596 

Calibration Data by Bin:
# A tibble: 3 × 7
  bin          n mean_confidence accuracy n_correct n_incorrect
  <chr>    <dbl>           <dbl>    <dbl>     <dbl>       <dbl>
1 [0,60]     305            42.6    0.764       233          72
2 (60,80]     94            74.0    0.872        82          12
3 (80,100]   104            96.5    0.962       100           4
# ℹ 1 more variable: mean_confidence_prop <dbl>

Plot: save this object first, then extract it with x$plot

The confidence values in lineup_example (20 to 100) are ordinal categories that carry probability labels by stipulation, not probabilities the witnesses stated, so O/U depends on those labels. Relabelling the same nine categories as 90 to 100 changes O/U from -0.229 to +0.124, with the same accuracy in every bin. Accuracy here is also the proportion of suspect identifications that are guilty-suspect identifications, which depends on the 50:50 mix of target-present and target-absent lineups in the data. The figures below illustrate the output; they do not describe witness behaviour.

The output provides three key statistics:

• C statistic: Overall calibration quality (0 = perfect, higher = worse)
• O/U (Over/Under confidence): Positive = overconfident, Negative = underconfident
• NRI (Normalized Resolution Index): Ability to discriminate accuracy levels with confidence (higher = better)

Interpreting the Calibration Plot

The calibration plot shows:

• Diagonal line: Perfect calibration
• Points above diagonal: Underconfident (accuracy exceeds confidence)
• Points below diagonal: Overconfident (confidence exceeds accuracy)
• Point size: Sample size in each bin

cal_result$plot

Accuracy rises steadily across the confidence bins, from about 76% in the lowest to about 96% in the highest, so confidence here is genuinely informative. The points sit above the diagonal, however, so with the 20 to 100 labels the confidence categories are underconfident: accuracy exceeds the label. That is the direction the O/U statistic reports, and it follows from the labels chosen, as noted above.

The plot is an ordinary ggplot2 object, so it can be modified and saved in the usual way. ggsave() picks its output format from the file extension alone, so the same call writes PNG, PDF, SVG or TIFF depending on what you ask for:

outfile <- file.path(tempdir(), "calibration_plot.png")
ggplot2::ggsave(outfile, cal_result$plot, width = 8, height = 6, dpi = 300)

# Change only the extension to change the format:
#   ggsave("calibration_plot.pdf", cal_result$plot)   # vector, for print
#   ggsave("calibration_plot.svg", cal_result$plot)   # vector, for the web

Calibration by Condition

The question here is whether calibration differs between two identification procedures. To answer it we need data from two procedures, and lineup_example records only one, so we simulate them.

The comparison is only meaningful if the two conditions genuinely differ. We give Procedure A the stronger memory evidence (d_prime = 1.8 against 1.1), as might arise from better viewing conditions or a better-constructed lineup, and give the two lineups different foil quality as well. Note that both are seeded, so the results below are reproducible rather than changing on every rebuild.

# Procedure A: stronger evidence, and foils that are reasonable alternatives
data_proc_a <- as.data.frame(simulate_lineup_data(
  n_tp = 400, n_ta = 400, d_prime = 1.8, c_criterion = 0.9,
  lineup_size = 6, conf_levels = 9,
  foil_bias = c(0.4, 0.2, 0.0, -0.3, -0.8), seed = 101
))

# Procedure B: weaker evidence, and one near-useless foil
data_proc_b <- as.data.frame(simulate_lineup_data(
  n_tp = 400, n_ta = 400, d_prime = 1.1, c_criterion = 0.9,
  lineup_size = 6, conf_levels = 9,
  foil_bias = c(0.5, 0.1, -0.2, -0.9, -2.0), seed = 102
))

# Put confidence on the same 20-100 scale used by lineup_example
conf_scale <- round(seq(20, 100, length.out = 9))
data_proc_a$confidence <- conf_scale[data_proc_a$confidence]
data_proc_b$confidence <- conf_scale[data_proc_b$confidence]

# Combine into one frame with a condition variable
procedures <- rbind(
  transform(data_proc_a, procedure = "Procedure A"),
  transform(data_proc_b, procedure = "Procedure B")
)

cal_by_cond <- make_calibration_by_condition(
  procedures,
  condition_vars = "procedure",
  confidence_bins = c(0, 60, 80, 100)
)

knitr::kable(
  cal_by_cond$condition_summary, digits = 3,
  caption = "Calibration statistics by procedure, simulated data (suspect identifications)."
)

The per-bin detail for each procedure is in cal_by_cond$by_condition. The object does not carry a plot; draw one with make_calibration_by_condition_gg():

make_calibration_by_condition_gg(cal_by_cond)

The arithmetic, briefly

Let \(G\) denote the event that the suspect is guilty and \(x\) a particular identification response — say, a suspect identification made with confidence above 80. Bayes’ theorem gives the posterior probability of guilt as

\[ P(G \mid x) \;=\; \frac{P(x \mid G)\,P(G)}{P(x \mid G)\,P(G) + P(x \mid \neg G)\,\bigl(1 - P(G)\bigr)} \]

The two conditional probabilities are estimated directly from the data: \(P(x \mid G)\) is the proportion of target-present witnesses who gave response \(x\), and \(P(x \mid \neg G)\) the proportion of target-absent witnesses who did. Their ratio is the likelihood ratio,

\[ \mathrm{LR}(x) \;=\; \frac{P(x \mid G)}{P(x \mid \neg G)} \]

and it is the whole of what the identification contributes. An \(\mathrm{LR}\) of 1 means the response is uninformative; larger values point towards guilt, smaller towards innocence. make_bayes_curves() returns these quantities in $likelihoods:

lik <- make_bayes_curves(
  lineup_example,
  response_categories = "confidence",
  confidence_bins = c(0, 60, 80, 100)
)$likelihoods

knitr::kable(
  transform(
    as.data.frame(lik)[, c("response", "p_x_given_guilty",
                           "p_x_given_innocent", "n_guilty", "n_innocent")],
    LR = round(lik$p_x_given_guilty / lik$p_x_given_innocent, 2)
  ),
  digits = 3, row.names = FALSE,
  caption = "Response likelihoods and likelihood ratios, lineup_example."
)

A high-confidence suspect identification carries a likelihood ratio of roughly 25 here, while a rejection carries one of about 0.3 — that is, it is evidence against guilt. Note also that the likelihood ratio, not the posterior, is the portable quantity: it belongs to the identification procedure, whereas the posterior additionally depends on the prior, which belongs to the investigation.

One practical warning. If no target-absent witness ever produces a given response, the estimated \(P(x \mid \neg G)\) is zero, the likelihood ratio is infinite, and the posterior curve is pinned at 1.0 for every prior. That is almost always a sign of too few observations or too narrow a bin rather than of perfectly diagnostic evidence, and make_bayes_curves() warns when it happens.

Prior-Posterior Curves

# Compute Bayesian curves for simple responses
bayes_result <- make_bayes_curves(
  lineup_example,
  response_categories = "simple",  # "suspect", "filler", "reject"
  prior_grid = seq(0.01, 0.99, 0.01)
)

print(bayes_result)
$curves
# A tibble: 297 × 4
   response prior posterior information_gain
   <chr>    <dbl>     <dbl>            <dbl>
 1 filler    0.01   0.00533          0.00467
 2 filler    0.02   0.0107           0.00929
 3 filler    0.03   0.0161           0.0139 
 4 filler    0.04   0.0216           0.0184 
 5 filler    0.05   0.0272           0.0228 
 6 filler    0.06   0.0327           0.0273 
 7 filler    0.07   0.0384           0.0316 
 8 filler    0.08   0.0441           0.0359 
 9 filler    0.09   0.0498           0.0402 
10 filler    0.1    0.0557           0.0443 
# ℹ 287 more rows

$likelihoods
# A tibble: 3 × 5
  response p_x_given_guilty p_x_given_innocent n_guilty n_innocent
  <chr>               <dbl>              <dbl>    <int>      <dbl>
1 filler             0.311               0.587      218        411
2 reject             0.0957              0.287       67        201
3 suspect            0.593               0.126      415         88

$response_counts
# A tibble: 3 × 3
  response n_guilty n_innocent
  <chr>       <int>      <dbl>
1 filler        218        411
2 reject         67        201
3 suspect       415         88

$n_guilty
[1] 700

$n_innocent
[1] 700

$response_categories
[1] "simple"

$lineup_size
[1] 6

$innocent_suspect_method
[1] "designated"

$gain
[1] "probability"

# Plot prior-posterior relationships
plot_bayes_prior_posterior(bayes_result)

The plot shows how each response type updates prior belief:

• Lines above diagonal: Evidence increases guilt belief (diagnostic of guilt)
• Lines below diagonal: Evidence decreases guilt belief (diagnostic of innocence)
• Distance from the diagonal: at a given prior, the further a curve lies from the diagonal, the larger the likelihood ratio of that response. Compare curves at the same prior, not by their local slopes.

The prior-posterior curve is due to Wells, Yang and Smalarz (2015), who introduced it along with the base-rate effect-equivalency curve used later in this vignette. The implementation here is written directly from that paper.

On the shape of these curves. They are curved, and that curvature is not an artefact of any smoothing: no loess or spline is fitted anywhere in these plots. For a response with likelihood ratio \(L\), the posterior as a function of the prior \(p\) is \[ q(p) = \frac{Lp}{1 + (L - 1)p}, \qquad q''(p) = \frac{-2L(L - 1)}{[1 + (L - 1)p]^3}. \] The curvature has one sign throughout: the curve is concave (bowed above the diagonal) when \(L > 1\), convex (bowed below it) when \(L < 1\), and the diagonal itself when \(L = 1\). It is not S-shaped. What the display shows is that the same response moves belief very little when the prior is already near 0 or near 1. The absolute change \(|q(p) - p|\) peaks at the prior \(1/(1 + \sqrt{L})\): about .17 for a response with \(L = 25\), such as the highest-confidence suspect identification, and about .64 for a rejection with \(L = 0.33\). Only responses with \(L\) near 1 peak near .5.

Information Gain

Wells and Lindsay (1980) and Wells, Yang and Smalarz (2015) measure the information gained from a response as the absolute change in the probability of guilt, |posterior − prior|: what counts is how far the response moves belief, not its direction. This is the default (gain = "probability"), and for a response with likelihood ratio \(L\) it is largest at the prior \(1/(1 + \sqrt{L})\), at lower priors for stronger incriminating responses. gain = "signed" keeps the direction, positive for responses that point to guilt and negative for responses that point to innocence:

plot_bayes_information_gain(bayes_result)

A second measure, used by Starns, Cohen and Rotello (2023) in expected information gain, is the reduction in uncertainty in bits: the entropy of the prior minus the entropy of the posterior (gain = "entropy"):

bayes_entropy <- make_bayes_curves(
  lineup_example,
  response_categories = "simple",
  prior_grid = seq(0.01, 0.99, 0.01),
  gain = "entropy"
)
plot_bayes_information_gain(bayes_entropy)

The two measures answer different questions and can differ in sign. A suspect identification given a low prior raises the probability of guilt towards .5: its probability gain is positive, but its entropy reduction is negative because .5 is the most uncertain state. That is a correct update, not misleading evidence. Responses that point to guilt reduce entropy most at priors above .5, and responses that point to innocence reduce it most at priors below .5. State which measure you report.

Confidence-Based Bayesian Analysis

Analyze by confidence level:

bayes_conf <- make_bayes_curves(
  lineup_example,
  response_categories = "confidence",
  confidence_bins = c(0, 60, 80, 100)
)

# Get suspect responses only
suspect_responses <- grep("^suspect_", unique(bayes_conf$curves$response), value = TRUE)
plot_bayes_prior_posterior(bayes_conf, selected_responses = suspect_responses)

Splitting suspect identifications by confidence separates three quite different kinds of evidence. The highest-confidence band sits well above the others across the whole range of priors, which is the Bayesian restatement of the calibration result found earlier: high-confidence identifications from this procedure are worth considerably more than low-confidence ones.

The three curves stay clearly apart and none is pinned to the top of the plot. A curve that is pinned flat at 1.0 signals the zero-cell problem described above, not evidence of unusual strength.

Base-Rate Equivalency of Evidence (BREE)

The base-rate effect-equivalency curve, also from Wells, Yang and Smalarz (2015), answers a question that comes up whenever two procedures are compared and neither dominates: how much better is one than the other, expressed in units anyone can interpret?

The idea is to express the difference as an exchange rate between base rates. Suppose Procedure A, applied to a case with a 20% prior probability of guilt, yields a certain posterior after a suspect identification. We can then ask what prior Procedure B would need in order to reach that same posterior. If B needs a 40% prior to get where A got from 20%, then A is worth as much as a doubling of the base rate — and a doubling of the base rate is something an investigator has an intuition about, in a way that a difference in pAUC is not.

Plotting that correspondence for every prior gives the BREE curve. plot_bree() draws the base-rate shift \(\Delta\) = prior needed by B − prior used by A, against A’s prior. The dashed horizontal line at \(\Delta = 0\) is the null result of no difference; positive \(\Delta\) means B needs the larger prior, so A’s suspect identification is the more diagnostic, and the further the curve lies above zero, the larger A’s advantage. (Plotting B’s prior directly against A’s would put the reference on the diagonal instead, with A’s advantage above it.)

The two procedures are the ones simulated in the Calibration by Condition section: data_proc_a and data_proc_b are data frames of target-present and target-absent identifications with confidence, one row per witness (not lineup vectors). Procedure A has the stronger memory evidence (d′ = 1.8 against 1.1).

bree_result <- make_bree_curve(
  data_proc_a,
  data_proc_b,
  reference_response = "suspect"
)

plot_bree(bree_result)

The shift is positive throughout, from 0.003 to 0.079, so Procedure B always needs a higher prior probability of guilt to reach the posterior that A reaches — which is to say that a suspect identification from Procedure A is worth more. At a prior of 0.2 for A, B needs 0.255. The shift is largest, 0.079, near a prior of 0.46, and shrinks towards both ends of the scale, where neither procedure’s evidence can move belief much.

Define Utility Matrix

The costs and benefits must be stated explicitly. That is often felt to be the uncomfortable part of the exercise, but the values are being assumed either way; utility analysis only makes them visible and open to argument.

Four outcomes need a value, on any common scale:

utility_matrix <- c(
  tp =  1.0,   # Correct identification of a guilty suspect: a benefit
  fn = -0.5,   # Failing to identify a guilty suspect: a cost
  fp = -2.0,   # Identifying an innocent suspect: the most serious cost
  tn =  0.5    # Correctly declining to identify an innocent suspect: a benefit
)

Only two differences between these values affect which criterion is chosen, and any difference between procedures: tp - fn, the gain from acting on a guilty suspect rather than missing him or her (1.5 here), and tn - fp, the loss from acting on an innocent suspect rather than declining (2.5 here). The trade-off is the ratio of those differences, 2.5 to 1.5 or about 1.67; the raw ratio of -2.0 to -0.5 plays no part, and adding the same constant to tp and fn, or to tn and fp, leaves the optimum unchanged. That ratio is a moral and legal judgement, not a statistical one, and it is the natural place for disagreement. Blackstone’s ratio, that it is better for ten guilty people to escape than for one innocent person to suffer, would make tn - fp ten times tp - fn, which in this matrix needs fp = -14.5; the optimum then moves from a criterion of 30 to 100, the strictest one. Because the choice is this important, report the sensitivity of the conclusion to it, which is what the base-rate section does for the other key assumption.

Compute Utility Curves

# Compute expected utility across confidence criteria
util_result <- make_utility_curves(
  lineup_example,
  base_rate = 0.5,  # Assume 50% guilty suspects
  utility_matrix = utility_matrix,
  lineup_size = 6
)

print(util_result)
$utility_data
# A tibble: 10 × 7
   criterion strategy   hit_rate false_alarm_rate expected_utility n_ids_tp
       <dbl> <chr>         <dbl>            <dbl>            <dbl>    <dbl>
 1       100 threshold    0.0971          0                 0.0729       68
 2        90 threshold    0.143           0.00571           0.1         100
 3        80 threshold    0.189           0.0143            0.124       132
 4        70 threshold    0.26            0.0229            0.166       182
 5        60 threshold    0.351           0.03              0.226       246
 6        50 threshold    0.436           0.0543            0.259       305
 7        40 threshold    0.507           0.0771            0.284       355
 8        30 threshold    0.556           0.103             0.288       389
 9        20 threshold    0.593           0.126             0.288       415
10       101 reject_all   0               0                 0             0
# ℹ 1 more variable: n_ids_ta <dbl>

$max_utility
$max_utility$expected_utility
       tp 
0.2882143 

$max_utility$criterion
[1] 30

$max_utility$strategy
[1] "threshold"

$max_utility$hit_rate
[1] 0.5557143

$max_utility$false_alarm_rate
[1] 0.1028571


$avg_utility
[1] 0.2008333

$utility_all_ids
    tp 
0.2875 

$base_rate
[1] 0.5

$utility_matrix
  tp   fn   fp   tn 
 1.0 -0.5 -2.0  0.5 

$n_target_present
[1] 700

$n_target_absent
[1] 700

$innocent_suspect_method
[1] "designated"

$criteria
[1] "confidence"

# Plot utility curves
plot_utility_curves(util_result)

The plot shows:

• Expected utility at each confidence threshold
• Optimal threshold: Maximum utility point
• Comparison with “all identifications” strategy

Reading it from right to left: a very strict threshold acts on almost nothing, so little is gained and little is risked. Relaxing the threshold admits identifications that are mostly correct, and utility climbs. Past the optimum, each further relaxation admits more wrongful identifications than correct ones, which are weighted by tn - fp against tp - fn (2.5 against 1.5), and utility falls away again. The maximum is the criterion at which the marginal identification is exactly worth acting on.

This is the shape Lampinen et al. (2019) describe. The optimum reported is the in-sample maximum over the observed criteria, with no interval, so it is optimistic as an estimate of the best rule for new lineups. Here the curve is nearly flat at the top (0.284 at 40, 0.288 at 30 and 0.288 at 20), which means the data do not support a firm choice among those three criteria. Thresholds close to the optimum lose very little, but a flat curve is also a reason not to read the exact criterion that wins as more than a sample quantity.

Compare Procedures

max_utility is a list rather than a single number: it reports the best expected utility together with the criterion that achieves it, and the hit and false-alarm rates at that criterion. Ask for the element you want.

The optimum can be a confidence criterion or the decision to reject every identification. max_utility$strategy says which, so a small helper turns the result into a rule that can be stated:

utility_rule <- function(m) {
  if (m$strategy == "reject_all") "Act on no identification"
  else paste0("Act on confidence >= ", m$criterion)
}

# Compare the two procedures simulated in the Calibration by Condition section
util_proc_a <- make_utility_curves(data_proc_a, base_rate = 0.5,
                                   utility_matrix = utility_matrix)
util_proc_b <- make_utility_curves(data_proc_b, base_rate = 0.5,
                                   utility_matrix = utility_matrix)

knitr::kable(
  data.frame(
    Procedure = c("A (d' = 1.8)", "B (d' = 1.1)"),
    `Max expected utility` = round(c(util_proc_a$max_utility$expected_utility,
                                     util_proc_b$max_utility$expected_utility), 3),
    `Optimal rule` = c(utility_rule(util_proc_a$max_utility),
                       utility_rule(util_proc_b$max_utility)),
    check.names = FALSE
  ),
  caption = "Expected utility at the optimal confidence criterion, simulated data (400 target-present and 400 target-absent lineups per procedure)."
)

The procedure with the stronger evidence has the higher maximum expected utility in this simulation. The optimal criteria differ between the procedures, but the curves are nearly flat at the top (procedure A: 0.367 at 40 and 0.366 at 30; procedure B: 0.181 at 40 and 0.185 at 30), so that difference is incidental and should not be read as a property of the procedures.

Sensitivity to Base Rate

Examine how utility depends on base rate:

# Test multiple base rates
base_rates <- c(0.2, 0.5, 0.8)

sensitivity <- do.call(rbind, lapply(base_rates, function(br) {
  util <- make_utility_curves(lineup_example, base_rate = br,
                              utility_matrix = utility_matrix,
                              lineup_size = 6)
  data.frame(
    `Base rate` = br,
    `Max expected utility` = round(util$max_utility$expected_utility, 3),
    `Optimal rule` = utility_rule(util$max_utility),
    check.names = FALSE
  )
}))

knitr::kable(
  sensitivity,
  row.names = FALSE,
  caption = "Sensitivity of expected utility to the assumed base rate, lineup_example data."
)

Base rate is very important. When guilty suspects are rare, any given identification is more likely to be a false one, so the optimal criterion becomes stricter: here “act on confidence >= 60” at a base rate of 0.2, against “act on confidence >= 20” at 0.8. The attainable utility is not ordered by base rate in any simple way, because it also depends on how the cost matrix values the outcomes that become more common; here it is 0.345, 0.288 and 0.349 at the three base rates. Compare utilities across procedures at the same base rate, not across base rates.

Basic DPP

dpp_result <- make_dpp(
  lineup_example,
  lineup_size = 6
)

dpp_result$dpp
[1] 0.7001587
dpp_result$dpp_by_point
# A tibble: 9 × 4
  confidence culprit_rate innocent_rate   dpp
       <dbl>        <dbl>         <dbl> <dbl>
1        100       0.0971       0       0.903
2         90       0.143        0.00571 0.863
3         80       0.189        0.0143  0.826
4         70       0.26         0.0229  0.763
5         60       0.351        0.03    0.679
6         50       0.436        0.0543  0.619
7         40       0.507        0.0771  0.57 
8         30       0.556        0.103   0.547
9         20       0.593        0.126   0.533

plot_dpp(dpp_result)

The table lists each operating point, from the strictest threshold to the most lenient, which counts every identification. The plot shows DPP(\(c\)) at each of them, the average as a dashed line, and the chance reference at 1. The circled point has the lowest DPP and so the highest utility; it is the threshold at which acting on identifications would do the most good under equal costs and a base rate of .5.

Different costs and base rates

Smith et al.’s Equation 3 weights the two errors:

\[DPP_{\alpha\beta}(c) = \beta[1 - \text{Culprit}(c)] + \alpha(1 - \beta)\text{Innocent}(c),\]

where \(\alpha\) is the cost of identifying an innocent suspect relative to missing the culprit, and \(\beta\) is the base rate of culprit presence. With \(\alpha = 1\) and \(\beta = .5\) this is half of Equation 2. Weighted values no longer range from 0 to 2, so they are comparable only with other values computed with the same weights.

# An innocent-suspect identification three times as costly as a miss,
# with culprits present in 30% of lineups
make_dpp(lineup_example, alpha = 3, beta = 0.3)$dpp
[1] 0.296619

Compare DPP Between Groups

The comparison is the usual use. Here the two simulated procedures from earlier differ in the strength of the evidence they generate, so Procedure A should show the smaller deviation. As Smith et al. did, compare_dpp() tests the difference with a nonparametric bootstrap, resampling target-present and target-absent trials separately within each procedure:

# Stored with the vignette; set R4LINEUPS_PRECOMPUTE=true to recompute.
dpp_comparison <- compare_dpp(
  data_proc_a,
  data_proc_b,
  lineup_size = 6,
  n_bootstrap = 500,
  seed = 2026
)

dpp_comparison

=== DPP Comparison ===

Average DPP, procedure A:  0.6022 
Average DPP, procedure B:  0.8247 
Difference (A - B):        -0.2225  (negative = A better)
Bootstrap SE:              0.0268
95% percentile interval:  [-0.2797, -0.1726]  (500 resamples)

plot_dpp_comparison(dpp_comparison)

A negative difference favours Procedure A. The interval is a percentile bootstrap interval for the difference in average DPP.

When to Use DPP

Use DPP when:

• You have confidence-based ROC data
• ROC curves are truncated (witnesses may not use full scale)
• You want a single performance metric for procedures rated on the same confidence scale in the same study
• You need to compare procedures with different confidence distributions

Basic ANRI

# Compute ANRI
anri_result <- compute_anri(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100)
)

print(anri_result)
$anri
[1] 0.04145948

$nri
[1] 0.04527077

$n_total
[1] 503

$n_bins
[1] 3

$calibration_data
# A tibble: 3 × 7
  bin          n mean_confidence accuracy n_correct n_incorrect
  <chr>    <dbl>           <dbl>    <dbl>     <dbl>       <dbl>
1 [0,60]     305            42.6    0.764       233          72
2 (60,80]     94            74.0    0.872        82          12
3 (80,100]   104            96.5    0.962       100           4
# ℹ 1 more variable: mean_confidence_prop <dbl>

$overall_accuracy
[1] 0.8250497

$overall_confidence
[1] 0.5960239

$choosers_only
[1] TRUE

ANRI is always ≤ NRI because it removes positive bias. The bias of NRI is about (J − 1)/N, so the correction is larger when the sample size N is small, the number of bins J is large, or, in short, when the ratio J/N is high.

Bootstrap Confidence Intervals

A point estimate is not much use on its own, so the interval comes from a bootstrap. n_bootstrap = 100 keeps this vignette quick; use 2000 or more for anything you intend to report.

anri_boot <- bootstrap_anri(
  lineup_example,
  confidence_bins = c(0, 60, 80, 100),
  n_bootstrap = 100,
  conf_level = 0.95,
  seed = 123
)

knitr::kable(
  data.frame(
    Statistic = c("NRI (uncorrected)", "ANRI (bias-corrected)"),
    Estimate  = round(c(anri_boot$nri, anri_boot$anri), 3),
    `95% CI`  = c("—", sprintf("[%.3f, %.3f]",
                               anri_boot$ci_lower, anri_boot$ci_upper)),
    check.names = FALSE
  ),
  row.names = FALSE,
  caption = "Resolution with bootstrap interval, lineup_example, 3 confidence bins."
)

No interval is given for NRI because it would add nothing. ANRI is a linear rescaling of NRI, \(ANRI = (N \cdot NRI - J + 1)/(N - J + 1)\), so with \(N =\) 503 suspect identifications and \(J =\) 3 bins the two differ only by the bias correction (0.045 against 0.041 here). An interval for NRI would be the ANRI interval shifted upward by that bias, so it is ANRI and its interval that should be reported and compared.

Compare Groups

The comparison uses the two simulated procedures, which differ in the strength of the evidence they generate. The two procedures differ in d’ by construction, but that does not make the ANRI difference between these samples real: the interval below is what shows whether the data can separate them.

anri_comparison <- compare_anri(
  procedures,
  group_var = "procedure",
  confidence_bins = c(0, 60, 80, 100),
  n_bootstrap = 100,
  seed = 456
)

print(anri_comparison)
$anri_group1
[1] 0.06778469

$anri_group2
[1] 0.02682759

$nri_group1
[1] 0.07322035

$nri_group2
[1] 0.03571501

$difference
[1] 0.04095711

$ci_lower
[1] -0.0162656

$ci_upper
[1] 0.09288628

$conf_level
[1] 0.95

$significant
 2.5% 
FALSE 

$group_names
[1] "Procedure A" "Procedure B"

$n_bootstrap
[1] 100

$n_bootstrap_requested
[1] 100

$n_failed
[1] 0

$bootstrap_results_group1
$bootstrap_results_group1$anri
[1] 0.06778469

$bootstrap_results_group1$nri
[1] 0.07322035

$bootstrap_results_group1$ci_lower
[1] 0.02912137

$bootstrap_results_group1$ci_upper
[1] 0.1149089

$bootstrap_results_group1$conf_level
[1] 0.95

$bootstrap_results_group1$n_bootstrap
[1] 100

$bootstrap_results_group1$n_bootstrap_requested
[1] 100

$bootstrap_results_group1$n_failed
[1] 0

$bootstrap_results_group1$n_total
[1] 343

$bootstrap_results_group1$n_bins
[1] 3

$bootstrap_results_group1$bootstrap_distribution
  [1] 0.06708598 0.06570604 0.08047661 0.04987933 0.07221240 0.06148205
  [7] 0.07212585 0.08571190 0.06816675 0.06999423 0.05916390 0.09510135
 [13] 0.03142914 0.04505713 0.05639861 0.09915104 0.03885619 0.07716533
 [19] 0.05518367 0.06160329 0.08011800 0.03582855 0.05369086 0.08262844
 [25] 0.11270127 0.09078520 0.11118346 0.06107025 0.11095935 0.03686786
 [31] 0.07824433 0.10365266 0.06466421 0.07758830 0.06246753 0.04141894
 [37] 0.08328425 0.08469806 0.08008407 0.06726736 0.02703340 0.06715568
 [43] 0.06081143 0.04384714 0.06404480 0.08060125 0.09375399 0.04073706
 [49] 0.05561147 0.06537103 0.08259529 0.07587945 0.06234151 0.11957211
 [55] 0.06314098 0.02570420 0.04332888 0.09238761 0.10012490 0.11869072
 [61] 0.05276179 0.09538591 0.07677354 0.04634813 0.08990006 0.05857359
 [67] 0.10507931 0.07623093 0.11105741 0.07602279 0.08272019 0.04113382
 [73] 0.08082205 0.05360715 0.09284835 0.01839824 0.03796410 0.07069711
 [79] 0.07777091 0.08588510 0.11690619 0.06030880 0.10590118 0.10782745
 [85] 0.06639759 0.06499843 0.07933454 0.06463416 0.09112998 0.06640864
 [91] 0.06755599 0.07182240 0.09123954 0.07012033 0.09326807 0.08488978
 [97] 0.08600358 0.07038152 0.06785717 0.03480324


$bootstrap_results_group2
$bootstrap_results_group2$anri
[1] 0.02682759

$bootstrap_results_group2$nri
[1] 0.03571501

$bootstrap_results_group2$ci_lower
[1] 0.009110349

$bootstrap_results_group2$ci_upper
[1] 0.06445527

$bootstrap_results_group2$conf_level
[1] 0.95

$bootstrap_results_group2$n_bootstrap
[1] 100

$bootstrap_results_group2$n_bootstrap_requested
[1] 100

$bootstrap_results_group2$n_failed
[1] 0

$bootstrap_results_group2$n_total
[1] 219

$bootstrap_results_group2$n_bins
[1] 3

$bootstrap_results_group2$bootstrap_distribution
  [1] 0.054817973 0.022421218 0.026550014 0.047942448 0.026436048 0.051313022
  [7] 0.026715655 0.035606414 0.049266532 0.033202453 0.065100176 0.019272279
 [13] 0.017972351 0.024914903 0.031759105 0.023415879 0.039935663 0.016485797
 [19] 0.028106056 0.026755049 0.042393733 0.014044901 0.047430438 0.025694351
 [25] 0.009603482 0.031974042 0.025750449 0.023649520 0.016634482 0.034929618
 [31] 0.044207564 0.042362668 0.025991477 0.033185154 0.037640835 0.026160620
 [37] 0.011972828 0.007036663 0.041360143 0.024607043 0.027630872 0.030317349
 [43] 0.032542761 0.019641566 0.022733987 0.010714669 0.031485628 0.021524720
 [49] 0.021008661 0.029182388 0.023620195 0.032096170 0.032127258 0.039164861
 [55] 0.022085988 0.045748433 0.022176174 0.023266895 0.034157548 0.023283074
 [61] 0.064850996 0.008664181 0.014826744 0.024504903 0.010977051 0.019082504
 [67] 0.013783056 0.023553399 0.032484769 0.026147752 0.022248487 0.030642495
 [73] 0.007978659 0.057524655 0.027271420 0.064017894 0.080392203 0.014551924
 [79] 0.051559618 0.026730444 0.026491746 0.023184755 0.035857128 0.028955733
 [85] 0.019554188 0.048800873 0.022485638 0.034746265 0.046738180 0.030777372
 [91] 0.026480496 0.025514253 0.037440071 0.038654235 0.034623890 0.033019727
 [97] 0.053811990 0.030239350 0.028706437 0.010009702


$difference_distribution
  [1]  0.0122680065  0.0432848222  0.0539265958  0.0019368818  0.0457763559
  [6]  0.0101690294  0.0454101932  0.0501054840  0.0189002169  0.0367917783
 [11] -0.0059362774  0.0758290756  0.0134567857  0.0201422261  0.0246395094
 [16]  0.0757351573 -0.0010794705  0.0606795334  0.0270776135  0.0348482420
 [21]  0.0377242712  0.0217836462  0.0062604171  0.0569340862  0.1030977881
 [26]  0.0588111547  0.0854330116  0.0374207301  0.0943248658  0.0019382433
 [31]  0.0340367627  0.0612899930  0.0386727321  0.0444031450  0.0248266917
 [36]  0.0152583218  0.0713114246  0.0776613940  0.0387239227  0.0426603164
 [41] -0.0005974759  0.0368383268  0.0282686652  0.0242055779  0.0413108181
 [46]  0.0698865853  0.0622683605  0.0192123438  0.0346028060  0.0361886426
 [51]  0.0589750985  0.0437832779  0.0302142505  0.0804072525  0.0410549946
 [56] -0.0200442308  0.0211527091  0.0691207197  0.0659673547  0.0954076458
 [61] -0.0120892085  0.0867217281  0.0619467959  0.0218432288  0.0789230117
 [66]  0.0394910885  0.0912962539  0.0526775336  0.0785726379  0.0498750399
 [71]  0.0604717015  0.0104913202  0.0728433901 -0.0039175017  0.0655769263
 [76] -0.0456196567 -0.0424281019  0.0561451813  0.0262112892  0.0591546516
 [81]  0.0904144437  0.0371240484  0.0700440502  0.0788717134  0.0468434025
 [86]  0.0161975531  0.0568489007  0.0298878990  0.0443917958  0.0356312668
 [91]  0.0410754920  0.0463081431  0.0537994645  0.0314660929  0.0586441834
 [96]  0.0518700487  0.0321915922  0.0401421675  0.0391507322  0.0247935330

plot_anri_comparison(anri_comparison)

The ANRI difference is 0.041 with a 95% interval of [-0.016, 0.093], which includes 0, so with these samples and 100 resamples the ANRI difference between the procedures cannot be distinguished from zero. Use 2000 or more resamples for anything you report.

The comparison provides:

• Point estimates for each group
• Difference with CI: Tests whether groups differ
• Significance test: CI excludes 0 = significant difference

The bootstrap difference distribution is available as plot_anri_difference_distribution(anri_comparison) if you want to inspect its shape. Inspect it when an interval looks surprising: a heavily skewed or multimodal distribution is a sign the bins are too thin somewhere.

When to Use ANRI vs NRI

The considerations, rather than rules:

ANRI is the safer default whenever the bias it corrects could be material — modest samples (say a few hundred or fewer), several confidence bins, or any comparison across studies that binned confidence differently. Because the bias grows with the bin-to-sample ratio, it is exactly the studies most likely to be compared informally that are least comparable on raw NRI. Formal tests should use ANRI, since testing a biased estimate against zero tests the wrong thing.

NRI remains reasonable when the correction is negligible — large samples with few bins, where ANRI and NRI will agree closely anyway — or when the point is to compare with older published work that reported NRI. In the analysis above the two differ by very little, which is what a sample of this size should produce.

Reporting both, as the table above does, costs nothing and lets a reader see the size of the correction for themselves.

Example Write-Up

Every number below is computed from the objects created above rather than typed in, so this passage cannot drift out of step with the analysis it describes. The same practice suits a manuscript.

We analysed identification performance using calibration analysis, deviation from perfect performance (DPP), and the adjusted normalized resolution index (ANRI) with bootstrap inference, using 1400 lineups of size 6 (700 target-present, 700 target-absent).

Under the stipulated labels, calibration was poor (C = 0.073), mainly because witnesses were underconfident (O/U = -0.229): accuracy in the lowest confidence band was substantially higher than its label implied. The confidence values are ordinal categories given probability labels by stipulation, so C and the direction of O/U depend on the labels and are not findings about witness behaviour. Average DPP was 0.7 (0 = perfect, 1 = chance), and the operating point of maximum utility was confidence of 20 or above (DPP = 0.533). Resolution, measured by ANRI, was 0.041 (95% CI [0.021, 0.072]), against an uncorrected NRI of 0.045.

Expected utility analysis assumed a base rate of 50% and the cost matrix u(correct suspect ID) = 1, u(missed guilty suspect) = -0.5, u(innocent suspect ID) = -2, u(correct rejection) = 0.5; only the differences between these affect the optimum, and they put the cost of acting on an innocent suspect at 2.5 against 1.5 for the gain from acting on a guilty one (a ratio of about 1.67). In this sample the rule “act on confidence >= 30” maximised expected utility (EU = 0.288); this is an in-sample optimum with no interval, and the curve was nearly flat from 40 to 20.

Bayesian analysis showed that high-confidence suspect identifications were the most informative response, with a likelihood ratio of 25 (based on 100 guilty-suspect and 4 innocent-suspect identifications in that band).

Note the last parenthesis. The likelihood ratio there rests on very few target-absent observations, and reporting the counts alongside the ratio is what allows a reader to judge how much weight it will bear.