New function infer_curve() to conduct inference on
the specification curve (the third step outlined by Simonsohn et al.,
2020), based on resampling under the null. See this
vignette for a comprehensive tutorial.
simplify, add_to_formula, and specifications
that were filtered out after setup().lm are refitted with a fast
least-squares routine (which gives the same results as
lm()), and only the relevant parameters are extracted for
other models.infer_curve() checks that the null can be imposed
(models that are linear in the dependent variable, e.g.,
lm, lmer, or gaussian glm, and
numeric variables) and stops otherwise.furrr) and is
reproducible with set.seed(), regardless of the number of
workers.summary() of the resulting specr.infer
object reports all three test statistics proposed by Simonsohn et
al. (2020): the median estimate, the number of significant
specifications, and the aggregated z-value (Stouffer’s Z). Tests are
two-sided by default (direction = "greater" or
"less" for predicted directions), p-values cannot be zero,
and the significance level can be set via alpha. Results
can be summarized by groups of specifications (group).
plot() shows the observed curve against the median and the
2.5% and 97.5% quantiles of the curves under the null.
The argument model of setup() now also
accepts a named list of functions,
e.g. model = list(ols = lm, weighted = my_function). The
names are used as labels for the models, and the functions are used
directly (instead of being looked up by name, which failed e.g. for
functions stored in lists; GitHub issue #29).
plot() gains the argument
choice_labels: with choice_labels = "left",
the names of the analytical choices are shown on the left, next to their
options (GitHub issue #39).
setup() warns if an independent variable is not
numeric, because no estimates can be extracted for factors or character
variables (GitHub issue #47).
New section “Other types of models” in
vignette("different-specifications") with examples for
weighted regressions (GitHub issue #27), survival models with
survival::coxph() (#44), fixed effects models with
fixest::feols() (#41, #43), and multilevel models with
p-values from lmerTest (#48).
Revised documentation: all help pages now describe their return
values, example_data is documented in detail, and outdated
or wrong statements in the vignettes were corrected (e.g., the mean
index in vignette("different-specifications") only
contained x1, and the custom structural equation model
function did not remove the “+ 1” placeholder as described). Vignettes
only evaluate their code if the suggested packages they need are
installed.
boot_null(), which was only part of the development
version 1.0.1 on GitHub, has been removed; use
infer_curve() instead. Results differ:
boot_null() stacked the data of all specifications and
refitted each specification on all stacked rows that contained its
variables. Each specification was therefore refitted on too many rows
(and partly on rows in which the null was imposed for another
specification), which made the curves under the null too narrow. In
simulations without any true effect, boot_null() rejected
the null for the median in about 30% of the cases (at alpha = .05);
infer_curve() rejects it in about 5%. P-values obtained
with boot_null() were thus too small and should be
recomputed.
specr now requires R >= 4.1.0 (as do its dependencies).
specr() stored the model names instead of the
control variables in the controls element of the resulting
specr.object.
plot(..., type = "variance") failed when no control
variables were specified, or when the specifications were filtered down
to a single option of a choice (“grouping factors must have > 1
sampled level”). The variance decomposition now only includes analytical
choices that have more than one option in the results.
In specr(), rows with missing values in a subset
variable are now excluded from the subset analyses (previously, they
were passed to the model as rows of missing values, which most models
dropped anyway).
setup() matched control variables against
x and y as text patterns, so that e.g. a
control x10 was silently dropped from all specifications
with x = "x1". Control variables are now matched exactly.
Specifications in which x and y are the same
variable are now excluded.
setup() built the labels of subset combinations by
removing the text “NA” from them, which garbled values containing “NA”
(e.g., “DNA” became “D”) and turned a value “NA” into “all”. Missing
values among the subset values (e.g., from unique()) are
now ignored, and subsets is checked for unnamed, unknown,
reserved, or duplicated entries, with informative errors instead of
failures in specr().
setup() gives an informative error if
model is neither a character vector nor a named list of
functions.
The model functions in the specifications kept a reference to the
internal environment of setup(), so that saved results
contained an extra copy of the data. This no longer happens, which makes
results objects much smaller.
With fun2 = NULL, the sample size
(fit_nobs) is now obtained with nobs() instead
of broom::glance(), so that it also works for models
without a glance() method.
specr() reported durations of more than a minute in
minutes but labelled them as seconds. It now warns if specifications
return no estimate for the independent variable (e.g., if x
is a factor; GitHub issue #47) instead of silently dropping them, and if
data is given together with a specification setup (in which
case it is ignored).
Plots failed if the results contained a column named
df (e.g., models fitted with lmerTest; GitHub
issue #48).
plot(..., type = "boxplot"), "variance"
and "samplesizes" no longer require confidence intervals,
and summary() and plot() no longer fail if the
results contain no sample sizes. Invalid values of type now
give an error instead of returning NULL. The argument
group of plot() can now also be given as a
string, as in summary(), and grouped summaries
(summary(..., type = "curve", group = ...)) are no longer
returned as grouped tibbles.
print() of a specification setup now counts the
specifications after filtering, the summary of a setup always prints the
head of the specifications table, and all print() and
summary() methods return their input invisibly.
The variance decomposition
(plot(..., type = "variance") and icc_specs())
counted covariances of random slopes (possible with a custom
formula) as variance components.
The functions of the old framework (deprecated since version
1.0.0) keep working with current versions of ggplot2, dplyr, tidyr, and
purrr (no longer using the deprecated purrr::cross(), which
would have made specr fail to install once it is removed from purrr).
Furthermore:
run_specs() checks the confidence level also without
subsets, and no longer counts duplicated rows of the data several times
in combined subsets.plot_summary() no longer merges different sets of
control variables into one “all covariates” box when
all.comb = TRUE.setup_specs() accepts an empty vector of control
variables.icc_specs() is now formally deprecated (it was already
listed as deprecated in the documentation).The package no longer depends on glue and
parallelly.
CRAN release: 2023-01-20
This new version introduces a completely new analytic framework which breaks with earlier versions of specr. See this vignette for a comprehensive tutorial on how to use this new framework.
A new function called setup() is introduced and
replaces the original setup_specs() to make the
specification of analytical choices more intuitive and comprehensive.
Most importantly, it allows to set up all specifications (now also
including subset analyses, additions to the formula, etc.) a priori,
i.e., before the estimation of all models.
tidyverse functions such as e.g., filter().
This way, one can make sure a priori that only reasonable specifications
are actually included.specr.setup, which can be
investigated using e.g., summary() or
plot().The function run_specs() is replaced by
specr(), which now only wraps around setup()
to estimate all models. Most changes are related to increasing speed of
the computations.
furrr) to reduce fitting time (finally solving
github issue #1).specr.object, which can be
investigated using generic function such as summary() and
plot().For more information about these major changes and how to use the new version of specr, see this vignette.
Please note that the functions from earlier versions are still available, but deprecated. It is hence still possible to use the older framework as implemented in version 0.2.1, but we suggest to move to the new framework of version 1.0.0 due to its increased speed and flexibility.
This version of specr was something I wanted to tackle
for a while. Although the previous versions seem to have been embraced
by the community and several relevant papers used them in interesting
ways, certain issues kept coming up (ranging from slow fitting process
to wanting to incorporate more complex analytical decisions and
choices). This version is thus heavily inspired by the growing community
of scholars who provide feedback and suggestions via github. Thank you to
all of you! But for this version, I want to thank two people
particularly:
Big thanks go to Matti Vuorre who explored ways to parallelize specification curve analysis in this blogpost, which became the basis for the implementation in specr.
Many thanks to Kasper Welbers who contributed some essential code within the core function.
Thanks also to Johannes Gruber who provided support in the final stages of building this package.
Development version released: 2020-12-04
Some minor updates (related to github issues) and bug fixes:
all.comb = TRUE; solved github issue #21)run_specs() now allows to add sets of control variables
(github issue #11)broom::tidy() and
broom::glance()The package further now allows to integrate:
CRAN release: 2020-03-26
First stable version
Tested in several environments.
Primary function is run_specs(), which allows to
specify analytic choices and estimate all models across
specifications.
No parallelization of the fitting process (can take very long if model fitting process is complex).
Implementation of random effect modelling and structural equation modeling potentially possible, but still unclear.
Does not allow to specify all possible combinations of control variables (github issue #21).
Does not allow to specify sets of control variables (github issue #11)