# Checking and Improving Results of package Synth

## Introduction

This vignette illustrates the usage of `improveSynth`. For a more general introduction to package `MSCMT` see its main vignette.

Estimating an SCM model involves searching for an approximate solution of a nested optimization problem. Although the formulation of the optimization problem is quite simple, finding a (good approximate) solution can be hard for several reasons, see Becker and Klößner (2017) and Becker and Klößner (2018). While implementing package `MSCMT` we put a lot of effort into the design of a smart and robust (but still fast) optimization procedure.

Apart from function `mscmt` for the estimation of SCM models based on our model syntax, we also included the convenience function `improveSynth`, which implements checks for feasibility and optimality of results delivered by package `Synth`. Below, we illustrate how to use `improveSynth`.

## First Example

We exemplify the usage of `improveSynth` based on the first example of function `synth` in package `Synth`.

### Generating the result of package `Synth`

The following code is thus essentially borrowed from the `example` section of the corresponding help page (all comments have been removed):

    load("synth.out.RData")
    # using cached copy because package Synth has been archived
    # 'synth.out' and 'dataprep.out' have been generated with the following code:
    #
    #  library(Synth) {
    #  data(synth.data)
    #  dataprep.out <-
    #    dataprep(
    #      foo = synth.data,
    #      predictors = c("X1", "X2", "X3"),
    #      predictors.op = "mean",
    #      dependent = "Y",
    #      unit.variable = "unit.num",
    #      time.variable = "year",
    #      special.predictors = list(
    #        list("Y", 1991, "mean"),
    #        list("Y", 1985, "mean"),
    #        list("Y", 1980, "mean")
    #      ),
    #      treatment.identifier = 7,
    #      controls.identifier = c(29, 2, 13, 17, 32, 38),
    #      time.predictors.prior = c(1984:1989),
    #      time.optimize.ssr = c(1984:1990),
    #      unit.names.variable = "name",
    #      time.plot = 1984:1996
    #    )
    #
    #  synth.out <- synth(dataprep.out)

### Checking the result

We check the result by applying function `improveSynth` to `synth.out` and `dataprep.out`:

    library(MSCMT)
    synth2.out <- improveSynth(synth.out,dataprep.out)

    ## Results reported by package Synth
    ## =================================
    ##
    ## Optimal V    : 0.00397203172199428 0.0022275802340665 0.128885551930044
    ##                0.327768944797584 0.00814476840194525 0.529001122914366
    ## Optimal W*(V): 0.00100619117797987 0.00524663354000337 0.169165120873083
    ##                0.213620460951039 0.610958634379188 2.95858187768921e-06
    ## with corresponding predictor loss ('loss W') of 0.004733129
    ## and corresponding dependent loss ('loss V') of 4.720551.
    ##
    ##
    ## Components of W*(V) do not sum to 1, dependent loss ('loss V') of
    ## rescaled W*(V) is 4.720551.
    ##
    ##
    ## Feasibility of W*(V)
    ## ====================
    ##
    ## WARNING: W*(V) is NOT optimal and thus infeasible!
    ## 'True' W*(V): 0 0.00338943037392679 0.158483318047272 0.217351422652097
    ##               0.620775828926704 0
    ## with corresponding predictor loss ('loss W') of 0.004715856
    ## and corresponding dependent loss ('loss V') of 4.719351.
    ##
    ##
    ## Optimality of V
    ## ===============
    ##
    ## WARNING: 'Optimal' V (as reported by package Synth) is not optimal (W*(V) was
    ##          infeasible), (one of potentially many) 'true' optimal V* (with
    ##          sum(V*)=1):
    ## Optimal V*    : 5.60003568150536e-09 0.560003568150536 1.77473253948985e-08
    ##                 5.60003568150536e-09 5.60285964068872e-08 0.439996346873471
    ## Optimal W*(V*): 0 0 0.045688074469268 0.250556360555955 0.627179685612033
    ##                 0.0765758793627435
    ## with corresponding predictor loss ('loss W') of 7.597302e-09
    ## and corresponding dependent loss ('loss V') of 4.440946.

Package `Synth` generated a (slightly) infeasible solution, returning a (slightly) suboptimal weight vector `w` for the control units. However, the predictor weights `v` are (considerably) suboptimal anyway, because the original dependent loss of 4.720551 (as well as the dependent loss for the corrected `w` 4.719351) is considerably larger than the dependent loss 4.440946 for the optimal predictor weights obtained by `improveSynth`.

## Second Example

In the second example, we modify the first example by allowing package `Synth` to use `genoud` as (outer) optimization algorithm.

### Generating the result of package `Synth`

`genoud` is switched on by the corresponding function argument. We capture the output with `capture.output` because it is very verbose. Furthermore, the calculation is quite lengthy, therefore the results have been cached.1

    load ("synth3.out.RData")
    # using cached copy because package Synth has been archived
    # 'synth3.out' has been generated with the following code:
    #
    #  set.seed(42)
    #  out <- capture.output(synth3.out <- Synth::synth(dataprep.out,genoud=TRUE))

### Checking the result

We again check the result by applying function `improveSynth` to `synth3.out` and `dataprep.out`:

    synth4.out <- improveSynth(synth3.out,dataprep.out)

    ## Results reported by package Synth
    ## =================================
    ##
    ## Optimal V    : 2.0267533885719e-10 0.282598702518196 0.0021778890504857
    ##                0.00278189732871801 0.000373699249646763 0.712067811650278
    ## Optimal W*(V): 0.0420138249827668 0.0112849868071455 0.0223432791829297
    ##                0.218134146153805 0.595395073764248 0.110828688687666
    ## with corresponding predictor loss ('loss W') of 0.0001350879
    ## and corresponding dependent loss ('loss V') of 4.328506.
    ##
    ##
    ## Components of W*(V) do not sum to 1, dependent loss ('loss V') of
    ## rescaled W*(V) is 4.328506.
    ##
    ##
    ## Feasibility of W*(V)
    ## ====================
    ##
    ## WARNING: W*(V) is NOT optimal and thus infeasible!
    ## 'True' W*(V): 0 0 0 0.229091487590933 0.715755416007496 0.0551530964015711
    ## with corresponding predictor loss ('loss W') of 4.4355e-05
    ## and corresponding dependent loss ('loss V') of 6.09574.
    ##
    ##
    ## Optimality of V
    ## ===============
    ##
    ## WARNING: 'Optimal' V (as reported by package Synth) is not optimal (W*(V) was
    ##          infeasible), (one of potentially many) 'true' optimal V* (with
    ##          sum(V*)=1):
    ## Optimal V*    : 5.60003568150536e-09 0.560003568150536 1.77473253948985e-08
    ##                 5.60003568150536e-09 5.60285964068872e-08 0.439996346873471
    ## Optimal W*(V*): 0 0 0.045688074469268 0.250556360555955 0.627179685612033
    ##                 0.0765758793627435
    ## with corresponding predictor loss ('loss W') of 7.597302e-09
    ## and corresponding dependent loss ('loss V') of 4.440946.

Now, package `Synth` generated a solution with a dependent loss of 4.328506 which is even smaller than the dependent loss 4.440946 obtained by `improveSynth`. However, the solution generated by `Synth` is severely infeasible: the inner optimization failed, returning a suboptimal weight vector `w` for the control units, which itself lead to a wrong calculation of the dependent loss (which, of course, depends on `w`). Implanting the true optimal `w` (depending on `v`) leads to a large increase of the dependent loss, which uncovers the suboptimality of `v`.

`improveSynth` is able to detect this severe problem and calculates an improved and feasible solution (the improved solution matches the solution obtained from the first call to `improveSynth` above, with a dependent loss of 4.440946).

## Summary

Issues with the inner and outer optimizers used in `synth` from package `Synth` may lead to infeasible or suboptimal solutions. This vignette illustrated the usage of the convenience function `improveSynth` from package `MSCMT` for checking and potentially improving results obtained from `synth`.

## References

Becker, Martin, and Stefan Klößner. 2017. “Estimating the Economic Costs of Organized Crime by Synthetic Control Methods.” Journal of Applied Econometrics 32 (7): 1367–9. http://dx.doi.org/10.1002/jae.2572.

———. 2018. “Fast and Reliable Computation of Generalized Synthetic Controls.” Econometrics and Statistics 5: 1–19. https://doi.org/10.1016/j.ecosta.2017.08.002.

1. To reproduce from scratch, please delete `"synth3.out.RData"` from the `vignettes` folder.↩︎