---
title: "Getting started with ihsMW"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with ihsMW}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

The `ihsMW` package is a dedicated toolkit designed to clean, harmonise, and aggregate household survey data from the Malawi Integrated Household Survey (IHS) series. It is built to support researchers and analysts working with the IHS2 (2004/05), IHS3 (2010/11), IHS4 (2016/17), and IHS5 (2019/20) datasets.

## 1. Installation

You can install the stable release of `ihsMW` from CRAN:

```{r eval=FALSE}
install.packages("ihsMW")
```

Or install the development version from GitHub:

```{r eval=FALSE}
# Install using pak
pak::pak("vituk123/ihsMW")

# Or using remotes
remotes::install_github("vituk123/ihsMW")
```

## 2. The IHS Landscape

The Malawi National Statistical Office (NSO) conducts the Integrated Household Survey (IHS) periodically to track poverty, household expenditure, agriculture, and other socio-economic indicators. The primary rounds include:

- **IHS2**: 2004–2005
- **IHS3**: 2010–2011
- **IHS4**: 2016–2017
- **IHS5**: 2019–2020

Due to licensing restrictions, the raw microdata cannot be redistributed directly within R packages. Researchers must first register and manually download the survey data in Stata (`.dta`) format from the [World Bank Microdata Library](https://microdata.worldbank.org/).

Once downloaded, place the files in a structured folder hierarchy on your local machine.

## 3. Loading and Harmonising

Each round of the IHS uses different variable names for the same question. For example, household size is recorded under different column names depending on the round. `ihsMW` uses a comprehensive crosswalk to harmonise these variable names.

To load and harmonise a raw survey file:

```{r eval=FALSE}
library(ihsMW)
library(haven)

# Load the raw Stata file
raw_data <- read_dta("path/to/IHS5/hh_mod_a_filt.dta")

# Harmonise variables to standard names
harmonised_data <- ihs_harmonise(raw_data, round = "IHS5")
```

## 4. Searching Variables

To find variables mapped in the crosswalk, use `ihs_search()`. You can search by keywords or labels:

```{r eval=FALSE}
# Search for consumption-related variables
ihs_search("consumption")

# Search for age within a specific round
ihs_search("age", round = "IHS5")
```

To view a summary of the crosswalk coverage and flag variables needing review, use `ihs_crosswalk_check()`:

```{r eval=FALSE}
ihs_crosswalk_check()
```

## 5. Data Cleaning

`ihsMW` provides tools to clean standard survey anomalies, handle missing value codes, and winsorize extreme values:

```{r eval=FALSE}
# Convert standard survey missing codes (-99, -98, etc.) to NA
df_clean <- ihs_standardize_missing(harmonised_data)

# Winsorize outliers (e.g. food expenditure) stratified by urban/rural
df_winsor <- ihs_winsorize(df_clean, value_col = "food_exp", strata_col = "urban")

# Run the master cleaning wrapper which applies both steps and logs changes
df_cleaned <- ihs_clean(
  data = harmonised_data,
  missing_cols = c("food_exp", "nonfood_exp"),
  winsorize_cols = "food_exp",
  strata_col = "urban"
)
```

## 6. Unit Conversion

Agricultural modules in the IHS allow households to report harvest quantities in non-standard units (e.g., pails, basins, ox-carts, bags) rather than standard kilograms. `ihsMW` bundles official NSO conversion factors to convert these quantities to standard kilograms:

```{r eval=FALSE}
# Convert quantities reported in non-standard units to kilograms
crop_data <- data.frame(
  crop_code = c(1, 2),
  unit_code = c(3, 4),
  quantity = c(10, 5),
  region = c(1, 2)
)

crop_data_kg <- ihs_convert_units(
  data = crop_data,
  crop_col = "crop_code",
  unit_col = "unit_code",
  qty_col = "quantity",
  region_col = "region"
)
```

## 7. Aggregation

To aggregate member-level or agricultural plot-level data up to the household level, use `ihs_aggregate()`:

```{r eval=FALSE}
# Aggregate individual-level education to household level
hh_edu <- ihs_aggregate(
  data = member_data,
  id_cols = "case_id",
  val_cols = c("years_education", "completed_primary")
)
```