2012-12-13
0.4

&kmyapplication; is an acoustic model manager.

KDE
kdeutils
Kapp
Simon
recognition
speech
voice
command
control
model
compilation
Sam
accessibility
&FDLNotice;
2009-2012
Peter Grasch
Peter
H.
Grasch
peter.grasch@bedahr.org

# The &kmyapplication; Handbook

## Introduction

&kmyapplication; is a tool to create and test acoustic models. It can compile new speech models, use models created by Simon and produce models that can be used by Simon later on.

It is targeted towards people wanting more control over their acoustic model and provides much lower level access to the build process. &kmyapplication; is mainly geared towards speech professionals wanting to improve and / or test their acoustic model.

For more information on the architecture of the Simon suite please see the Simon manual.

### Background

This section will provide a bit of background information on the compilation and testing process.

#### Effective testing

One of the major features of Sam is to test the generated acoustic models.

The basic testing procedure is to run recognition on samples where the transcription is already known and comparing the results. &kmyapplication; also takes the confidence score of the recognition into account to measure how robust the created system is.

Due to the way acoustic models are created, both the recognition accuracy and confidence will be highly skewed when the same samples are used both for training and testing. This is called "in corpus" testing (the samples used for testing are also in your training corpus).

While in corpus testing might tell you if the compilation process failed or produced sub par results it won't tell you the "real" recognition rate of the created model. Therefore it is recommended to do "out of corpus" testing: Use different samples for training and for testing.

For out of corpus testing simply split your prompts file in two parts: One used to compile the model and one to test the model. Of course the test set doesn't need to be very big to get a representative result.

If you don't have a lot of training data, you can also split the complete corpus in ten parts. Compile 10 models, each leaving out one part of the corpus. Then do then individual tests (always with the test set that was excluded during compilation) and average the results.

## Using &kmyapplication;

&kmyapplication; provides a graphical interface that is divided into five areas.

Main window Main window

The order of the tabs represents a complete Sam workflow from creating a model to testing it.

### Input & output files

Define what files to work on. You can save and load this configuration by using the Save and Load buttons respectively.

The Save and Load menus only save the paths and options defined here. They don't save the associated files!

Here you also have the option to select between supported model compilation / test backends. Depending on your Simon installation, not all might be available.

Input & output selection Input & output selection

If you want to compile and / or test the Simon model, you can use the Modify Simon's model option to load the appropriate files. Both scenarios and prompts will be serialized as appropriate.

When Simon compiles the model, it will automatically remove untrained words during serialization as well as adapting the prompts appropriately. This will also happen when selecting Modify Simon's model. If you do, however serialize them separately using Serialize scenarios and Serialize prompts this adaption will not be performed and you alone are responsible for the validity of the input files. If you provide file names directly this of course applies even more so.

When selecting a static model as the used model type &kmyapplication; will still compile the language model, just like Simon would. It also copies the given base model input files to the output paths.

&kmyapplication; allows the user to provide different types of prompts files for the creation and the testing of the model. For more information about how to use this feature effectively, please refer to the section on effective testing.

Test configuration Test configuration

Users can set up an arbitrary number of test configurations to compare different model parameters. All configured tests will be run and all are listed in any reports that might be exported. Through the option of supplying different input files than the compiled model, reference models (baselines) can be included in the report.

Since Simon 0.4, the primary file format for models is the newly introduced SBM format. While Sam will also create SBM models, the testing still requires low level input files. This may seem counter intuitive but presents certain advantages in practice: The Simon model compilation (and thus also the SAM model compilation) still builds the low level output files for both SPHINX and HTK model compilers before packaging them into SBMs so pointing SAM to these (temporary) files is not difficult and in fact the default when using Modify Simon's model. Because acoustic model compilation is typically an iterative process, the compiler typically produces multiple temporary versions of different stages of parameter estimation. By allowing low level input, testing and comparing these intermediate results is much easier.

If you want to test a model you received as a SBM container, you can extract it with SAM through the Extract Simon Model option to gain access to the low level input files required for testing with &kmyapplication;.

### Conservative Training

If you have configured Simond to keep recognition samples, Sam can import this data through the Import recognition samples option in the "Input & output" section.

Conservative training Conservative training

The label of the prompts file will be built automatically from the most likely recognition result. You can limit the selection to only include samples that achieved a given confidence score.

If the Simon setup you're importing the samples from uses a global filter to deactivate recognition, then you can provide Sam with the command that activates and deactivates this filter (pausing and resuming the recognition, respectively) to ignore any samples produced while the recognition was deactivated.

### Adapt scenarios

Simon stores the language model in scenarios. When you want to use them to compile the model, you first need to serialize them to files readable by SPHINX or Julius and the HTK.

Adapt scenarios Adapt scenarios

To do this, select Serialize scenarios button in the Input & output tab. Simon's scenarios will also be serialized when using the Modify Simon's model options.

In this page you can find status information of the adaption and read detailed error messages if an error occurs.

### Create model

Here you can build the model using the input and output files defined in the input & output section.

The compilation process is identical to the one used by Simond. Unlike Simond, however, the full log of all called external programs, their output as well as progress information is shown regardless of whether the compilation was successful or not.

Create model Create model

Using this verbose output it is much easier to find issues with the input data.

### Test model

Here you can test the speech model.

&kmyapplication; will test the model set by in the Output files section in the input & output section.

During testing, the recognizer of the selected backend will run to recognize on the input files as set by the test prompts. Resulting word- and sentence errors will be counted as well as the overall robustness by analyzing confidence scores of the recognizer.

The main test screen will show all started tests and report when they complete. The tests themselves will run threaded (at the same time). The amount of parallelism best suited for your machine is determined automatically.

Test model Test model

As soon as all tests are finished, the test results will be displayed automatically.

### Test results

After a successful model test you can find a detailed report of the recognition accuracy here.

#### Scoring

To best reflect the recognition performance, &kmyapplication; uses multiple, ranked results for the tests.

A correctly recognized word or sentence will be scored with the confidence score the word achieved. If a word is recognized correctly but another, wrong, result is ranked higher, the accuracy of that recognition will be 0%.

The displayed overall recognition rate is the average of all confidence rates. The shown overall accuracy rating shows the average over all accuracy scores which represents the likelihood that given any word in any sentence, this word has been recognized correctly.

#### Word

&kmyapplication; will list recognition accuracy for each word individually.

Word results Word results

If you have samples containing more than one word they will be segmented during recognition. Each word will be scored individually (although the different words of course still influence each other).

#### Sentences

This section lists each prompt as "sentence".

Prompts that were recorded more than once are combined.

Sentence results Sentence results

#### Files

In the files section you can see the recognition results for each file. Each file will list the 10 most likely results in the details pane when you select it.

File results File results

When you identify problematic samples, you can re-record (or remove) them by selecting them and clicking on Edit sample.

You can sort the files by each column simply by clicking on the column header. This way it is very easy to find bad samples by sorting by recognition rate.

#### Log

Here you can review the full protocol of this subtest.

Test log Test log

### Reports

&kmyapplication; can export reports about test results.

The export of a report can be started by selecting Export test result in the &kmyapplication; main window.

#### Templates

Reports are created by filling in templates with information. Principally, these templates are either plain text files or LaTeX files.

Three templates of LaTeX templates, two German and one English one, are shipped with &kmyapplication; but adding new ones is easy. Just install new templates to one of the following locations depending on your operating system:

- Linux: `~/.kde/share/apps/sam/reports/templates`

- Windows: `%appdata%\.kde\share\apps\sam\reports\templates`

The existing templates can be used as references. They are to be found at the following path:

- Linux: `/usr/share/apps/sam/reports/templates`

- Windows: `<installation path>\share\apps\sam\reports\templates`

#### Providing meta-information

The reporting engine allows the input of meta information about the used input data as well as the test conditions, speakers, etc.

Report: General information Report: General information

This information can be entered directly through the graphical user interface. Depending on the used template, not all provided information might be used by the exported report.

Report: Experiment information Report: Experiment information

Information entered in the dialog will also be saved with your &kmyapplication; project file.

Report: Test set Report: Test setinformation

Some meta information, like the amount of samples in your test- and training corpus and the system information can be either filled in by &kmyapplication; automatically or provided manually.

#### Output

Exported reports are stored as ZIP archives. In the case of LaTeX templates they may contain graphics in addition to the raw `.tex` file.

Below you can see the beginning of an exported full LaTeX report of a simple test of the Amarok scenario with the static Voxforge base model.

Exported report Exported report

### Command line arguments

&kmyapplication; provides command line arguments for the most common actions to automatize model creation and testing.

All actions provided as command line arguments will be taken immediately after &kmyapplication; has loaded. Options that build on one another (like compiling and testing) will be re-ordered automatically.

#### Command line arguments

Argument | Description
--- | ---
`-m <type>` | Selects the used backend; Provide either "sphinx" or "htk" to select the specified backend. Depending on your installation, "sphinx" might not be available.
`-s` | Retrieves all paths from Simon; (equivalent to Modify Simon's model)
`-c` | Starts the model compilation;
`-t` | Starts the model test;
`-e <filename>` | Exports a report to the given filename;
`-b` | Enables batch processing; &kmyapplication; will quit automatically after all arguments have been processed. The main window will not be shown.
`-l <filename>` | Export the complete build log to the given file;
`-w` | Automatically safe any opened &kmyapplication; project file when exiting the application;
`<filename>` | Load this &kmyapplication; project file during startup.

## Questions and Answers

In an effort to keep this section always up-to-date it is available at our online wiki.

## Credits and License

&kmyapplication;

Program copyright 2008-2010 Peter Grasch peter.grasch@bedahr.org

Documentation Copyright © 2009-2010 Peter Grasch peter.grasch@bedahr.org

&underFDL;

&underGPL;

## Installation

Please see our wiki for installation instructions.

&documentation.index;