Browse
 
Tools
Rss Categories

Running Tests

Reference Number: AA-01122 Views: 12871 0 Rating/ Voters

Much of what you will be doing with the Tuner revolves around running tests. Once you have data loaded, you can see statistics and graphs, especially if you have already transcribed the audio.

While this data is useful for understanding how your application is currently performing, the Tester also allows you to make changes to the grammar or Engine settings and run the audio again, generating new statistics. This is very useful when you wish to see what the effect of a modified grammar will be on your accuracy levels.

To run a new test, simply make a change to an active grammar (using the Grammar Editor) or edit the Engine settings (go to the Settings menu by choosing Edit and then Settings from the top menu).

When you've made your changes, from the Tester window click on the Recycle Bin icon to clear the old results, then click the Green Start icon to start a new test. You will see the Interactions List be populated with new data as the test progresses, and the Statistics pane will also get updated.

        

Once you have run a test, you can save the new results by using File and then Save or Save As to save a new Tuner database.

SRE Settings

The SRE Settings area displays information about the current Speech Engine settings, as configured in the Tuner's Settings menu (accessed by going to Edit > Settings in the menu). These are the settings that will be used if you run a test.

ASR Settings

Save Speech Engine Results
This value controls how the Tuner will save .callsre files it generates. The options are the same options that the Speech Engine supports: None, Basic, Advanced, and All.

Max N-Best
This is the maximum number of N-best results that the Engine will return.

Noise Reduction
What sort of noise reduction the Engine should use. Options are None, Standard, Alternative, and Adaptive.

Test Looping
If test looping is enabled, this will display the number of times the test will loop.

SRE IP Address
The IP address of the Speech Engine the Tuner will send decodes to.

Audio
The format of audio to be used. The default is auto selection. Other options are 16-bit PCM 16 kHz, 8-bit u-law, 8-bit A-law, 16-bit PCM 8 kHz, and 16-bit PCM 44.1 kHz.

Speed/Accuracy
The speed-to-accuracy ratio, expressed as a number between 1 to 100. Lower numbers indicate faster speeds while higher numbers indicate higher accuracy.

Decode Timeout
The timeout for each decode expressed in milliseconds. If the Engine does not return a result by this timeout, the Tuner will return and move to the next interaction.

Decode Mode
How the Tuner will load audio into the Engine. The default setting is Load Files, which means it will simply load the audio into the Engine from the file. Stream Files means the Tuner will stream the audio into the Engine. This means the Engine will use voice activity detection, as it would in most calls. Stream Files - Add Silence is like Stream Files, but a little bit of silence will be added to the start and end of the file.

Grammar
The default grammar the Engine will use.

Interactions List

The Interactions tab displays results for each individual reaction. The following information is shown:

Interactions List

#
The index number for the interaction.

Status
Information about the status of the interaction. Possible status types are:

  • Load Failure: the audio failed to load for some reason.
  • Decode Failure: the decode failed.
  • No Transcript: no transcript is available for this interaction.
  • No Decode: the interaction was successfully loaded but no decode has been performed by the Speech Engine.
  • No Transcript/Decode: no transcript is available for this interaction and there has not yet been a decode.
  • Correct: the decoded text matches the transcript.
  • SI Match: the decoded text differs from the transcript, but both have the same semantic interpretation.
  • Incorrect: the Speech Engine returned a result that does not match the transcript (and the decoded text's semantic interpretation does not match the semantic interpretation of the transcript).
  • OOV: the transcript's text is out of vocabulary, meaning the utterance is not a valid in-grammar sentence.
  • Unprocessed: the Tuner has not yet processed this interaction.
  • Inactive: This interaction has been filtered out.
    

Date/Time
The date/time of the interaction.

Transcript Text
If available, the transcript for the interaction.  

Decoded Text
If available, what the Speech Engine recognized as the raw text for the interaction.  

Transcript SI
The semantic interpretation for the transcript. This is calculated by parsing the transcript against the grammar for the interaction and using the semantic interpretation.  

Decode SI
The semantic interpretation for the decoded text.  

Conf
The confidence score for the interaction.  

Time
The amount of time the interaction took to process.  

Error Type
If the Speech Engine made an error in recognizing the interaction, this column contains the error.