Go to the first, previous, next, last section, table of contents.


24 Building models from databases

Because our research interests tend towards creating statistical models trained from real speech data, Festival offers various support for extracting information from speech databases, in a way suitable for building models.

Models for accent prediction, F0 generation, duration, vowel reduction, homograph disambiguation, phrase break assignment and unit selection have been built using Festival to extract and process various databases.

24.1 Labelling databases

In order for Festival to use a database it is most useful to build utterance structures for each utterance in the database. As discussed earlier, utterance structures contain streams of ordered items, with relations between these items. Given such a structure we can easily read in the utterance representation and access it, dumping information in a normalised way allowing for easy building and testing of models.

Of course the level of labelling that exists, or that you are willing to do by hand or using some automatic tool, for a particular database will vary. For many purposes you will at least need phonetic labelling. Hand labelled data is still better than auto-labelled data, but that could change. The size and consistency of the data is important too.

For this discussion we will assume labels for: segments, syllables, words, phrases, intonation events, pitch targets. Some of these can be derived, some need to be labelled. This would not fail with less labelling but of course you wouldn't be able to extract as much information from the result.

In our databases these labels are in Entropic's Xlabel format, though it is fairly easy to convert any reasonable format.

Segment
These give phoneme labels for files. Note the these labels must be members of the phoneset that you will be using for this database. Often phone label files may contain extra labels (e.g. beginning and end silence) which are not really part of the phoneset. You should remove (or re-label) these phones accordingly.
Word
Again these will need to be provided. The end of the word should come at the last phone in the word (or just after). Pauses/silences should not be part of the word.
Syllable
There is a chance these can be automatically generated from Word and Segment files given a lexicon. Ideally these should include lexical stress.
IntEvent
These should ideally mark accent/boundary tone type for each syllable, but this almost definitely requires hand-labelling. Also given that hand-labelling of accent type is harder and not as accurate, it is arguable that anything other than accented vs. non-accented can be used reliably.
Phrase
This could just mark the last non-silence phone in each utterance, or before any silence phones in the whole utterance.
Target
This can be automatically derived from an F0 file and the Segment files. A marking of the mean F0 in each voiced phone seem to give adequate results.

Once these files are created an utterance file can be automatically created from the above data. Note it is pretty easy to get the streams right but getting the relations between the streams is much harder. Firstly labelling is rarely accurate and small windows of error must be allowed to ensure things line up properly. The second problem is that some label files identify point type information (IntEvent and Target) while others identify segments (e.g. Segment, Words etc.). Relations have to know this in order to get it right. For example is not right for all syllables between two IntEvents to be linked to the IntEvent, only to the Syllable the IntEvent is within.

The script `festival/examples/make_utts' is an example shell script which automatically builds the utterance files from the above labelled files. It mostly gets it right, but is not particularly efficient and can take over an hour to run over a large database. Also you will probably have to edit it to make it suitable for other databases. At least you will need to identify the phone used for silence to the shell script.

This script will generate utterance files for each example file in the database which can be loaded into Festival and used either to do "natural synthesis", or used to extract data for training or test data for building models.

24.2 Extracting features

The easiest way to extract features from a labelled database of the form described in the previous section is by loading in each of the utterance structures and dumping the desired features.

Using the same mechanism to extract the features as will eventually be used by models built from the features has the important advantage of avoiding spurious errors easily introduced when collecting data. For example a feature such as n.accent in a Festival utterance will be defined as 0 when there is no next accent. Extracting all the accents and using an external program to calculate the next accent may make a different decision so that when the generated model is used a different value for this feature will be produced. Such mismatches in training models and actual use are unfortunately common, so using the same mechanism to extract data for training, and for actual use is worthwhile.

The Festival function utt.features takes an utterance, a stream name and a list of desired features as an argument. This function can be used to dump desired features for each item in a desired stream in each utterance.

The file `festival/examples/dumpfeats.scm' gives example Scheme code to dump features. To use this, copy it and edit it for your database. The values you should change are

db_dir
The path name of the directory containing the database base, this should be slash terminated. This should be the directory which contains `utts/seg/'.
database_files
A list of file (ids) in the database such that for each ID in the list
(string-append db_dir "utts/seg/" ID ".seg.utt")
identifies a file.
desired_stream
The name of the stream you wish to extract features from, e.g. Word, Syllable, Segment, etc.
desired_feats
A list of features use wish to extract. section 32 Function list
outskeleton
This describes how to construct the output file names for the features. The features are saved in separated files one for each utterance (based on the file ids). The feature vectors are saved one per line for each item in the desired stream. An example outskeleton is "feats/%s.dur" which would save the feature files in a directory called `feats', one for each file id with the extension `.dur'.

After editing the file you can load it into Festival or run it as a batch file with the command.

festival -b dumpfeats.scm

Note the call that does the work is at the end of `dumpfeats.scm' file. The output will be done with respect to the current directory, and you must create the directory where the output files are to go (in this example `feats/' which should be created before running the extraction).

24.3 Building models

This section describes how to build models from data extracted from databases as described in the previous section. It uses the CART building program, `wagon' which is available in the speech tools distribution. But the data is suitable for many other types of model building techniques, such as linear regression or neural networks.

Wagon is described in the speech tools manual, though we will cover simple use here. To use Wagon you need a datafile and a data description file.

A datafile consists of a number of vectors one per line each containing the same number of fields. This, not coincidentally, is exactly the format produced by the `dumpfeats.scm' described in the previous section. The data description file describes the fields in the datafile and their range. Fields may be of any of the following types: class (a list of symbols), floats, or ignored. Wagon will build a classification tree if the first field (the predictee) is of type class, or a regression tree if the first field is a float. An example data description file would be

(
( duration float )
( name # @ @@ a aa ai au b ch d dh e e@ ei f g h i i@ ii jh k l m n 
    ng o oi oo ou p r s sh t th u u@ uh uu v w y z zh )
( n.name # @ @@ a aa ai au b ch d dh e e@ ei f g h i i@ ii jh k l m n 
    ng o oi oo ou p r s sh t th u u@ uh uu v w y z zh )
( p.name # @ @@ a aa ai au b ch d dh e e@ ei f g h i i@ ii jh k l m n 
    ng o oi oo ou p r s sh t th u u@ uh uu v w y z zh )
( Syllable.position_type 0 final initial mid single )
( pos_in_syl float )
( syl_initial 0 1 )
( syl_final 0 1)
( Syllable.p.syl_break 0 1 3 )
( Syllable.syl_break 0 1 3 4 )
( Syllable.n.syl_break 0 1 3 4 )
( Syllable.p.stress 0 1 )
( Syllable.stress 0 1 )
( Syllable.n.stress 0 1 )
)

The script `speech_tools/bin/make_wagon_desc' goes some way to helping. Given a datafile and a file containing the field names, it will construct an approximation of the description file. This file should still be edited as all fields are treated as of type class by `make_wagon_desc' and you may want to change them some of them to float.

The data file must be a single file, although we created a number of feature files by the process described in the previous section. From a list of file ids select, say, 80% of them, as training data and cat them into a single datafile. The remaining 20% may be catted together as test data.

To build a tree use a command like

wagon -desc DESCFILE -data TRAINFILE -test TESTFILE

The minimum cluster size (default 50) may be reduced using the command line option -stop plus a number.

Varying the features and stop size may improve the results.

Building the models and getting good figures is only one part of the process. You must integrate this model into Festival if its going to be of any use. In the case of CART trees generated by Wagon, Festival supports these directly. In the case of CART trees predicting zscores, or factors to modify duration averages, ees can be used as is.

Note there are other options to Wagon which may help build better CART models. Consult the chapter in the speech tool manual on Wagon for more information.

Other parts of the distributed system use CART trees, and linear regression models that were training using the processes described in this chapter. Some other parts of the distributed system use CART trees which were written by hand and may be improved by properly applying these processes.


Go to the first, previous, next, last section, table of contents.