Program gtconvert.py - User Manual

Information about the program:

The program gtconvert.py converts the legacy file formats (fileCxxxf, fileCxxxr, fileDxxxf and fileGxxxr, for xxx in 010, 015-020, 115) into the new vertical file formats that will be used for submitting EBVs to the IDEA DB in the near future. The program will find all the file{A}xxx{b} files in a specified DATADIR and convert them all, creating four files (file300Cf, file300Df, file300Cr and file300Gr) with separate bull proof records for all traits found in all the xxx files matching the specified breed of evaluation (BRD) and population/country code (POP). The program also converts the legacy parameter file into a trait info file specifically designed for the gebvtest program and creates a file of birth dates extracted from fileC010f.
All of the input files may contain data for more than one breed or population. The input files may have a SUFFIX, like ".usa" for example, but in this case all the files must have the same suffix.

For information about installation and sample data for testing the program, see public/GEBVtest_software.

Input files:

fileCxxxf - national official genetic evaluations sent by the NGEC as input for the most recent Interbull MACE evaluation (formats: 010, 115, 015, 016, 017, 018, 019, 020)
fileDxxxf - daughter deviation file, including either DD or D_PGM for the same animals included in fileCxxxf (same formats as for fileCxxxf)
fileCxxxr - reduced conventional genetic evaluation file, obtained from conventional genetic evaluations using truncated data (same formats as for fileCxxxf)
fileGxxxr - reduced genomic evaluation file, obtained from genomic evaluations truncated data (same formats as for fileCxxxf)
parameter file - parameters used in most recent Interbull MACE evaluation - one file may contain all trait groups (Format: parameter file)

Running the Program:

The program should be run from within the programs directory. Typing

python gtconvert.py --help

will give a summary of the program usage:

usage: gtconvert.py [-h] [-v] [-s SUFFIX] [-p PARFILE] [-d {DD,GM}] [-y YEAR]
                    [-x {Y,N}] [-o OUTDIR]
                    brd pop datadir

positional arguments:
  brd                   evaluation breed code (BSW/GUE/JER/HOL/RDC/SIM)
  pop                   population code (same as country code except for
                        CHR/DEA/DFS/FRR/FRM)
  datadir               absolute or relative path to data files

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         increase output verbosity
  -s SUFFIX, --suffix SUFFIX
                        suffix to add to all input file names, eg. ".usa" if
                        file names are like fileC010f.usa (default=none)
  -p PARFILE, --parfile PARFILE
                        path+name of input "parameter" file
                        (default=DATADIR/parameterSUFFIX)
  -d {DD,GM}, --depvar {DD,GM}
                        type of daughter performance on Df file (default=GM)
  -y YEAR, --year YEAR  minimum birth year for test bulls (default is year of
                        EVALDATE on parameter file less 8 years)
  -x {Y,N}, --type2x {Y,N}
                        inclusion of type 21+22 bulls in test group
                        (default=N)
  -o OUTDIR, --outdir OUTDIR
                        directory for output files (default=DATADIR)

Note that the input parameter file may be in a different directory than the other files or have a different name or suffix, in which case the -p option must be specified.
The program adds defaults for several options to the trait info file it creates. This file may need to be edited manually or programmatically if different options are needed for some traits compared to other traits.
You may also choose to put the output files from this program into a different directory than the input files. In this case, the specified OUTDIR from this program should be used as the DATADIR for the next program.

Warning:

If the gtconvert.py program crashes with a UnicodeEncodeError, it means there are likely binary character codes, most likely in bull names, which do not fit the standard utf-8 encoding scheme. You can try specifying the option '-e iso-8859-1' or some other encoding listed at http://docs.python.org/2/library/codecs.html#standard-encodings. If that fails, you could try to set the name field to blank in all input files, since the bull name field on the 010 files are no longer used at the Interbull Centre. Also, make sure there is no binary data in any other field, due to uninitialized variables in some Fortran program for example.

Output files:

traits - GEBV test options file(Format: traits)
file300Cf - national official genetic evaluations written in trait-independent format (Format: File300)
file300Df - daughter deviation file written in trait-independent format
file300Cr - reduced conventional genetic evaluation file written in trait-independent format
file300Gr - reduced genomic evaluation file written in trait-independent format
file736 - file with birth date and genotyping flag (Format: File736)
gtconvert.log - summary of the inputs and outputs (Example)

If you need assistance, please do not hesitate to contact us at interbull@slu.se .