Differences between revisions 13 and 17 (spanning 4 versions)
⇤ ← Revision 13 as of 2013-11-21 13:09:01 →
Size: 6032
Editor: Hjerpe
Comment:
← Revision 17 as of 2013-11-21 13:30:41 → ⇥
Size: 5310
Editor: Hjerpe
Comment:
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
A Python program called CheckProofsPara.py will check the 300/700 proof file format and the associated 301/701 parameter file for format correctness and prepare a zip file, if no errors are found in the files. The zipped file created with the name IB-ORGCODE-IG-yyymmddThhmmss.zip (for conventional MACE) or IB-ORGCODE-GG-yyymmddThhmmss.zip (for GMACE) containing the input proof and parameter files, renamed to proof.dat and param.dat, respectively. The zip file represents your checked data file to upload to the Interbull Centre IDEA database. The correctness of the files are checked against the IDEA EBV User Manual Appendix II. For technical reason the program rejects files containing more than a million of data.  . A Python program called CheckProofsPara.py will check the 300/700 proof file format and the associated 301/701 parameter file for format correctness against the IDEA EBV User Manual Appendix II and prepare a zip file, if no errors are found in the files. The zipped file created with the name IB-ORGCODE-IG-yyymmddThhmmss.zip (for conventional MACE) or IB-ORGCODE-GG-yyymmddThhmmss.zip (for GMACE) contains the input proof and parameter files, renamed to proof.dat and param.dat, respectively. The zip file represents your checked data file to upload to the Interbull Centre IDEA database. For technical reason the program rejects files containing more than a million of data.
Line 6: Line 6:
Each record in the proof file must have associated information for a single brd-pop-trt combination in the parameterfile. One file may contain records for as many or few brd-pop-trt combinations as desired. All traits in a trait groupcan be put in the same file, as they have in the past, but this is not essential.

They may also choose to upload evaluations for all breeds, populations and traits in a single file if they find that more convenient. Every proof file must be accompanied by a parameter file!. The records in the parameter file give information about the national evaluations that is common to all records in the proof file for a given brd-pop-trt combination. There must be a record in the parameter file for each brd-pop-trt combination present in the proof file. A proof file may contain only conventional (300) or only GEBV (700) records, not both. CheckProofsPara.py will fail if both record types are found in one file. 		 . Every proof file must be accompanied by a parameter file and each record in the proof file must have associated information for a single brd-pop-trt combination in the parameterfile. One file may contain records for as many or few brd-pop-trt combinations as desired. All traits in a trait group can be put in the same file, as they have in the past, but this is not essential. There is also a choice to upload evaluations for all breeds, populations and traits in a single file if that is more convenient. A proof file may contain only conventional (300) or only GEBV (700) records, not both. CheckProofsPara.py will fail if both record types are found in one file. The program checks its internal version with the value stored on the Interbull server. You will have to download the most recent version if there is a mismatch.
Line 18: Line 16:
=== The Program: ===
  . Execute: python26 CheckProofsPara.py -m <ORGCODE> –f <prooffile> –g <paramfile> [-o <outpath>]
  . Use your uppercase ORGCODE (DFSNAV, CAN CDN, DEUVIT).
  . <prooffile> is the filename of the format 300 or 700 national (G)EBV file to be checked.
  . <paramfile> is the filename of the associated format 301 or 701 evaluation parameter file.
  . <outpath> is an optional path for creation of the zipfile for uploading. 		=== The Program ===
Line 25: Line 18:
 . Example on how to run the program:
   .For MACE write: python26 CheckProofsPara.py -m CDN -f file300.GUE.CAN.prod -g file301.GUE.CAN.prod
   .For GMACE write: python26 CheckProofsPara.py -m CDN -f file700.GUE.CAN.prod -g file701.GUE.CAN.prod 		  . '''Execute''': python26 CheckProofsPara.py -m <ORGCODE> –f <prooffile> –g <paramfile> [-o <outpath>]
  <BR>
     .where:
     . '''<ORGCODE>''' is an assigned member organization code (upper case) Use your uppercase ORGCODE as shown on the upper right hand side of the IDEA page.Your organization code is reported within brackets beside the "'''Logged in as'''" information
     . '''<prooffile>''' is /path/to/filename of the format 300/700 national EBV/GEBV file
     . '''<paramfile>''' is /path/to/filename of the format 301/701 evaluation parameter file
     . '''<outpath>''' is an optional path for creation of the zipfile for uploading
Line 29: Line 26:
 .Output screen: Eventual errors are displayed on the screen (which can redirect to alog file if desired).      . Example on how to run the program:
     .For MACE write: python26 CheckProofsPara.py -m CDN -f file300.GUE.CAN.prod -g file301.GUE.CAN.prod
     .For GMACE write: python26 CheckProofsPara.py -m CDN -f file700.GUE.CAN.prod -g file701.GUE.CAN.prod

 .''' Output screen''': Eventual errors are displayed on the screen (which can redirect to alog file if desired).
Line 47: Line 48:
 * Use the command: '''python !CheckProofPara.py -m <ORGCODE> -f <prooffile> -g <paramfile> [-o outpath]''' where:
Line 49: Line 49:
 * '''<ORGCODE>''' is an assigned member organization code (upper case)  * .
Line 51: Line 51:
 * '''<prooffile>''' is /path/to/filename of the format 300 national EBV file

 * '''<paramfile>''' is /path/to/filename of the format 301 evaluation parameter file

 * '''<outpath>''' is an optional path for creation of the zipfile for uploading

 * Use your uppercase ORGCODE as shown on the upper right hand side of the IDEA page.Your organization code is reported within brackets beside the "'''Logged in as'''" information.

 * The program checks its internal version with the value stored on the Interbull server. You will have to download the most recent version if there is a mismatch.		  *

Contents

  1. General information
  2. Before Running the Programs
  3. The Program
  4. After Running the Program:
  5. Note

General information

  • A Python program called CheckProofsPara.py will check the 300/700 proof file format and the associated 301/701 parameter file for format correctness against the IDEA EBV User Manual Appendix II and prepare a zip file, if no errors are found in the files. The zipped file created with the name IB-ORGCODE-IG-yyymmddThhmmss.zip (for conventional MACE) or IB-ORGCODE-GG-yyymmddThhmmss.zip (for GMACE) contains the input proof and parameter files, renamed to proof.dat and param.dat, respectively. The zip file represents your checked data file to upload to the Interbull Centre IDEA database. For technical reason the program rejects files containing more than a million of data.

  • Every proof file must be accompanied by a parameter file and each record in the proof file must have associated information for a single brd-pop-trt combination in the parameterfile. One file may contain records for as many or few brd-pop-trt combinations as desired. All traits in a trait group can be put in the same file, as they have in the past, but this is not essential. There is also a choice to upload evaluations for all breeds, populations and traits in a single file if that is more convenient. A proof file may contain only conventional (300) or only GEBV (700) records, not both. CheckProofsPara.py will fail if both record types are found in one file. The program checks its internal version with the value stored on the Interbull server. You will have to download the most recent version if there is a mismatch.

Before Running the Programs

  • a. Ensure there is a working network connection

  • b. Install Python (version 2.6 to 2.7) if necessary

  • b. Create a working directory/folder

  • d. Download the CheckProofPara.py program from https://idea.interbull.org/software and copy it to your new directory

  • e. Copy your proof and parameter file to the working directory

The Program

  • Execute: python26 CheckProofsPara.py -m <ORGCODE> –f <prooffile> –g <paramfile> [-o <outpath>] <BR>

    • where:

    • <ORGCODE> is an assigned member organization code (upper case) Use your uppercase ORGCODE as shown on the upper right hand side of the IDEA page.Your organization code is reported within brackets beside the "Logged in as" information

    • <prooffile> is /path/to/filename of the format 300/700 national EBV/GEBV file

    • <paramfile> is /path/to/filename of the format 301/701 evaluation parameter file

    • <outpath> is an optional path for creation of the zipfile for uploading

    • Example on how to run the program:

    • For MACE write: python26 CheckProofsPara.py -m CDN -f file300.GUE.CAN.prod -g file301.GUE.CAN.prod

    • For GMACE write: python26 CheckProofsPara.py -m CDN -f file700.GUE.CAN.prod -g file701.GUE.CAN.prod

  • Output screen: Eventual errors are displayed on the screen (which can redirect to alog file if desired).

    • If no errors are detected, the data files will be written into a zip file called IB-ORGCODE-IG-yyyymmddThhmmss.zip for mace and IB-ORGCODE-GG-yyyymmddThhmmss.zip for gmace
    • (an example of output in case of no errors below).

      • Running CheckProofsPara.py version 2013-06-04 v0.6, provided by the Interbull Centre

        • . Organization code
        CDN Parameter file :: file301.GUE.CAN.prod Proof file :: file300.GUE.CAN.prod
      • 0 errors in 3 lines from paramfile
      • 0 errors in 309 lines from prooffile
      • Record counts by breed_population_trait combination
      • GUE CAN fat 103 GUE CAN mil 103 GUE CAN pro 103
      • Everything OK. Zip file is ready for upload. IB-CDN-IG-20131021T152303.zip
    • If errors are found in the file In case of errors, no zip file will be created. Correct the data and re-run the program until the data successfully pass all required checks.
    • (an example of output in case of errors below.

      • Running CheckProofsPara.py version 2013-06-04 v0.6, provided by the Interbull Centre

        • . Organization code
        CDN Parameter file :: file701.HOL.CAN.work Proof file :: file700.HOL.CAN.work

      • 2 errors in 2 lines from paramfile Parameter line 1. Base definition error: <BB12345>. See file format (eg. H00BB95) Parameter line 2. Base definition error: <BB12345>. See file format (eg. H00BB95)

      • Error(s) in parameter file. Skipping proof file.
  • .

After Running the Program:

If no errors are detected, the pedigree file will be written into a zip file called IB-ORGCODE-IG-yyyymmddThhmmss.zip. Upload the zip file to IDEA: https://idea.interbull.org/.

In case of errors, no zip file will be created. Please correct your data and re-run the program until the data successfully pass all required checks.The first 10 errors of each kind will be printed on the screen.

Note

Please do not modify the program to circumvent any checks. Doing so would be pointless because the same checking routine is used again inside IDEA to double-check the data file uploaded in the zip file.

If you need assistance, please do not hesitate to contact us at interbull@slu.se .

public/CheckProofPara (last edited 2021-09-03 10:20:08 by Valentina)