Differences between revisions 3 and 31 (spanning 28 versions)
⇤ ← Revision 3 as of 2013-08-16 10:12:39 →
Size: 3837
Editor: Valentina
Comment:
← Revision 31 as of 2013-11-21 15:12:58 → ⇥
Size: 4795
Editor: Hjerpe
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= README for CheckProofPara.py =
'''Information about the program:'''		 <<TableOfContents>>
Line 4: Line 3:
'''!CheckProofPara.py''' will perform a series of checks on your performance and parameter data to insure that the data is all right. If no errors are detected then a zip file will be created. The zip file represents your checked data file to upload to the Interbull Centre IDEA database. For technical reason the program rejects files containing more than a million of data. === General information ===
 . A Python program called CheckProofsPara.py will check the 300/700 proof file format and the associated 301/701 parameter file for format correctness against the IDEA EBV User Manual Appendix II and prepare a zip file if no errors are found in the files. The zipped file created with the name IB-ORGCODE-IG-yyymmddThhmmss.zip (for conventional MACE) or IB-ORGCODE-GG-yyymmddThhmmss.zip (for GMACE) contains the input proof and parameter files, renamed to proof.dat and param.dat, respectively. The zip file represents your checked data file to upload to the Interbull Centre IDEA database (https://idea.interbull.org/). For technical reason the program rejects files containing more than a million of data.
Line 6: Line 6:
The checks performed on the parameter file relate to:  . Every proof file must be accompanied by a parameter file and each record in the proof file must have associated information for a single brd-pop-trt combination in the parameterfile. One zip file may contain records for as many or few brd-pop-trt combinations as desired. All traits in a trait group can be put in the same file, as they have in the past, but this is not essential. There is also an option to upload evaluations for all breeds, populations and traits in a single file if that is more convenient. A proof/parameter file may contain only conventional (300/301) or only GEBV (700/701) records, not both. CheckProofsPara.py will fail if both record types are found in one file. The program checks its internal version with the version stored on the Interbull server and the program must be re downloaded if there is a mismatch in versions.
Line 8: Line 8:
 * Check correct record type (301) and record length (46)
 * Check correct use of breed abbreviation as reported in the IDEA EBV User Manual, Appendix II
 * Check correct use of population codes as reported in the IDEA EBV User Manual, Appendix II
 * Check correct use of trait codes as evaluated at ITBC and reported in the IDEA EBV User Manual, Appendix II
 * Check that the organization has upload authority for the 'brd;pop;trt' combination, fetched from IDEA
 * Check that the other information in the file are according to the specification reported in the IDEA EBV User Manual, Appendix II		 Please do not modify the program to circumvent any checks. Doing so would be pointless because the same checking routine is used again inside IDEA to double-check the data file uploaded in the zip file. If you need assistance, please do not hesitate to contact us at [[mailto:interbull-hgen@slu.se|interbull@slu.se]] .
Line 15: Line 10:
The checks performed on the proof file relate to:
Line 17: Line 11:
 * Check correct record type (300) and record length (85)
 * Check that correct values for type of proofs are used according to what reported in the IDEA EBV User Manual, Appendix I
 * Check that correct publication codes are used according to what reported in the IDEA EBV User Manual, Appendix I
 * Check that correct values for bull status are used according to what reported in the IDEA EBV User Manual, Appendix I
 * Check that the 'brd,pop,trt' combination has also an entry in the parameter file.
 * Check that the other information in the file are according to the specification reported in the IDEA EBV User Manual, Appendix I		 === Before Running the Programs ===
Line 24: Line 13:
=== Before Running the Program: ===
 1. Install [[http://www.python.org|Python]] (version 2.5 to 2.7)     if necessary
 1. Create a working directory/folder
 1. Download the !CheckProofPara.py program from https://idea.interbull.org/software and copy it to your new directory
 1. Copy your proof and parameter file to the working directory		   . a. Ensure there is a working network connection
  . b. Install [[http://www.python.org|Python]] (version 2.6 to 2.7) if necessary
  . b. Create a working directory/folder
  . d. Download the CheckProofPara.py program from https://idea.interbull.org/software and copy it to your new directory
  . e. Copy your proof and parameter file to the working directory
Line 30: Line 19:
=== Running the Program: ===
 * Ensure there is a working network connection		 === The Program ===
Line 33: Line 21:
 * Use the command: '''python !CheckProofPara.py -m <ORGCODE> -f <prooffile> -g <paramfile> [-o outpath]''' where:   . '''Execute''': <<span(style="background-color: #00FF00;")>> python26 CheckProofsPara.py -m <ORGCODE> –f <prooffile> –g <paramfile> [-o <outpath>] <<span>>
     .where:
     . <ORGCODE>=the assigned member organization code (upper case) as shown on the upper right hand side of the IDEA page. Your organization code is reported within brackets beside the "'''Logged in as'''" information
     . <prooffile>= /path/to/filename of the format 300/700 national EBV/GEBV file
     . <paramfile>= /path/to/filename of the format 301/701 evaluation parameter file
     . <outpath>= optional path for creation of the zipfile for uploading
Line 35: Line 28:
 * '''<ORGCODE>''' is an assigned member organization code (upper case)      . '''''Example on how to run the program:'''''
     .For MACE write: python26 CheckProofsPara.py -m CDN -f file300.GUE.CAN.prod -g file301.GUE.CAN.prod
     .For GMACE write: python26 CheckProofsPara.py -m CDN -f file700.GUE.CAN.prod -g file701.GUE.CAN.prod
Line 37: Line 32:
 * '''<prooffile>''' is /path/to/filename of the format 300 national EBV file  .''' Output''': Eventual errors are displayed on the screen and can redirect to alog file if desired.
     . Example of output on the screen in case of no errors:
Line 39: Line 35:
 * '''<paramfile>''' is /path/to/filename of the format 301 evaluation parameter file        . Running CheckProofsPara.py version 2013-06-04 v0.6, provided by the Interbull Centre
       . Organization code : CDN
       . Parameter file : file301.GUE.CAN.prod
       . Proof file : file300.GUE.CAN.prod
       . 0 errors in 3 lines from paramfile
       . 0 errors in 309 lines from prooffile
       . Record counts by breed_population_trait combination
       . GUE CAN fat 103 GUE CAN mil 103 GUE CAN pro 103
       .Everything OK. Zip file is ready for upload. IB-CDN-IG-20131021T152303.zip
Line 41: Line 45:
 * '''<outpath>''' is an optional path for creation of the zipfile for uploading      . In case of errors, no zip file will be created. Correct the data and re-run the program until the data successfully pass all required checks.The first 10 errors of each kind will be printed on the screen.
Line 43: Line 47:
 * Use your uppercase ORGCODE as shown on the upper right hand side of the IDEA page.Your organization code is reported within brackets beside the "'''Logged in as'''" information.      . Example of output in case of errors.
      . Running CheckProofsPara.py version 2013-06-04 v0.6, provided by the Interbull Centre
      . Organization code : CDN
      . Parameter file : file701.HOL.CAN.work
      . Proof file : file700.HOL.CAN.work
      . 2 errors in 2 lines from paramfile
      . Parameter line 1. Base definition error: <BB12345>. See file format (eg. H00BB95)
      . Parameter line 2. Base definition error: <BB12345>. See file format (eg. H00BB95)
      . Error(s) in parameter file. Skipping proof file.
Line 45: Line 57:
 * The program checks its internal version with the value stored on the Interbull server. You will have to download the most recent version if there is a mismatch.

=== After Running the Program: ===
If no errors are detected, the pedigree file will be written into a zip file called '''IB-ORGCODE-IG-yyyymmddThhmmss.zip'''. Upload the zip file to IDEA: https://idea.interbull.org/.

In case of errors, no zip file will be created. Please correct your data and re-run the program until the data successfully pass all required checks.The first 10 errors of each kind will be printed on the screen.

=== Note ===
Please do not modify the program to circumvent any checks. Doing so would be pointless because the same checking routine is used again inside IDEA to double-check the data file uploaded in the zip file.

----
If you need assistance, please do not hesitate to contact us at [[mailto:interbull-hgen@slu.se|interbull@slu.se]] .

Contents

  1. General information
  2. Before Running the Programs
  3. The Program

General information

  • A Python program called CheckProofsPara.py will check the 300/700 proof file format and the associated 301/701 parameter file for format correctness against the IDEA EBV User Manual Appendix II and prepare a zip file if no errors are found in the files. The zipped file created with the name IB-ORGCODE-IG-yyymmddThhmmss.zip (for conventional MACE) or IB-ORGCODE-GG-yyymmddThhmmss.zip (for GMACE) contains the input proof and parameter files, renamed to proof.dat and param.dat, respectively. The zip file represents your checked data file to upload to the Interbull Centre IDEA database (https://idea.interbull.org/). For technical reason the program rejects files containing more than a million of data.

  • Every proof file must be accompanied by a parameter file and each record in the proof file must have associated information for a single brd-pop-trt combination in the parameterfile. One zip file may contain records for as many or few brd-pop-trt combinations as desired. All traits in a trait group can be put in the same file, as they have in the past, but this is not essential. There is also an option to upload evaluations for all breeds, populations and traits in a single file if that is more convenient. A proof/parameter file may contain only conventional (300/301) or only GEBV (700/701) records, not both. CheckProofsPara.py will fail if both record types are found in one file. The program checks its internal version with the version stored on the Interbull server and the program must be re downloaded if there is a mismatch in versions.

Please do not modify the program to circumvent any checks. Doing so would be pointless because the same checking routine is used again inside IDEA to double-check the data file uploaded in the zip file. If you need assistance, please do not hesitate to contact us at interbull@slu.se .

Before Running the Programs

  • a. Ensure there is a working network connection

  • b. Install Python (version 2.6 to 2.7) if necessary

  • b. Create a working directory/folder

  • d. Download the CheckProofPara.py program from https://idea.interbull.org/software and copy it to your new directory

  • e. Copy your proof and parameter file to the working directory

The Program

  • Execute: python26 CheckProofsPara.py -m <ORGCODE> –f <prooffile> –g <paramfile> [-o <outpath>]

    • where:

    • <ORGCODE>=the assigned member organization code (upper case) as shown on the upper right hand side of the IDEA page. Your organization code is reported within brackets beside the "Logged in as" information

    • <prooffile>= /path/to/filename of the format 300/700 national EBV/GEBV file

    • <paramfile>= /path/to/filename of the format 301/701 evaluation parameter file

    • <outpath>= optional path for creation of the zipfile for uploading

    • Example on how to run the program:

    • For MACE write: python26 CheckProofsPara.py -m CDN -f file300.GUE.CAN.prod -g file301.GUE.CAN.prod

    • For GMACE write: python26 CheckProofsPara.py -m CDN -f file700.GUE.CAN.prod -g file701.GUE.CAN.prod

  • Output: Eventual errors are displayed on the screen and can redirect to alog file if desired.

    • Example of output on the screen in case of no errors:

      • Running CheckProofsPara.py version 2013-06-04 v0.6, provided by the Interbull Centre

      • Organization code : CDN
      • Parameter file : file301.GUE.CAN.prod
      • Proof file : file300.GUE.CAN.prod
      • 0 errors in 3 lines from paramfile
      • 0 errors in 309 lines from prooffile
      • Record counts by breed_population_trait combination
      • GUE CAN fat 103 GUE CAN mil 103 GUE CAN pro 103
      • Everything OK. Zip file is ready for upload. IB-CDN-IG-20131021T152303.zip
    • In case of errors, no zip file will be created. Correct the data and re-run the program until the data successfully pass all required checks.The first 10 errors of each kind will be printed on the screen.
    • Example of output in case of errors.

      • Running CheckProofsPara.py version 2013-06-04 v0.6, provided by the Interbull Centre

      • Organization code : CDN
      • Parameter file : file701.HOL.CAN.work
      • Proof file : file700.HOL.CAN.work
      • 2 errors in 2 lines from paramfile

      • Parameter line 1. Base definition error: <BB12345>. See file format (eg. H00BB95)

      • Parameter line 2. Base definition error: <BB12345>. See file format (eg. H00BB95)

      • Error(s) in parameter file. Skipping proof file.

public/CheckProofPara (last edited 2021-09-03 10:20:08 by Valentina)