Differences between revisions 17 and 19 (spanning 2 versions)

USER MANUAL FOR THE IDEA EBV INTERFACE

Preface

The following is a manual to guide the user through the features of the new IDEA. IDEA stands for Interbull Data Exchange Area. IDEA is a restricted area accessible only to member countries through the Interbull website.

Contents

USER MANUAL FOR THE IDEA EBV INTERFACE

File Encoding

For the checking programs to run successfully the encoding of your files must be set to UTF-8.

If this is not the case a message saying

The file seems not to be in UTF-8 encoding and an illegal character has been detected. Please, convert the file to UTF-8 and rerun the check program

will appear on the screen. This means the program DID NOT run successfully.

To change the encoding of a file you have at least two options:

in Unix/Linux: use the command

iconv –f “ISO-8859-1” –t “UTF-8” –o outputfile inputfile

where ISO-8859-1 stands for the current encoding of your file, here we assume is the standard ISO encoding. But if the encoding used is a different one you have to change it accordingly.

you can use Notepad++: open the file, click on Encoding and choose the option Convert to UTF8 without BOM and save the file.

Software

The Software menu gives you access to the Interbull checking programs. By clicking on Software a drop down menu will open and you will be able to choose the type of checking program you are interested in, i.e. Pedigree, Proofs or GenoList. Under Software you will also find information on the programs and instructions on how to run them.

Proof's Checking Program

Member organizations will upload national evaluation datasets through IDEA in much the same way as they upload pedigree datasets. There is a downloadable Python program called CheckProofsPara.py available for checking proofs and parameter files. The program will check a format 300 (proof file) and an associated format 301 (parameter file) for formal correctness and prepare a zip file for upload if no errors are found in the files.

The proof file (format 300) is designed to allow considerable flexibility for the user. Each record contains an evaluation (i.e. proof) and associated information for a single brd-pop-trt combination. One file may contain records for as many or few brd-pop-trt combinations as desired. It may be natural for organizations to include all traits in a trait group in the same file, as they have in the past, but this is not essential. They may also choose to upload evaluations for all breeds, populations and traits in a single file if they find that more convenient. If problems are found with some specific combinations, only those combinations need to be re-loaded.

It is essential that proof records for all bulls are included in the same file for any one brd-pop-trt combination whether the dataset is a first submission for the IGE run in question, or a re-submission with some problem corrected. When new data are uploaded for any combination, all records present in IDEA for that combination are deleted prior to loading the new records. Thus, it is not acceptable to submit a partial dataset in order to correct the evaluations of some subset of bulls.

Every proof file must be accompanied by a parameter file!. The records in the parameter file give information about the national evaluations that is common to all records in the proof file for a given brd-pop-trt combination. There must be a record in the parameter file for each brd-pop-trt combination present in the proof file. Extra records present in the parameter file will simply be ignored. This allows users to maintain a single parameter file to be submitted with several proof files, if desired.

The user instructions and file formats (see Appendix I and II) give details on how to run the program and on the checks performed. If no errors are found, a zip file is created with the name IB-ORGCODE-IG-yyymmddThhmmss.zip containing the input proof and parameter files.

The program requires access to the internet and specifically to a few functions/pages in the IDEA web application. Firstly, in order to ensure that the user has the most recent version of the software, the internal version tag is compared to the version listed on the download page and if the user’s copy is older an appropriate message is printed and the program exits. Lists of valid orgcode’s and breed-pop-trait combinations for which the user’s organization has EBV upload authority are also obtained from the IDEA web application. If errors occur they are listed to the screen and no zip file is created. When checking the proof file only the first 10 errors of each type are listed.

Proofs

The Proofs menu gives you access to the main proofs functions which are: Upload, Review, Messages.

Upload and Verify program

By uploading data in IDEA users will no longer need to run the Verify program prior sending data to Interbull Centre. The Verify program will, in fact, be run automatically in IDEA during uploading.

To uploading functionality for parameter and proof files is available under 'Proofs/Upload'. The only file accepted by IDEA is the zip file IB-ORGCODE-IG-yyymmddThhmmss.zip produced by the CheckProofsPara.py, no other files are accepted. Once in 'Proofs/Upload' users will be able to browse the appropriate file and upload it by click on "Submit query".

The uploading of data will not happen on real time but data will be set on a queue and processed on a later time. Right after clicking "Submit query" a message on the screen will display the amount of parameter and proofs records submitted. A confirmation email will be sent to the email address associated to the user that has uploaded data.
The following are the steps the data will go through during uploading:

CheckProofsPara.py will run once again inside IDEA to assure correctness of format and contents
The Verify program will check your data against the previous one available.

Confirmation email

Either an "EBV upload success" or an "EBV upload failure" email will be sent to you upon completion of the uploading process. If the uploading is successful the "EBV upload success" email will list some basic information on your data such as:

Number of rows in parameter file associated with proofs
Number of records read from the proof file
Number of animals found by real AID
Number of animals found by alias ID
Number of flagged rows in proof file

The email can contain up to three different WARNINGs associated to three attachments:

Warning: [n] animal(s) were referenced in your file, but not present in the pedigree database. These animals were discarded! See a detailed list in the attached 'missing_animals.csv' file.
Warning: [n] animal(s) needing updated pedigree records detected! See 'ped_needed.csv' for a complete list of the animals.
Warning: Use of aliases detected! See 'ped_alias.csv' for a complete list of the animals.

An important source of information comes from Number of flagged rows in proof file: this number represents the amount of discrepancies found by the Verify program. If it is 0 (zero) it means that no discrepancies have been found and your data is automatically submitted for the IGE. If it is not 0 (zero) it means that the Verify program has found some discrepancies in your data compared to the previous one available or the system has detected animals with missing pedigree. To double check the data you need to log in IDEA and go to Proof/Review.

Animals not present in IDEA pedigree or lacking pedigree information (i.e. present in the pedigree database but with sire and dam unknown) will be excluded from the international evaluation .

Table 1 summarizes the action needed upon reception of a confirmation email with such warnings and attachments:

Attachments	Meaning	Action Required	Consequences
missing_animals.csv	The animals listed in this attachment are not present in IDEA pedigree	Log in IDEA, go to Proofs/review, withdraw the data file, prepare a file200 for these animals, upload the file in IDEA pedigree, after reception of the pedigree confirmation email upload again the proof file	If pedigree is not provided, animals are excluded from the international evaluation
ped_needed.csv	The animals listed in this attachment have sire/dam set to unknown	If you have pedigree information for these animals: log in IDEA, go to Proof/review, withdraw the data file, prepare a file200 for these animals, upload the file in IDEA pedigree, after reception of the pedigree confirmation email upload again the proof file. If you don't have such information you can submit the file: you will be asked to write an explanation	If pedigree is not provided, animals are excluded from the international evaluation
ped_alias.csv	The animal IDs listed in this attachment are alias IDs	You are requested to update your own database with the correct animals' IDs.	Alias IDs are automatically switched to their corresponding official IDs.
Refer to the section 'Submit/Withdraw data' for more information

After uploading: What's next?

Uploading represents only the very first step for submitting your data for an IGE. Here is a description of the actions you need to follow to submit your data for an IGE.

IDEA Proof/Review

The Proof/Review page contains all the information you need to review and submit your data for a IGE.

The page contains several important information grouped into three different tables: Filters table, Central table and Actions table

The Filters table allows you to filter and display the information you have uploaded in a way that better suits you. By default the Central table displays all the information associated to your account but you can costumize the displaying of information choosing among:

A list of the different breeds you have uploaded data for
A list of the population for which you have uploaded data for
A list of the traits uploaded
A list of the different status of the data uploaded
A list of the data set codes you have uploaded

Clicking "Reset all" will erase all your previous filters and display again all the information associated to your account.

The Central table displays as many rows as the breed-pop-trait combinations you have uploaded. Each row shows the following information:

Status: refers to the status of your data. There are a total of 5 different status: Pending, Submitted, Withdrawed, Accepted, Rejected
Flagged: refers to the outcome of the Verify program. Value for this column are either YES or NO. Breed-Pop-trait combination flagged YES are also highlighted in yellow.
C/G: refers to the nature of your data, Conventional/Genomic. At the moment only Conventional data are accepted.
Datasets: refers to the dataset used to upload that given Breed-Pop-trait combination
Reports: for each Breed-Pop-trait combination you get access to a "Brief" and a "Full" output of the Verify program. The "Bulls" report lists all bulls highlighted by the Verify program.

The Breed-Pop-Trait combinations not highlighted and with Flagged=NO are combinations for which the Verify program did not find any discrepancies therefore they get automatically a status=Submitted. No more actions is required from you for these combinations.

The Breed-Pop-Trait combinations highlighted in yellow require your attention. For each of them you need to check the Verify output. For your convenience a "View Summary for Selected" at the end of the Central table will display in a new page the main key checking points of the Verify output. In order for this option to work you need to select some combinations. You can do that either manually of by clicking on "Reverse selection" and then click on "View Summary for Selected".

The Actions table displays the options you have for your data with status=PENDING. You can decide to submit or withdraw such data by selecting the desired action and clicking on "Submit".

Submit data

If, after checking the Verify output ,you consider your data to be ok and want to include it in the IGE you can do so by selecting the box beside each Breed-Pop-Trait combination you want to include, select the action "Submit" and click on the "Submit" button.

Every time you submit a pending data you are required to explain the reasons for the discrepancies found by the Verify program. If the reason are breed-trait dependent you need to processed these cases one by one and provide the full explanation in the designed space. If, on the hand, the same explanation applies to several Breed-Pop-Traits combinations you can select them together in the Review page so that you will be required to write only once the explanation that all these data share.By clicking "Send message" your message will be recorded under IDEA Proofs/Messages and will be visible by you and the Interbull Centre Staff.

In the Review page, the Breed-Pop-Trait combinations you have submitted will be now displayed with status=Submitted.

Withdraw data

If you realize the data you have uploaded for some Breed-Pop-trait combinations is wrong or you want to upload pedigree information for the animals reported in the confirmation email attachments you need to withdraw your data before doing anything else. You do so by simply selecting the affected Breed-Pop-Trait combinations in the Review page, select the action Withdraw and then press the button "Submit".

When you withdraw a given Breed-Pop-Trait all records present in IDEA for that combination are deleted thus, it is not acceptable to submit a partial dataset in order to correct the evaluations of some subset of bulls. It is essential that proof records for all bulls be included in the same file for any one brd-pop-trt combination whether the dataset is a first submission for the IGE run in question, or a re-submission with some problem corrected.

-  ⇤ ← Revision 17 as of 2013-08-15 10:37:25 → 
  Size: 45406
  Editor: Valentina
  Comment:
+   ← Revision 19 as of 2013-08-15 14:16:17 → ⇥
  Size: 14537
  Editor: Valentina
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 39:
-It is essential that proof records for all bulls be included in the same file for any one brd-pop-trt combination whether the dataset is a first submission for the IGE run in question, or a re-submission with some problem corrected. When new data are uploaded for any combination, all records present in IDEA for that combination are deleted prior to loading the new records. Thus, it is not acceptable to submit a partial dataset in order to correct the evaluations of some subset of bulls.
+It is essential that proof records for all bulls are included in the same file for any one brd-pop-trt combination whether the dataset is a first submission for the IGE run in question, or a re-submission with some problem corrected. When new data are uploaded for any combination, all records present in IDEA for that combination are deleted prior to loading the new records. Thus, it is not acceptable to submit a partial dataset in order to correct the evaluations of some subset of bulls.
 Line 86:
-||||||||<style="text-align:center">Refer to the section 'Submit/Withdraw your data' for more information ||
+||||||||<style="text-align:center">Refer to the section 'Submit/Withdraw data' for more information ||
 Line 97:
+The page contains several important information grouped into three different tables: Filters table, Central table and Actions table

The __Filters__ table allows you to filter and display the information you have uploaded in a way that better suits you. By default the Central table displays all the information associated to your account but you can costumize the displaying of information choosing among:

 * A list of the different breeds you have uploaded data for
 * A list of the population for which you have uploaded data for
 * A list of the traits uploaded
 * A list of the different status of the data uploaded
 * A list of the data set codes you have uploaded

Clicking "Reset all" will erase all your previous filters and display again all the information associated to your account.
-Line 99:
+Line 111:
+The __Central__ table displays as many rows as the breed-pop-trait combinations you have uploaded. Each row shows the following information:
-Line 100:
+Line 113:
+ * Status: refers to the status of your data. There are a total of 5 different status: Pending, Submitted, Withdrawed, Accepted, Rejected
 * Flagged: refers to the outcome of the Verify program. Value for this column are either YES or NO. Breed-Pop-trait combination flagged YES are also highlighted in yellow.
 * C/G: refers to the nature of your data, Conventional/Genomic. At the moment only Conventional data are accepted.
 * Datasets: refers to the dataset used to upload that given Breed-Pop-trait combination
 * Reports: for each Breed-Pop-trait combination you get access to a "Brief" and a "Full" output of the Verify program. The "Bulls" report lists all bulls highlighted by the Verify program.
-Line 101:
+Line 119:
+The Breed-Pop-Trait combinations not highlighted and with Flagged=NO are combinations for which the Verify program did not find any discrepancies therefore they get automatically a status=Submitted. No more actions is required from you for these combinations.
-Line 102:
+Line 121:
+The Breed-Pop-Trait combinations highlighted in yellow require your attention. For each of them you need to check the Verify output. For your convenience a "View Summary for Selected" at the end of the Central table will display in a new page the main key checking points of the Verify output. In order for this option to work you need to select some combinations. You can do that either manually of by clicking on "Reverse selection" and then click on "View Summary for Selected".
-Line 103:
+Line 123:
+The __Actions__ table displays the options you have for your data with status=PENDING. You can decide to submit or withdraw such data by selecting the desired action and clicking on "Submit".
-Line 104:
+Line 125:
+=== Submit data ===
If, after checking the Verify output ,you consider your data to be ok and want to include it in the IGE you can do so by selecting the box beside each Breed-Pop-Trait combination you want to include, select the action "Submit" and click on the "Submit" button.
-Line 105:
+Line 128:
+Every time you submit a pending data you are required to explain the reasons for the discrepancies found by the Verify program. If the reason are breed-trait dependent you need to processed these cases one by one and provide the full explanation in the designed space. If, on the hand, the same explanation applies to several Breed-Pop-Traits combinations you can select them together in the Review page so that you will be required to write only once the explanation that all these data share.By clicking "Send message" your message will be recorded under IDEA Proofs/Messages and will be visible by you and the Interbull Centre Staff.
-Line 106:
+Line 130:
+In the Review page, the Breed-Pop-Trait combinations you have submitted will be now displayed with status=Submitted.
-Line 107:
+Line 132:
-{{attachment:pot_dupl.jpg}}
+{{attachment:Submit_pending.png}}
-Line 109:
+Line 134:
-=== Statistics menu ===
By clicking on "Statistics" you will get access to your "Request of Pedigree", "Corrected Animals" file, statistics on overall number of animals pending verification by other organizations and overall number of animals pending verification by your organization.
+=== Withdraw data ===
If you realize the data you have uploaded for some Breed-Pop-trait combinations is wrong or you want to upload pedigree information for the animals reported in the confirmation email attachments you need to withdraw your data before doing anything else. You do so by simply selecting the affected Breed-Pop-Trait combinations in the Review page, select the action Withdraw and then press the button "Submit".
-Line 112:
+Line 137:
-{{attachment:statistic_1.jpg}}

=== Request of Pedigree ===
You can either view or download your request of pedigree. the file has the format shown in Appendix IV. This file contains a list of all animals as loaded by reporting countries and for which you appear as the authoritative organization. This means that only you have the right to say if the pedigree record is correct or not by verifying their pedigree. ''' '''

=== Verification Process: How does it work? ===
For each animal listed in your “Request of Pedigree” file you have to upload its pedigree information as it appears in your own database, using a file format 200 or 601 and CheckPedigree.py. By doing so you are ''verifying ''the reported information as the system considers verified only the information provided by the authority organization. When you are verifying an animal's record you might find yourself in front of four possible scenarios, below you find instructions on what to do for each of them. ''' '''

'''Scenario 1'''''' '''

If you find that the reported animal ID is correct then you will have to:

 * Upload a pedigree file (file200 or 601 previously inspected by''' CheckPedigree'''.py) as it appears in your own database.

'''Scenario 2'''''' '''

If you notice that an animal has been reported with an incorrect international id then you will have to:

 * Upload a pedigree record for that animal using its correct id
 * AND upload a file202 previously checked by CheckLinks.py. The file202 will provide the cross-reference link between the reported animal's id and the animal's id you verified.

Example:

A country submitted a pedigree for the animal BSWUSAF000000569466 using the ID BSWUSAF000000''65''9466.

As it is an USA animal, the animal will appear in the USA list of Request of Pedigree. When looking at the record USA notices that two numbers have been misplaced. So USA will upload a pedigree file using the correct id for the animal (BSWUSAF000000569466) and create a file202 which will look like:

202 BSWUSAF000000659466 BSWUSAF000000569466

'''Scenario 3 '''''' '''

If you realize that a given animal has been erroneously associated to your organization and you know to which organization it should belong:

 * Prepare two file202 containing two entries for the same animal: first file sets the wrong ID to UUUUUUUUUUUUUUUUUUU then second file links the wrong ID to the new organization's ID.

N.B. it is extremely important that you upload these file in the right order,  i.e. first the link that sets the AID to invalid (first file202) and  then the link creating the cross-reference with the other AID (second  file202).

Example:

Animal BSWUSAM000803611398 is sent to USA for validation but USA realizes the animal belongs to DEU instead.

USA prepares two file202 that will look like:

first file202: 202 BSWUSAM000803611398 UUUUUUUUUUUUUUUUUUU

second file202: 202 BSWUSAM000803611398 BSWDEUM000803611398

In this way the USA ID will get status INVALID and will be linked to the DEU ID.

'''Scenario 4'''''' '''

If you do not know anything about that animal and it does not appear in your database then you have to:

 * Create  a file202 containing “UUUUUUUUUUUUUUUUUUU” in the “correct IID” field.

Doing so the animal’s international ID will be marked as invalid.

Example:

Animal BSWUSAM000000000008 cannot be found in the USA database.The file 202 will be:

202 BSWUSAM000000000008 UUUUUUUUUUUUUUUUUUU

=== Invalid IDs ===
Anytime an AID  is submitted to the database with a link to UUUUUUUUUUUUUUUUUUU it will  be marked as invalid. This means that its authoritative organization did  not find any records of it in its database. Nevertheless such AIDs can  still be valid AIDs in other countries, especially if the animal is an  old animal or if it was sold as a calf or embryos. Therefore such AID  will still be extracted from the database and it could partecipate into a  gentic evaluation if it appears itself connected to a genetic proofs or  connected via the pedigree to an animal with proofs.

''' '''

==== How can I set an animal ID as alias with an invalid status? ====
In IDEA you can now set an animal ID to invalid and then link it to another ID so that it will become an invalid alias of that given international ID.

In order to do so you need to fullfill two conditions:

1) You need to be either the authoritative organization or the first reporting organization for the ID1 (i.e. the ID you want to set as invalid alias)

2) the new ID must not be present in IDEA

Given that you fullfill both conditions, you will have to proceed in two steps:

First step: you need to set the alias-to-be ID to invalid:

1) Prepare a file 202 linking the wrong ID to UUUUUUUUUUUUUUUUUUU

2) run the file202 with the CheckLinks.py you download from IDEA/Software

3) upload the IB-202.zip file into IDEA

 . Second step: you now need to alias the invalid ID to a new ID

1) prepare a file 202 linking the set-to invalid ID (in the previousstep) with the new international ID

2) run the file202 with the CheckLinks.py

3) upload the IB-202.zip file into IDEA

The feedback you will receive in the confiramtion email will say "The AID of the first animal was replaced with the second AID, and turned into an invalid alias."

=== Corrected Foreign Animals ===
In this file you find information on the validation activity over an IID, and its pedigree, done by each authoritative organization on your foreign animals. Therefore by checking it you can get corrected IDs and/or corrected pedigree information of the animals you have reported..You can either view or download the file. The file format is specified in Appendix V.

Once you have gone through the file and applied the necessary corrections in your own database, you should reset the content of the file by clicking on "Reset these to the latest updates". By doing so the content of the file will be emptied. Only changes registered in IDEA after your reset day will be listed again in this file.

As these information are retrieved from IDEA in real time it is very important that you use the "Reset these to the latest updates" fuctionality, this will speed up the time necessary for IDEA to display the file.

=== Pending Animals ===
The file contains all your foreign animals which appear still unverified by their authoritative organization. By looking at the list you can see which animals the authoritative organization is not able to validate, you might also realize that you had made an error in the ID of the animal. If you wish to correct the animal ID of any one of these animals you can do so by uploading a file 202 previously inspected by the CheckLink.py.

The file format is reported in APPENDIX VI, the file is sorted by the authoritative organization code.

=== Animals submitted by your organization and awaiting verification by authoritative organization ===
The table shows the general situation of your  pedigree data as it is in the database at the time of your log in. The  table contains the authoritative organization, the status of the animals and the overall count. By clicking on "Download" you will be able to download the list. The file has format as descripted in APPENDIX VI.

N.B. Animals associated to your organization and with animal status set to AUTO_CREATED  are parents for which you have not provided a pedigree record. To have them dropped from this table you have to upload a pedigree file for these animals.

=== View statistics for animals that need verification from you ===
''' '''

This  table contains the list of all animals submitted to the Interbull  database by other organizations for which you appear to be the  authoritative organization and therefore you need to provide the correct  information (see verification procedure above) . The table contains the reporting organization, the country it belongs to and the overall count. This table is useful to monitor your verification process.

=== Pedigree Conflicts ===
Although your national pedigree file has been already checked by the '''CheckPedigree.'''py for pedigree conflicts, there is the possibility that the international pedigree still shows some birthdates conflicts.To check for that click on ''"Conflicts",'' you will access a table showing all conflicts where your organization is involved.

The table contains the "first animal" and the "second animal" involved in the conflict together with their relationship status, i.e. parent, grand-parent, offspring.

As every tables in IDEA, each header is an active header so you can sort the content of teh table as it better fits you. Also you can modify the number of entries shown or search for a particular animal's ID.

=== How to handle animal's conflicts? ===
Sometimes a conflict could be due to unverified animals, if this is the case please proceed on verifying the animals double checking that the new birthdate you are going to provide will no more be in conflict.

If you do not know how to fix the conflict you should contact Interbull Centre deciding what action you, as the authoritative organization for that animal, want to take.

So, in general, to fix a conflict you should:

 * Upload new pedigree file of the conflicting animal with a new no-conflicting birthdate
 * In case you do not know how to fix it you should contact Interbull Centre ''' '''

== APPENDIX I: Organization Codes as provided by Interbull Centre ==
'''<<Include(public/Org_codes)>> '''

== APPENDIX II: File Format 200 for Dairy Pedigree ==
||||||||<tablestyle="text-align:left"width="616px" style="text-align:center;vertical-align:top"> ||
||<width="96px" style="vertical-align:top">Starting position''' ''' ||<width="277px" style="vertical-align:top">Field''' ''' ||<width="115px" style="vertical-align:top">Format''' ''' ||<width="127px" style="vertical-align:top">Example''' ''' ||
||<width="96px" style="vertical-align:top">1 ||<width="277px" style="vertical-align:top">Record type^1^ ||<width="115px" style="vertical-align:top">Character 3 ||<width="127px" style="vertical-align:top">200 ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top">International ID of ANIMAL''' ''' ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top">5 ||<width="277px" style="vertical-align:top">Breed of animal ^2^ ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">HOL ||
||<width="96px" style="vertical-align:top">8 ||<width="277px" style="vertical-align:top">Country of first registration of animal^3^ ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">CAN ||
||<width="96px" style="vertical-align:top">11 ||<width="277px" style="vertical-align:top">Sex ||<width="115px" style="vertical-align:top">character 1 ||<width="127px" style="vertical-align:top">M ||
||<width="96px" style="vertical-align:top">12 ||<width="277px" style="vertical-align:top">ID number of animal^4^ ||<width="115px" style="vertical-align:top">character 12 ||<width="127px" style="vertical-align:top">000000A12345 ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top"> ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top">International ID of Sire of ANIMAL''' ''' ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top">25 ||<width="277px" style="vertical-align:top">Breed of sire of animal ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">HOL ||
||<width="96px" style="vertical-align:top">28 ||<width="277px" style="vertical-align:top">Country of first registration of sire of animal^3^ ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">CAN ||
||<width="96px" style="vertical-align:top">31 ||<width="277px" style="vertical-align:top">Sex ||<width="115px" style="vertical-align:top">character 1 ||<width="127px" style="vertical-align:top">M ||
||<width="96px" style="vertical-align:top">32 ||<width="277px" style="vertical-align:top">ID number of sire of animal^4^ ||<width="115px" style="vertical-align:top">character 12 ||<width="127px" style="vertical-align:top">556912367589 ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top"> ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top">International ID of Dam of ANIMAL''' ''' ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top">45 ||<width="277px" style="vertical-align:top">Breed of dam of animal ^2^ ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">HOL ||
||<width="96px" style="vertical-align:top">48 ||<width="277px" style="vertical-align:top">Country of first registration of dam of animal^3^ ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">CAN ||
||<width="96px" style="vertical-align:top">51 ||<width="277px" style="vertical-align:top">Sex ||<width="115px" style="vertical-align:top">character 1 ||<width="127px" style="vertical-align:top">F ||
||<width="96px" style="vertical-align:top">52 ||<width="277px" style="vertical-align:top">ID number of dam of animal^4^ ||<width="115px" style="vertical-align:top">character 12 ||<width="127px" style="vertical-align:top">123569874521 ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top"> ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top">65 ||<width="277px" style="vertical-align:top">Birth date of animal (YYYYMMDD) ||<width="115px" style="vertical-align:top">integer 8 ||<width="127px" style="vertical-align:top">19870215 ||
||<width="96px" style="vertical-align:top">74 ||<width="277px" style="vertical-align:top">Status of animal^5^ ||<width="115px" style="vertical-align:top">Integer 2 ||<width="127px" style="vertical-align:top">10 ||
||<width="96px" style="vertical-align:top">77 ||<width="277px" style="vertical-align:top">Birth date of first AI daughters (YYYYMMDD) ||<width="115px" style="vertical-align:top">Integer 8 ||<width="127px" style="vertical-align:top">19890314 ||
||<width="96px" style="vertical-align:top">86 ||<width="277px" style="vertical-align:top">Name of animal^6^ ||<width="115px" style="vertical-align:top">character 30 ||<width="127px" style="vertical-align:top">Cantarello ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top"> ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top">National ID of ANIMAL''' ''' ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top">117 ||<width="277px" style="vertical-align:top">Breed of animal^2^ ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">HOL ||
||<width="96px" style="vertical-align:top">120 ||<width="277px" style="vertical-align:top">Country ||<width="115px" style="vertical-align:top">character 3 ||<width="127px" style="vertical-align:top">CAN ||
||<width="96px" style="vertical-align:top">123 ||<width="277px" style="vertical-align:top">Sex ||<width="115px" style="vertical-align:top">character 1 ||<width="127px" style="vertical-align:top">M ||
||<width="96px" style="vertical-align:top">124 ||<width="277px" style="vertical-align:top">ID number of animal^4^ ||<width="115px" style="vertical-align:top">character 12 ||<width="127px" style="vertical-align:top">000000A12345 ||
||<width="96px" style="vertical-align:top"> ||<width="277px" style="vertical-align:top"> ||<width="115px" style="vertical-align:top"> ||<width="127px" style="vertical-align:top"> ||
||<width="96px" style="vertical-align:top;                                                                                 ">137 ||<width="277px" style="vertical-align:top;                                                                                 ">Country sending information ||<width="115px" style="vertical-align:top;                                                                                 ">character 3 ||<width="127px" style="vertical-align:top;                                                                                 ">CAN ||
||||||||<width="96px" style="vertical-align:top;text-align:center">Character encoding: UTF-8 ||


== APPENDIX IIa:File Format 601 for Beef Pedigree ==
||||||||<tablestyle="text-align:left"width="508px" style="text-align:center;vertical-align:top">File Format for Pedigree Delivery to the Interbull Centre for Interbeef Evaluation''' ''' ||
||<width="79px" style="vertical-align:top">Starting position''' ''' ||<width="229px" style="vertical-align:top">Field''' ''' ||<width="95px" style="vertical-align:top">Format''' ''' ||<width="105px" style="vertical-align:top">Example''' ''' ||
||<width="79px" style="vertical-align:top">1 ||<width="229px" style="vertical-align:top">Record type^1^ ||<width="95px" style="vertical-align:top">Character 3 ||<width="105px" style="vertical-align:top">601 ||
||<width="79px" style="vertical-align:top"> ||<width="229px" style="vertical-align:top">International ID of ANIMAL''' ''' ||<width="95px" style="vertical-align:top"> ||<width="105px" style="vertical-align:top"> ||
||<width="79px" style="vertical-align:top">5 ||<width="229px" style="vertical-align:top">Breed of animal ^2^ ||<width="95px" style="vertical-align:top">character 3 ||<width="105px" style="vertical-align:top">CHA ||
||<width="79px" style="vertical-align:top">8 ||<width="229px" style="vertical-align:top">Country of first registration of animal^3^ ||<width="95px" style="vertical-align:top">character 3 ||<width="105px" style="vertical-align:top">DNK ||
||<width="79px" style="vertical-align:top">11 ||<width="229px" style="vertical-align:top">Sex ||<width="95px" style="vertical-align:top">character 1 ||<width="105px" style="vertical-align:top">F ||
||<width="79px" style="vertical-align:top">12 ||<width="229px" style="vertical-align:top">ID number of animal^4^ ||<width="95px" style="vertical-align:top">character 12 ||<width="105px" style="vertical-align:top">008795005065 ||
||<width="79px" style="vertical-align:top"> ||<width="229px" style="vertical-align:top"> ||<width="95px" style="vertical-align:top"> ||<width="105px" style="vertical-align:top"> ||
||<width="79px" style="vertical-align:top"> ||<width="229px" style="vertical-align:top">International ID of Sire of ANIMAL''' ''' ||<width="95px" style="vertical-align:top"> ||<width="105px" style="vertical-align:top"> ||
||<width="79px" style="vertical-align:top">25 ||<width="229px" style="vertical-align:top">Breed of sire of animal ^2^ ||<width="95px" style="vertical-align:top">character 3 ||<width="105px" style="vertical-align:top">CHA ||
||<width="79px" style="vertical-align:top">28 ||<width="229px" style="vertical-align:top">Country of first registration of sire of animal^3^ ||<width="95px" style="vertical-align:top">character 3 ||<width="105px" style="vertical-align:top">DNK ||
||<width="79px" style="vertical-align:top">31 ||<width="229px" style="vertical-align:top">Sex ||<width="95px" style="vertical-align:top">character 1 ||<width="105px" style="vertical-align:top">M ||
||<width="79px" style="vertical-align:top">32 ||<width="229px" style="vertical-align:top">ID number of sire of animal^4^ ||<width="95px" style="vertical-align:top">character 12 ||<width="105px" style="vertical-align:top">654789321852 ||
||<width="79px" style="vertical-align:top"> ||<width="229px" style="vertical-align:top"> ||<width="95px" style="vertical-align:top"> ||<width="105px" style="vertical-align:top"> ||
||<width="79px" style="vertical-align:top"> ||<width="229px" style="vertical-align:top">International ID of Dam of ANIMAL''' ''' ||<width="95px" style="vertical-align:top"> ||<width="105px" style="vertical-align:top"> ||
||<width="79px" style="vertical-align:top">45 ||<width="229px" style="vertical-align:top">Breedof dam of animal ^2^ ||<width="95px" style="vertical-align:top">character 3 ||<width="105px" style="vertical-align:top">CHA ||
||<width="79px" style="vertical-align:top">48 ||<width="229px" style="vertical-align:top">Country of first registration of dam of animal^3^ ||<width="95px" style="vertical-align:top">character 3 ||<width="105px" style="vertical-align:top">DNK ||
||<width="79px" style="vertical-align:top">51 ||<width="229px" style="vertical-align:top">Sex ||<width="95px" style="vertical-align:top">character 1 ||<width="105px" style="vertical-align:top">F ||
||<width="79px" style="vertical-align:top">52 ||<width="229px" style="vertical-align:top">ID number of dam of animal^4^ ||<width="95px" style="vertical-align:top">character 12 ||<width="105px" style="vertical-align:top">852147963753 ||
||<width="79px" style="vertical-align:top">65 ||<width="229px" style="vertical-align:top">Birth date of animal (YYYYMMDD) ||<width="95px" style="vertical-align:top">integer 8 ||<width="105px" style="vertical-align:top">19861223 ||
||<width="79px" style="vertical-align:top">74 ||<width="229px" style="vertical-align:top">Name of animal^6^ ||<width="95px" style="vertical-align:top">character 30 ||<width="105px" style="vertical-align:top">CHANTAL ||
||<width="79px" style="vertical-align:top;                                                                                 ">105 ||<width="229px" style="vertical-align:top;                                                                                 ">Country sending information ||<width="95px" style="vertical-align:top;                                                                                 ">character 3 ||<width="105px" style="vertical-align:top;                                                                                 ">DNK ||
||||||||<width="79px" style="vertical-align:top;text-align:center">Character encoding: UTF-8 ||


== APPENDIX III: Format File202 Correcting International ID by the Authoritative Organization ==
'''(Creates Cross-reference links, valid for dairy and beef users)
||<tablestyle="text-align:left"style="vertical-align:top">Field description                                                                                     ''' ''' ||<style="vertical-align:top">Label                                                                                     ''' ''' ||<style="vertical-align:top">Format                                                                                     ''' ''' ||<style="vertical-align:top">Start                                                                                     ''' ''' ||<style="vertical-align:top">length                                                                                     ''' ''' ||<style="vertical-align:top">Comments                                                                                     ''' ''' ||
||<style="vertical-align:top">Record Type ||<style="vertical-align:top">Rec ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">1 ||<style="vertical-align:top">3 ||<style="vertical-align:top">202 ||
||<style="vertical-align:top">Wrong/Alias IID ||<style="vertical-align:top">W_AIID ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">5 ||<style="vertical-align:top">19 ||<style="vertical-align:top">usual structure of IID ||
||<style="vertical-align:top;">Correct IID ||<style="vertical-align:top;">R_AIID ||<style="vertical-align:top;">ASCII ||<style="vertical-align:top;">25 ||<style="vertical-align:top;">19 ||<style="vertical-align:top;">usual structure of IID ||
||||||||||||<style="vertical-align:top;text-align:center">Character encoding: UTF-8 ||


'''



== APPENDIX IV: Request of Pedigree for Authoritative Organization ==
||<tablewidth="621px" tablestyle="text-align:left"style="vertical-align:top">Field description''' ''' ||<style="vertical-align:top">Label''' ''' ||<style="vertical-align:top">Format''' ''' ||<style="vertical-align:top">Start''' ''' ||<style="vertical-align:top">length''' ''' ||<width="165px" style="vertical-align:top">Comments''' ''' ||
||<style="vertical-align:top">Record Type ||<style="vertical-align:top">Rec ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">1 ||<style="vertical-align:top">3 ||<width="165px" style="vertical-align:top">201 ||
||<style="vertical-align:top">Animal IID''' ''' ||<style="vertical-align:top">A_IID ||<style="vertical-align:top">ASCII''' ''' ||<style="vertical-align:top">5''' ''' ||<style="vertical-align:top">19''' ''' ||<width="165px" style="vertical-align:top">usual structure of IID''' ''' ||
||<style="vertical-align:top">Sire IID''' ''' ||<style="vertical-align:top">S_IID''' ''' ||<style="vertical-align:top">ASCII''' ''' ||<style="vertical-align:top">25''' ''' ||<style="vertical-align:top">19''' ''' ||<width="165px" style="vertical-align:top">usual structure of IID''' ''' ||
||<style="vertical-align:top">Dam IID''' ''' ||<style="vertical-align:top">D_IID''' ''' ||<style="vertical-align:top">ASCII''' ''' ||<style="vertical-align:top">45''' ''' ||<style="vertical-align:top">19''' ''' ||<width="165px" style="vertical-align:top">usual structure of IID''' ''' ||
||<style="vertical-align:top">Animal Birthdate ||<style="vertical-align:top">A_BDATE ||<style="vertical-align:top">yyyymmdd ||<style="vertical-align:top">65 ||<style="vertical-align:top">8 ||<width="165px" style="vertical-align:top">Date format ||
||<style="vertical-align:top">Animal Name ||<style="vertical-align:top">A_NAME ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">74 ||<style="vertical-align:top">30 ||<width="165px" style="vertical-align:top"> ||
||<style="vertical-align:top">Reporting Organization ||<style="vertical-align:top">A_RC ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">116 ||<style="vertical-align:top">3 ||<width="165px" style="vertical-align:top">Organization providing the data ||
||<style="vertical-align:top;">Breed composition ||<style="vertical-align:top;">A_breedcomp ||<style="vertical-align:top;">ASCII ||<style="vertical-align:top;">120 ||<style="vertical-align:top;">8 ||<width="165px" style="vertical-align:top;                                                                                 ">To be left blank for the time being. ||
||||||||||||<style="vertical-align:top;text-align:center">Character encoding: UTF-8 ||


 .

== APPENDIX V: Corrected Foreign Animals ==
Each record line is separated from the next by a ‘\n’ '''
||<tablewidth="621px" tablestyle="text-align:left"style="vertical-align:top">Field description                                                                                     ''' ''' ||<style="vertical-align:top">Label                                                                                     ''' ''' ||<style="vertical-align:top">Format                                                                                     ''' ''' ||<style="vertical-align:top">Start                                                                                     ''' ''' ||<style="vertical-align:top">length                                                                                     ''' ''' ||<width="165px" style="vertical-align:top">Comments                                                                                     ''' ''' ||
||<style="vertical-align:top">Reported IID ||<style="vertical-align:top">R_IID ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">1 ||<style="vertical-align:top">19 ||<width="165px" style="vertical-align:top">usual structure of IID ||
||<style="vertical-align:top">Official Animal IID ||<style="vertical-align:top">A_IID ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">21 ||<style="vertical-align:top">19 ||<width="165px" style="vertical-align:top">usual structure of IID ||
||<style="vertical-align:top">Sire IID ||<style="vertical-align:top">S_IID ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">41 ||<style="vertical-align:top">19 ||<width="165px" style="vertical-align:top">usual structure of IID ||
||<style="vertical-align:top">Dam IID ||<style="vertical-align:top">D_IID ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">61 ||<style="vertical-align:top">19 ||<width="165px" style="vertical-align:top">usual structure of IID ||
||<style="vertical-align:top">Animal Birthdate ||<style="vertical-align:top">A_BDATE ||<style="vertical-align:top">yyyymmdd ||<style="vertical-align:top">81 ||<style="vertical-align:top">8 ||<width="165px" style="vertical-align:top">yyyymmdd ||
||<style="vertical-align:top;">Animal Name ||<style="vertical-align:top;">A_NAME ||<style="vertical-align:top;">ASCII ||<style="vertical-align:top;">90 ||<style="vertical-align:top;">30 ||<width="165px" style="vertical-align:top;                                                                                 "> ||
||||||||||||<style="vertical-align:top;text-align:center">Character encoding: UTF-8 ||


'''



== APPENDIX VI: Animals Pending Verification ==
||<tablewidth="621px" tablestyle="text-align:left"style="vertical-align:top">Field description''' ''' ||<style="vertical-align:top">Label''' ''' ||<style="vertical-align:top">Format''' ''' ||<style="vertical-align:top">Start''' ''' ||<style="vertical-align:top">length''' ''' ||<width="165px" style="vertical-align:top">Comments''' ''' ||
||<style="vertical-align:top">Record Type ||<style="vertical-align:top">Rec ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">1 ||<style="vertical-align:top">3 ||<width="165px" style="vertical-align:top">203 ||
||<style="vertical-align:top">Animal IID''' ''' ||<style="vertical-align:top">A_IID ||<style="vertical-align:top">ASCII''' ''' ||<style="vertical-align:top">5''' ''' ||<style="vertical-align:top">19''' ''' ||<width="165px" style="vertical-align:top">usual structure of IID''' ''' ||
||<style="vertical-align:top">Sire IID''' ''' ||<style="vertical-align:top">S_IID''' ''' ||<style="vertical-align:top">ASCII''' ''' ||<style="vertical-align:top">25''' ''' ||<style="vertical-align:top">19''' ''' ||<width="165px" style="vertical-align:top">usual structure of IID''' ''' ||
||<style="vertical-align:top">Dam IID''' ''' ||<style="vertical-align:top">D_IID''' ''' ||<style="vertical-align:top">ASCII''' ''' ||<style="vertical-align:top">45''' ''' ||<style="vertical-align:top">19''' ''' ||<width="165px" style="vertical-align:top">usual structure of IID''' ''' ||
||<style="vertical-align:top">Animal Birthdate ||<style="vertical-align:top">A_BDATE ||<style="vertical-align:top">yyyymmdd ||<style="vertical-align:top">65 ||<style="vertical-align:top">8 ||<width="165px" style="vertical-align:top">yyyymmdd ||
||<style="vertical-align:top;">Animal Name ||<style="vertical-align:top;">A_NAME ||<style="vertical-align:top;">ASCII ||<style="vertical-align:top;">74 ||<style="vertical-align:top;">30 ||<width="165px" style="vertical-align:top;                   "> ||
||<style="vertical-align:top">Animal status ||<style="vertical-align:top">A_STATUS ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">106 ||<style="vertical-align:top">20 ||<width="165px" style="vertical-align:top">NOT_VERIFIED ||
||<style="vertical-align:top">Authoritative Organization ||<style="vertical-align:top">AUTH_ORG ||<style="vertical-align:top">ASCII ||<style="vertical-align:top">128 ||<style="vertical-align:top">3 ||<width="165px" style="vertical-align:top">Organization that will verify the record ||
||||||||||||<style="vertical-align:top;text-align:center">Character encoding: UTF-8 ||


== APPENDIX VII: FREQUENTLY ASKED QUESTIONS ==
Q.''' How do I know that the pedigree file I have submitted has been successfully uploaded into the database? '''

A.''' You will receive a confirmation email with some information on the data you have uploaded.'''

Q.

''' How do I know if I have any animal to verify? '''A.''' By clicking on “Statistic/'''View statistics for animals that need verification from you'''” the table will show you the number of animals you need to verify and their reporting organization. '''

Q.''' What do I do if I have animals pending verification? '''

A.''' Download from the statistic menu the request of pedigree file. For each of the animal listed in the file you need to provide pedigree information as they appear in your own database and upload them using a file200 format previously checked by CheckPedigree.py. If you notice that the IID of the animals is wrong or is an alias ID then you also have to submit a file202 previously checked by CheckLinks.py. '''

Q.''' What I have to do if some IIDs listed in the request of pedigree cannot be found in my own database? '''

A.''' In this case you only have to submit a file202, checked by CheckLinks.py, with 19 “U”s (i.e. UUUUUUUUUUUUUUUUUUU) in the correct IID field. '''

Q.''' What do I do if I notice that an animal has been reported using a cross-reference ID instead than the official IID? '''

A.''' Provide the pedigree information for that animal as it appears in your own database using a file200 previously checked by CheckPedigree.py and also provide a file202, previously checked by CheckLinks.py, putting the correct official IID in the correct IID field. '''

Q.''' Can the actual system distinguish between ID formally correct but written in a different format and the real wrong ID? '''

A.''' Anytime the authoritative organization provides a file202 the “wrong/alias ID” a croos-reference record is created. The system does not distinguish between “formally correct” or “real wrong” IDs. The system is built on the principle that every ID is considered as potentially wrong until it gets verified by the authoritative organization. If an authoritative organization provides a “correct ID” (cross-reference link, file202) it means that the ID provided by the reporting organization was not entirely correct so it gets replaced with the one provided by the authoritative organization. The information provided by the reporting country is not lost but stored in the alternative scheme table. In case the ID reported in the correct field is represented by the unknown ID (i.e. 19 “U”s) then that ID is treated as “real wrong” and and marked as invalid. '''
+When you withdraw a given Breed-Pop-Trait all records present in IDEA for that combination are deleted thus, it is not acceptable to submit a partial  dataset in order to correct the evaluations of some subset of bulls.  It is essential that proof records for all bulls be included in the  same file for any one brd-pop-trt combination whether the dataset is a  first submission for the IGE run in question, or a re-submission with  some problem corrected.