{{attachment:ibc_logo.jpg}}

= USER MANUAL FOR THE IDEA BEEF INTERFACE =
== Preface ==
The following is a  manual to guide the user through the features of the new '''IDEA'''.''' IDEA''' stands for ''Interbull Data Exchange Area.'' IDEA is a restricted area accessible only to member countries through the Interbull website.

<<TableOfContents()>>

== Software menu ==
The Software menu gives you access to the Interbull checking programs. By clicking on Software a drop down menu will open and you will be able to choose the type of checking program you are interested in, i.e. Pedigree, !AnimInfo, Performance. Under Software you will also find information on the programs and instructions on how to run them.

=== Performance Checking Program ===
A Python program called !CheckPerformance.py will check the xml file for format correctness against IDEA. The program prepares a zip file, IB-BEEF-ORGCODE-yyyymmddThhmmss.tbz2 if no errors are found in the file. The zip file contains your checked data file to upload to the Interbull Centre IDEA database (https://idea.interbull.org/).

The [[https://wiki.interbull.org/public/CheckPerformance?action=print|user instructions for the CheckPerformance.py program]] give details on how to run the program.

Another Python program called !MergePerformance.py could be used prior to !CheckPerformance.py to merge two or more performance files into a single performance file containing all data in the input files.

The [[https://wiki.interbull.org/public/MergePerformance?action=print|user instructions for the MergePerformance.py program]] give details on how to run the program.

To help users transitioning from the old system based on flat files and the new system based on xml files, a convert_performance.py is also available in the Software menu. The program will convert your file602, 603, 604 and 605 (D/M) into an xml file.

The [[https://wiki.interbull.org/public/performance_convert_program?action=print|user instructions for the convert_performance.py]] will give details on how to run the program and on the checks performed.

Useful information on how to create your xml file are available [[https://wiki.interbull.org/public/FilePerformanceXML?action=print|here]]

== Beef menu ==
The Beef menu gives you access to the main beef functions which are:Upload, Review and Messages.

=== Upload and Verify program ===
When the data is uploaded in IDEA, it will be checked for consistency against the previous data available in the system by a verify program which will be ran automatically in IDEA during uploading.

The uploading functionality is available under 'Beef/Upload'. The only file accepted by IDEA is the zip file IB-BEEF-ORGCODE-yyyymmddThhmmss.zip produced by the !CheckPerformance.py, no other files are accepted. Once in 'Beef/Upload' users will be able to browse the appropriate file and upload it by click on "Upload".

The uploading of data will not happen on real time but data will be set on a queue and processed on a later time. Short after clicking "Upload" a message of the type "Successfully uploaded file IB-BEEF-SDAB-20190403T120423.tbz2: compressed  file detected correctly File queued for batch processing. You will be  notified by email upon completion"  A confirmation email will be sent to the email address associated to the user that has uploaded data.<<BR>> The following are the steps the data will go through during uploading:

 a. !CheckPerformance.py will run once again inside IDEA to assure correctness of format and contents
 a. The Verify program will check your data against the previous one available.

==== Confirmation email ====
Either a "Performance file upload" or a "Performance upload error" email will be sent to you upon completion of the uploading process.

==== UPLOAD STATUS ====
IDEA will  soon remove the ability to receive feedback through email, and instead  switch to a dedicated page where you (and other members of your  organization) can view all feedback relevant to your organization. This  will be in the form of a table, where you can look back at current and  past feedbacks, download the relevant files and so on. There will also  be an RSS feed available, so you can set up notifications of new  feedback as it becomes available, in a way similar to previous email  feedback.

Communications  from IDEA regarding the status (success or failure) of all type of  uploads performed by your organization (pedigree, EBV, GEBV,  performance, animinfo) are now listed in here and divided between  “Successful” or “Failed” uploads. The information displayed are from a  RSS feed system. Here the steps you shall do in order to display such  feeds correctly:

 * Go to Upload Status
 * Select “Successful” (or Failed)
 * Click on “Create subscription feed link”
 * Click on “Copy subscription feed link”

If  you think the link has been leaked/spread (remember that the link is  supposed to be personal), you can remove it at any time by clicking on  “Remove subscription feed link”. If that happens you shall create and  copy a new subscription feed link.

With  this new feedback system, information on all type of uploads will be  available to each and every members of the same organization and not  only to the person making the upload. Moving away from an email based  feedback methodology and to an RSS based one, will make the feedback  more stable and reliable, without risk of spam or mail server problems.  Also, it makes it easier to programmatically handle feedback from the  IDEA system.

You should copy your subscription feed link inside your email provider. Here is how you do it if you are using Outlook:

 * Right click on File
 * Right click on “Account Settings” from the Account      Settings drop down menu
 * In the new window that opens click on “RSS feeds”
 * Click on “New” and paste the RSS feed link you have      copied from IDEA.

Using Thunderbird:

 * Right  click on the News Feed folder and choose      Subscription (you can  also add a sub-folder and add the subscription      there)
 * Add your Subscription link into the Feed URL text box      in the Subscription popup box and click Add

For  the time being, IDEA emails and RSS feeds will co-exists to give time  to all IDEA users to get acquainted with the RSS feed. From 1st January  2023 IDEA will provide solely feedback based on the RSS feeds.

==== Upload successful ====
If the uploading is successful the "Performance file upload" email will list some basic information on your data such as:  <<BR>>

 . Number of performance traits inserted
 . Number of performance traits removed
 . Number of animals removed
 . Number of alias detected
 . Number of animals needing pedigree

The email comes with an xml file attached providing detailed information on the data uploaded.

Reception of the "Performance file upload" email indicates only that the file has bees successfully uploaded into IDEA and the data is now ready to be review and submitted. Only submitted data can be used in a given evaluation.

<<span(style="color: #FF0000;")>> Please note that the data IS NOT AUTOMATICALLY submitted. After uploading to IDEA the data has to be submitted by the user in the Beef/Review page (see below for detailed actions)<<span>>. If entries in the review page appears highlighted in yellow ,it means that the Verify program has found some discrepancies  in your data compared to the previous one available or the system has  detected animals with missing pedigree.  To review the data you  need to log in IDEA and go to Beef/Review.

==== Actions for attachments ====
<<span(style="color: #FF0000;")>> Animals not present in IDEA pedigree or lacking pedigree information (i.e. present in the pedigree database but with sire and dam unknown) will be excluded from the international evaluation.<<span>>

Table 1 summarizes the action needed upon reception of a confirmation email with warnings and/or attachments:
||<tablestyle="text-align:center">'''Warning''' ||'''Meaning''' ||'''Action Required''' ||'''Consequences''' ||
||'''Animals needing pedigree''' ||The animals listed in the xml attachment have sire/dam set to unknown ||If you have pedigree information for these animals: <<BR>> log in IDEA,  <<BR>>go to Beef/review,  <<BR>>withdraw the data file,  <<BR>>prepare a file200 for these animals,  <<BR>>upload the file in IDEA pedigree, <<BR>> after reception of the pedigree confirmation email upload again the performance file.  <<BR>>If you don't have such information you can submit the file: you will be asked to write an explanation ||If pedigree is not provided, animals are excluded from the international evaluation ||
||'''Aliases detected''' ||The animal IDs listed in the xml attachment appear as alias IDs ||You are requested to update your own database with the correct animals' IDs. <<BR>> ||Alias IDs are automatically switched to their corresponding official IDs. ||
||'''Animals deleted''' ||Animals included in your performance file could be not inserted in the database if <<BR>>1) the animal is not present in the pedigree <<BR>> 2) the animal has duplicates in the performance file (either real duplicates or through aliases) ||If the animals are not included in the IDEA pedigree you should withdraw your data, upload the pedigree information in IDEA and upload your performance file again. ||Animals are not inserted in the database and therefore will not be included in the evaluation ||
||||||||<style="text-align:center">Refer to the section 'Submit/Withdraw data' for more information ||




==== Upload not successful ====
There can be three main reasons for an uploading to fail, the different reasons are descripted in the specific "Performance upload error" email you could receive. The table below presents and explaines each of the possible cases:
||<tablewidth="1281px" tableheight="308px">'''Email content''' ||'''Meaning''' ||'''Action Required''' ||
||'System internal error. This should usually not happen. Contact staff for clarification' ||Technical problems with the database server ||contact Interbull Centre ||
||'The data already exists in the database and no combinations were updated due to not having the right status for re-uploading; the proper way to re-upload combinations is to withdraw them first (or ask IBC staff to reset them).' ||The content of the file was not uploaded ||Log in IDEA, <<BR>>go to Beef/review,  <<BR>>withdraw the specific breed-trait combinations present in the file <<BR>> re-upload the file ||
||'Formal correctness errors were encountered in the uploaded file' ||The file was not uploaded due to format errors ||Review the file with the !CheckPerformance.py <<BR>> re-upload the file ||




<<BR>>

=== Review your data ===
==== General ====
Uploading represents only the very first step for submitting your data for an IGE. Here is a description of the actions you need to follow to submit your data for an IGE. The Beef/Review page contains all the information you need to review and submit your data for a IGE.

The page contains several important information grouped into three different tables: Filters table, Central (Include) table and Actions table

==== Filter table ====
The __Filters__ table allows you to filter and display the information you have uploaded in a way that better suits you. By default the Central (Include) table displays all the information associated to your account but you can costumize the displaying of information choosing among:

 * A list of the different breeds you have uploaded data for
 * A list of the population for which you have uploaded data for
 * A list of the traits uploaded
 * A list of the different status of the data uploaded
 * A list of the data set codes you have uploaded

Clicking "Reset all" will erase all your previous filters and display again all the information associated to your account.

{{attachment:review_page.jpg||width="1664",height="766"}}

==== Central (Include) table ====
The __Central__ (Include) table displays as many rows as the breed-pop-trait combinations you have uploaded. Each row shows the following information:

 * Status: refers to the status of your data. There are a total of 5 different status: Pending, Submitted, Withdrawed, Accepted, Rejected.
 * Flagged: refers to the outcome of the Verify program. Value for this column are either YES or NO. Breed-Pop-trait combination flagged YES are also highlighted in yellow.
 * P/G: refers to the nature of your data, Performance or Genomic. For the time being only Performance data are accepted.
 * Datasets: refers to the dataset used to upload a given Breed-Pop-trait combination
 * Reports: for each Breed-Pop-trait combination you uploaded you get access to three different links:
  * '''''Full''''': Gives you access to the whole report from the Verify program. ''' '''
  * '''''Json'': '''Gives you access to a brief overview of the possible issues related with the data''' '''
  * '''''Items:''''': Gives you access to the list of animals highlighted by the Verify program which show changes in information. ''' '''

The Breed-Pop-Trait combinations not highlighted and with Flagged=NO are combinations for which the Verify program did not find any discrepancies, nevertheless they will still get a status= Pending as your final approval on the data is still required.

The Breed-Pop-Trait combinations highlighted in yellow require your attention. For each of them you need to check the Verify output. For your convenience, selecting such combinations and clicking on "View Summary for Selected", located at the end of the Central table, will display in a new page the main key checking points of the Verify output. You can select the combination either manually, by clicking in the box in the "include" column,  or by clicking on "Reverse selection".'''  '''

==== Actions table ====
The Actions table displays the options you have for your data with status=PENDING. You can decide to submit or withdraw such data by selecting the desired action and clicking on "Submit".'''  '''

==== Submit data ====
If, after checking the Verify output ,you consider your data to be ok and want to include it in the IGE you can do so by selecting the box beside each Breed-Pop-Trait combination you want to include, select the action "Submit" and click on the "Submit" button.

Every time you submit a pending data you are required to explain the reasons for the discrepancies found by the Verify program. If the reason are breed-trait dependent you need to processed these cases one by one and provide the full explanation in the designed space "Change comment". If, on the other hand, the same explanation applies to several Breed-Pop-Traits combinations you can select them together in the Review page so that you will be required to write only once the explanation that all these data share.By clicking "Send message" your message will be recorded under IDEA EBV/Messages and will be visible by you and the Interbull Centre Staff.

In the Review page, the Breed-Pop-Trait combinations you have submitted will be now displayed with status=Submitted.

On the day of the data submission deadline for a given IGE, routine or test run, all your data in the Review page should have status=SUBMITTED. Your aim is therefore to check all pending data and either provide explanations or withdraw and upload new datasets before the data submission deadline.'''  '''

''' {{attachment:submit_action.png}}  '''

==== Withdraw data ====
If you realize the data you have uploaded for some Breed-Pop-trait combinations is wrong or you want to upload pedigree information for the animals reported in the confirmation email attachment, you need to withdraw your data before doing anything else. You do so by simply selecting the affected Breed-Pop-Trait combinations in the Review page, select the action Withdraw and then press the button "Submit". In the Review page that given Breed-Pop-trait combination will now have status=WITHDRAWN.

When you withdraw a given Breed-Pop-Trait all records present in IDEA for that combination are deleted thus you have to re upload the file before the data deadline. It is not acceptable to submit a partial  dataset in order to correct the evaluations of some subset of bulls.  It is essential that proof records for all bulls be included in the  same file for any one Breed-Pop-Trait combination whether the dataset is a  first submission for the IGE run in question, or a re-submission with  some problem corrected.

If you fail to withdraw all the traits you are going to re-upload in a new file, the confirmation email will contain the following warning:

 . WARNING: This dataset re-uploads already existing data, of which some has not been withdrawn first. These combinations have been skipped; the proper way to re-upload combinations is to withdraw them first (or ask IBC staff to reset them). These combinations have been skipped:[......]

Example: <<BR>>You notice some problem with the adjusted weaning weight data for LIM. You withdraw only the LIM-aww from your Review table but you upload a new file in IDEA also for LIM cae. As you did not withdraw the LIM-cae combination before uploading the new file, all the combinations referring to LIM-cae are skipped as data already exist in IDEA for that given combination.'''  '''

==== Accepted/Rejected ====
Once you have submitted your data for a given IGE it will be up to the ITBC staff to finally accept or reject it. In general all data with FLAGGED=NO will also get accepted as there are no issues pending on these data. For all the data with FLAGGED=YES, ITBC staff will go through the explanations you have reported and if found sufficients will mark that given Breed-Pop-Trait combination as accepted. If more clarifications are needed ITBC staff will email you via IDEA, anytime a new message will be posted in your IDEA account by the staff you will receive an email informing about the presence of unreplied messages in IDEA.

In case the discrepancies found are considered too big then ITBC staff can reject your data and use the data from the previous run. You will be able to follow what happens to your data by looking at the Review page as the status will change according to what decision has been made.

=== Messages ===
Beef/Messages represents the place where all your communication with the ITBC for a given IGE are displayed. All the explanations you provide for a given Breed-Pop-Trait combination are automatically listed in here. Unreplied messages will be marked in red. You can use this page to send further messages to ITBC staff. When ITBC staff replies, their messages will also be listed here and you will be notified by email about the presence of unreplied messages in IDEA.

''' {{attachment:message.png}} '''

=== Message Overview ===
'''I'''n the message overview you can review all the communications you had with the staff for any given evaluation.

By selecting the evaluation you are interested in and pressing "Set Run", all messages exchanged between you and the ITBC staff during the evaluation you have selected will be listed in a table.'''  '''

=== Performance Query ===
The query gives you the possibility to review the performance information you have provided for a given animal, you can choose if display the information related to a specific run, comparing the information provided in two runs or review the information related to all runs available in IDEA. You can filter your query per breed, traitgroup or individual traits. You can only lookup information your own organization has uploaded. For the query to work an international animal ID must be supplied.

=== Parameter Query ===
This query gives you the possibility to review the parameter information you have provided for a given run, breed, trait group or individual traits. You can choose one particular run or even compared the information provided between two runs.

You will be able to review only the information your own organization has uploaded.

== APPENDIX III - Frequently Asked Questions ==
=== My data has been rejected, what can i do? ===
National  data can sometimes be rejected from ITBC staff if changes if problems in the data are found. If your  data get rejected it means that the previous available data will be  used for the IGE. In case you are able to fix the problem(s), and the  data deadline has not yet passed, you can try to upload a new corrected  dataset. You will need to check the confirmation email, the verify  output and submit the data in the Review page again.'''  '''

=== I discovered a problem with a dataset with FLAG=NO, what do I do? ===
As long as status=PENDING you can always upload a new corrected dataset, the new upload will overwrite the previous one. If you discover the error after submitting the data, and if the data deadline has not passed yet, you need to inform the ITBC staff about the problem and the breed-pop-trait combination  affected by it. ITBC  staff will reset your data so that you will be able to upload a new  file again. You will need to check the confirmation email, the verify  output and submit the data in the Review page again.