• FEATURES
  • PRICING
  • MARKETPLACE
  • CASE STUDIES
  • BLOG
    •

    Bulk load update of institutes and departments

    Technical Questions
    #1

    For OS 2.3 Is there a known issue with bulk update and API update/edit of departments? It doesn’t seem to be supported. API 404’s out for rest/ng/departments and bulk load for updating an institute doesn’t update or log anything. Additionally the jobs page shows nothing. (Competed jobs or jobs that went in to the black hole.)

    I haven’t turned up logging yet, but thought I would check if departments are editable via csv upload or the API. It doesn’t seem so.

    #2

    Also is there a reason why departments don’t sort alphabetically in the drop down list? I added them manually for an Institute and it’s sorted by some other id or perhaps how the where arranged in the array. I put them in alpha order and they switch to another order.

    #3

    Hello @rkmadsen

    1. Bulk import of institutes and departments: There is no known issue. I checked both create and update bulk import of institutes and both workflows are working fine. May I request you to attach sample CSV file/s that you tried uploading?

    2. Departments APIs: We’ve not published any API for this, as departments are simple value objects that will be managed through institute APIs

    3. Jobs not listed: Did UI app display success message like - Bulk import job xyz submitted or did it display any error message when you submitted the import job? The job appears in list only if there was a success message, which is an indication that the job was accepted and queued for execution

    4. Department ordering: This is bug. We’ll fix it to sort departments by their names. Thanks for reporting this issue.

    Thanks.

    #4

    I was looking in the wrong place for the completed jobs on the main menu jobs item. I didn’t see the individual jobs page for the institute/department upload. I can create a new institute and department but I can’t update. Which can be a little problematic. So for the department API there is a page on the wiki that described the department entity as available in the API:
    https://openspecimen.atlassian.net/wiki/display/CAT/Department

    So I was confused a bit.

    Here is the csv that failed with the failed message from the bulk import on the institute page.BulkImportJob_21.csv (219 Bytes)
    The institute exists, and is referenced correctly but the job fails without much error reporting. I’ll search through the logs again for what may be reported.

    -Randy

    #5

    Bulk import supports partial updates. However collections value specified in CSV is treated as snapshot value.

    Taking your example:

    Name,Department#1#Name,Department#2#Name
    Health Sciences Center,Biochemistry,CCTS-TTR

    This means “Health Sciences Center” institute will be updated to have only 2 departments Biochemistry, and CCTS-TTR whilst removing other departments that were present, if any, prior to import. In your case the departments that were removed are linked to other active entities like user etc. Therefore the error message - “One or more entities refer to department specified in requested operation”

    Hope this helps

    Thanks

    #6

    Thanks, for looking at it. I tried updating both with and without the existing department. They both fail with the same error: “one or more entities refer to department specified in requested operation”

    It seems that this is a bug. If the update processing of the Institutes and departments requires that no entities be linked then they really can’t ever be updated in practice only created. I created an institute and a department to attach the main admin user to the installation. I deleted the out of the box institute and user of admin@admin.com. From there I wanted to bulk load the rest of the department for the new institute.

    Is the use-case that bulk load update only be done when no users are attached to the institutes/departments? Just trying to clarify what the expected functionality is intended to be.

    -Randy

    #7

    Hello @rkmadsen

    This is subtle bug in bulk import institutes logic. We’ll fix this issue in v2.5

    Given below is ticket for tracking:
    https://openspecimen.atlassian.net/browse/OPSMN-2636

    Thanks for giving feedback.

    Thanks.