IDC ENH: 2 servers

[Punzo]

@pieper @fedorov This is a first attempt to implement fetching from multiple servers.

The user case is 2 servers only in reading. The first with the source data, the second with annotation. Specifically the first is supposed to be a publlic server (e.g. dev-proxy.canceridc.dev), the second a google store.

This has been tested over this datasets https://viewer.imaging.datacommons.cancer.gov/viewer/1.3.6.1.4.1.14519.5.2.1.6655.2359.247426265538388028079845922798 (only source data)

against the dataset in this google store /projects/idc-tcia/locations/us-central1/datasets/tcia-idc-datareviewcoordination/dicomStores/planar_annotations/study/1.3.6.1.4.1.14519.5.2.1.6655.2359.247426265538388028079845922798 (source data + annotation)

Currently the logic is to fetch the data from both servers, merge the results (for series found twice, it ignores the ones coming from the second server).

the first server is specified in the config file and the config parameters for enabling the GoogleCloudAdapter as well. i.e.:

https://github.com/Punzo/Viewers/blob/IDC2servers/platform/viewer/public/config/default.js#L18-L51

Once this is setup, one can open the study from two servers as:

http://localhost:3000/viewer/1.3.6.1.4.1.14519.5.2.1.6655.2359.247426265538388028079845922798!secondGoogleServer=/projects/idc-tcia/locations/us-central1/datasets/tcia-idc-datareviewcoordination/dicomStores/planar_annotations

Here a video:

2022-05-12.18-26-33.mp4

Two issues:

1. Since we need to fetch all the metadata from the two servers and then operating filtering (e.g. removing double series), we can not start updating the UI until all the metadata have been fetched (i.e. this results in a small delay of the UI in showing the thumbnails)
2. We need to set the config parameters for enabling the GoogleCloudAdapter in the configuration file. This means taht if we try to load only from the first server (i.e. like, http://localhost:3000/viewer/1.3.6.1.4.1.14519.5.2.1.6655.2359.247426265538388028079845922798), still the user will be asked to log in google.

Before merging I would test this in the @pieper sandbox. Steve please let me know if you can deploy thiis branch https://github.com/ImagingDataCommons/Viewers/tree/IDC2servers against testing-viewer.canceridc.dev + googleAdapter

[Punzo]

Here tested against segmentation:

http://localhost:3000/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.899353684702041035700102438716!secondGoogleServer=/projects/idc-tcia/locations/us-central1/datasets/lidc-idri-seg-sr-dcm/dicomStores/lidc-idri-seg-sr-dcm

note that both IDC server and google store have both source data and segmentation.

So I tested using http://localhost:3000/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.899353684702041035700102438716!secondGoogleServer=/projects/idc-tcia/locations/us-central1/datasets/lidc-idri-seg-sr-dcm/dicomStores/lidc-idri-seg-sr-dcm?seriesInstanceUID=1.3.6.1.4.1.14519.5.2.1.6279.6001.188059920088313909273628445208

where the filter ?seriesInstanceUID= applies only to the IDC server (while I fetch all from the google store)

[Punzo]

1. while everything (source data, segmentation, etc..) will work with N servers, the parsing of SR annotation (SCOORD), still works only with one server (I set it to be automatically the first google server found, if there is no google server, it will use the first server. This is a big assumption). To work SR annotation with N server this would need a gigantic real major rework on how the 2D annotation/measurements work in OHIF-v2.
2. I made the option ?seriesInstanceUID= in a way that will not apply to the google servers. So if you have the IDC portal source data + segmentation and in the google store server the segmentation (e.g. http://localhost:3000/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.89935368470204103570[…].6.1.4.1.14519.5.2.1.6279.6001.188059920088313909273628445208) . You can fetch from both of them, but select only the source data from the IDC portal.

[Punzo]

Reporting here some discussion happened on slack for not losing it:

I am hesitant to put this directly in OHIF-v2 master branch for the following reasons:

1. I had to change the fetching for merging studies (with the same studyID) from different servers into 1. This means that we have to fetch all the metadata before the UI can be responsive (i.e. load the thumbnails, etc...). It is a small delay, but it changes a lot all the fetching logic and in the case of only 1 server it is not worth.
2. while everything (source data, segmentation, etc..) will work with N servers, the parsing of SR annotation (SCOORD), still works only with one server (I set it to be automatically the first google server found, if there is no google server, it will use the first server. This is a big assumption). To work SR annotation with N server this would need a gigantic real major rework on how the 2D annotation/measurements work in OHIF-v2.
3. I made the option ?seriesInstanceUID= in a way that will not apply to the google servers. So if you have the IDC portal source data + segmentation and in the google store server the segmentation (e.g. http://localhost:3000/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.89935368470204103570[…].6.1.4.1.14519.5.2.1.6279.6001.188059920088313909273628445208) . You can fetch from both of them, but select only the source data from the IDC portal.

for all these reasons I feel this should live only in a IDC branch (not even IDC master). My understating was that this should never go in production, wasn't it?

If we don't want to diverge and put this back to IDC/OHIF master fork, it would be possible but some more refactoring would be needed (for example if it is only one server, OHIF should bounce back to the old fetching behavior) and I would pass this also for Igor review. But this would imply some other days of dev (further refactoring/Igor review/etc...).

I am super open to discuss with you guys more this point. In general, I think this 'fast' approach was a good starting point to experiment with multiple servers. We can test this experiment on sandbox and then next week we can discuss the next step (further refactoring/merging into master IDC or OHIF forks).

[Punzo]

Update to OHIF v2.14.26 and added iin the thumbnail the origin server

i.e.:
https://idc2serversdeploy-3c769.web.app/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.899353684702041035700102438716!secondGoogleServer=/projects/idc-tcia/locations/us-central1/datasets/lidc-idri-seg-sr-dcm/dicomStores/lidc-idri-seg-sr-dcm

https://idc2serversdeploy-3c769.web.app/viewer/1.3.6.1.4.1.14519.5.2.1.6279.6001.899353684702041035700102438716!secondGoogleServer=/projects/idc-tcia/locations/us-central1/datasets/lidc-idri-seg-sr-dcm/dicomStores/lidc-idri-seg-sr-dcm?seriesInstanceUID=1.3.6.1.4.1.14519.5.2.1.6279.6001.188059920088313909273628445208,1.2.276.0.7230010.3.1.3.0.10555.1553293857.414765

NOTE: the logic is that if there are duplicate series during the merge, it will take the series from the first server (the test proxy).
However if you add ?seriesInstanceUID=1.3.6.1.4.1.14519.5.2.1.6279.6001.188059920088313909273628445208,1.2.276.0.7230010.3.1.3.0.10555.1553293857.414765 , it will filter from the first server (the Test proxy) to have only those selected series (so there are no duplicate anymore for the rest of the series)

[igoroctaviano]

@fedorov @pieper are we going to move forward with this implementation or we should archive/close this PR?

[pieper]

I think this is still an important issue, but not for v2. I think we should focus any new features on v3. What do you think @fedorov

[fedorov]

We are using this feature all the time, and in the past this branch of IDC fork was kept up to date with OHIF v2 master. Indeed, it was never the goal to add this feature to v2 proper. It would be great if the 2 server fork could be synced periodically, and we should probably retarget this issue for v3 as "idc:candidate".