Dashboard

[alistairewj]

This dashboard is intended to help navigate the data once in a standard format.

There is some redundant code in it, and we should determine what we want to cache in order to keep the dashboard responsive, but it's a start!

[alistairewj]

@ibevers you can give this a review

• Added [bids_folder] as an argument, rather than defaulting it to output
• I added a CLI entrypoint for streamlit, can be run with b2aiprep-cli dashboard [bids_folder]
• I tidied up the pages a bit, and added caching on a few pages
• I fixed memory hogging issues with the semantic search, tidied up the page, made it use the published dictionary on GitHub rather than a local copy, and gave some instructions on its use

Feel free to review. I haven't tested it outside of my environment but it should work with python packaging since I used pkg_resources

[ibevers]

@alistairewj nice! I will take a look later today

[ibevers]

@alistairewj It appears that 'gender_identity' is not a column in the dataframe. Perhaps you could change this part so that only columns that are actually in the dataframe are accessible?

Or perhaps there is a discrepancy in our datasets for some reason?

[ibevers]

I'm also getting a certificate error

[alistairewj]

For the column error, it should definitely be gender_identity. Can you take a look at one of your sub/ses/audio/*_qgenericdemographicsschema.json and tell me what you see? For example I have this item in mine:

    {
      "linkId": "gender_identity",
      "answer": [
        {
          "valueString": "Female gender identity"
        }
      ]
    },

There are two "flavours" of redcap column names but they should be harmonized and consistently converted to those linkId names.

For the second error, SSL certificate failing sounds like it's something with your local setup. Maybe you configured a proxy? Or there is some forced HTTPS mode that I'm not using? I don't know why the local run of streamlit would require a certificate.

[ibevers]

Hmm, all my *_qgenericdemographicsschema.json files are at sub/ses/beh/*_qgenericdemographicsschema.json. I do have this in them:

    {
      "linkId": "gender_identity",
      "answer": [
        {
          "valueString": "Female gender identity"
        }
      ]
    }

Is the dashboard looking for them in the wrong place, or is my BIDS-like directory structure out-of-date? Also, it seems a bit strange to have *_qgenericdemographicsschema.json in audio/.

[alistairewj]

Sorry, typo there, the file is in beh/ not audio/. So your structure is correct but it's still raising a key error when it loads in the dataframe... hmm, not sure why that would be, but then it seems like it's not an issue with the dashboard but with load_and_pivot_questionnaire not providing the right columns, specifically this snippet:

https://github.com/sensein/b2aiprep/pull/55/files#diff-65a298442627b9a09e73c1801d20e9239eeb21390966b6a87b30132b0f76ef22R20-R22

Maybe try that outside of the dashboard and see what the dataframe returns? Or if you want to debug within the dashboard, you can add st.write(df) there to look at the dataframe before the key error.