Using WhiteRabbit with Spark, Databricks (Delta Live tables) or Snowflake

[solmazeradat]

Hi,

Hope you are well.

We are looking at building a pipeline where the data volume/size is of the order of terabits. We want to ensure both the source data as well as the CDM data is compatible with analytical tools for big data as well as the OHDSI analytical tool kit.

Since the scanReport from WhiteRabbit tool is integral to the mapping process, wanted to check if WhiteRabbit supports the use of any of the following:

Spark
Databricks (Delta Live tables)
Snowflake

Many thanks,
Solmaz

[janblom]

Hi,

of the three data platforms you mention, support for Snowflake is currently being developed (some testing is being done as well). For the other two platforms WhiteRabbit currently has no (planned) support. Keep an eye on this issue, I will update it when there is a (test-) release available here.

Given the data volume you mention, you might also want to verify if this issue could be an obstacle for you. Work on this is planned to happen soon (after the Snowflake support).

If all goes a bit as planned, I hope to have a release with both issues solved before the end of 2023.

Best regards,
Jan

[solmazeradat]

Hi @janblom,

Thanks very much for the response. Will keep an eye on the proposed releases for Snowflake.

Solmaz

[janblom]

Release candidate 1 for WhiteRabbit 1.0.0, with Snowflake support, has just been made available: https://github.com/OHDSI/WhiteRabbit/releases/tag/v1.0.0-RC1

If you have an opportunity to test, any feedback will be highly appreciated.

Jan