Skip to content

Commit

Permalink
README: update with latest screenshots and final mapping step
Browse files Browse the repository at this point in the history
  • Loading branch information
abhidg committed Nov 27, 2024
1 parent 2039ca9 commit aaa999f
Show file tree
Hide file tree
Showing 5 changed files with 36 additions and 15 deletions.
51 changes: 36 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
An utility to map data dictionaries (usually in REDCap format) to the [ARC
schema](https://github.com/ISARICResearch/DataPlatform).

> [!IMPORTANT]
> Mapping files are generated using AI and may contain errors. Always
> review them before using in production applications!
## Installation

We use [`uv`](https://docs.astral.sh/uv/getting-started/installation/)
Expand All @@ -28,27 +32,44 @@ see debug information, to do so run arcmapper with the `-d` or `--debug` flag.

The browser should open with the ARCMapper interface

![ARCmapper screenshot](images/arcmapper-screenshot.png)
![ARCmapper initial state](images/initial-state.png)

There are two steps:
There are three steps:

**Step 1**: Upload a data dictionary. This is usually in REDCap format; if
column names differ from REDCap standards, then make sure to fill in the
correct column names. Click *Upload*.
**Step 1**. *Upload a data dictionary*. This is usually in REDCap format; if
column names differ from REDCap standards, then make sure to fill in the correct
column names *before* uploading.

**Step 2**: Map to ARC. First, choose an ARC version and a mapping method.
**Step 2**: *Map to ARC*. First, choose an ARC version and a mapping method.
There are two mapping methods supported currently (i) TF-IDF, which uses text
frequency for similarity matching, and (ii) sentence transformers, which uses
semantic word representation based on training large text corpuses and the
transformers architecture.
transformers architecture. This will create an intermediate mapping which will
give you a few options (upto *Number of matches*) for each mapping from the
uploaded data dictionary to ARC. Choose the correct mappings by clicking on the
first cell in the row which will display a green check mark indicator ✅. These
mappings will be used in Step 3.

In addition to the maximum number of matches, there is a similarity *threshold*
setting; the higher the similarity threshold, the more closer a match needs to
be for ARCmapper to display in the matches table.

![ARCmapper intermediate mapping](images/intermediate-mapping.png)

At this point you can also save and load this intermediate mapping file, if you
want to continue work later. If you are loading an intermediate mapping file,
skip **Step 1**.

> [!NOTE]
> Using sentence transformers for the first time will incur a delay
> as models are downloaded from HuggingFace.
Note that using sentence transformers for the first time will incur a delay
as models are downloaded from HuggingFace.
**Step 3**: *Generate final mapping*. Once you are happy with the final mapping,
you can merge with the [ARC to FHIRflat
mapping](arc-fhir/ARC_pre_1.0.0_preset_dengue.xlsx) to create the mapping file
suitable for being processed by [FHIRflat](https://fhirflat.readthedocs.io).
This can be done by clicking the *Download FHIRflat mapping* button which will
produce an Excel file, with a sheet for each FHIR resource.

Additionally, you can also select the maximum number of matches to return and a
similarity threshold. The higher the similarity threshold, the more closer a
match needs to be for ARCmapper to display in the matches table.
![ARCmapper toolbar](images/mapping-toolbar.png)

Once you have pruned or altered the mapping table, click on **Download mapping
file** to download a mapping file that can be used by
[FHIRflat](https://fhirflat.readthedocs.io) for data transformations.
Binary file removed images/arcmapper-screenshot.png
Binary file not shown.
Binary file added images/initial-state.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/intermediate-mapping.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/mapping-toolbar.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit aaa999f

Please sign in to comment.