10.1101/2021.12.12.469460

[Hrovatin]

• Link to publication: https://doi.org/10.1101/2021.12.12.469460
• Title of publication: Tabula Microcebus: A transcriptomic cell atlas of mouse lemur, an emerging primate model organism
• First author of publication: The Tabula Microcebus Consortium
• DOI preprint: 10.1101/2021.12.12.469460
• DOI journal:

[Hrovatin]

I tried to finalise the dataloader, but I get below error. I have checked that assay_sc_obs_key is in obs.
Unfinished loader: https://github.com/theislab/sfaira/tree/data/d10_1101_2021_12_12_469460/sfaira/data/dataloaders/loaders/d10_1101_2021_12_12_469460
@davidsebfischer any ideas how to fix this?

╭────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/pandas/core/indexes/base.py:3621 in get_loc                                                               │
│                                                                                                                                                                                                │
│   3618 │   │   │   │   )                                                                                                                                                                       │
│   3619 │   │   │   casted_key = self._maybe_cast_indexer(key)                                                                                                                                  │
│   3620 │   │   │   try:                                                                                                                                                                        │
│ ❱ 3621 │   │   │   │   return self._engine.get_loc(casted_key)                                                                                                                                 │
│   3622 │   │   │   except KeyError as err:                                                                                                                                                     │
│   3623 │   │   │   │   raise KeyError(key) from err                                                                                                                                            │
│   3624 │   │   │   except TypeError:                                                                                                                                                           │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/index.pyx:136 in pandas._libs.index.IndexEngine.get_loc                                                             │
│                                                                                                                                                                                                │
│ [Errno 2] No such file or directory: '/mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/index.pyx'                                                                    │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/index.pyx:163 in pandas._libs.index.IndexEngine.get_loc                                                             │
│                                                                                                                                                                                                │
│ [Errno 2] No such file or directory: '/mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/index.pyx'                                                                    │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/hashtable_class_helper.pxi:5198 in pandas._libs.hashtable.PyObjectHashTable.get_item                                │
│                                                                                                                                                                                                │
│ [Errno 2] No such file or directory: '/mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/hashtable_class_helper.pxi'                                                   │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/hashtable_class_helper.pxi:5206 in pandas._libs.hashtable.PyObjectHashTable.get_item                                │
│                                                                                                                                                                                                │
│ [Errno 2] No such file or directory: '/mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/pandas/_libs/hashtable_class_helper.pxi'                                                   │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'assay_sc_original'

The above exception was the direct cause of the following exception:

╭────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/bin/sfaira:33 in <module>                                                                                                             │
│                                                                                                                                                                                                │
│   30                                                                                                                                                                                           │
│   31 if __name__ == '__main__':                                                                                                                                                                │
│   32 │   sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])                                                                                                                     │
│ ❱ 33 │   sys.exit(load_entry_point('sfaira', 'console_scripts', 'sfaira')())                                                                                                                   │
│   34                                                                                                                                                                                           │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/cli.py:74 in main                                                                                                         │
│                                                                                                                                                                                                │
│    71 │   # Is the latest sfaira version installed? Upgrade if not!                                                                                                                            │
│    72 │   if not UpgradeCommand.check_sfaira_latest():                                                                                                                                         │
│    73 │   │   print('[bold blue]Run [green]sfaira upgrade [blue]to get the latest version.')                                                                                                   │
│ ❱  74 │   sfaira_cli()                                                                                                                                                                         │
│    75                                                                                                                                                                                          │
│    76                                                                                                                                                                                          │
│    77 @click.group()                                                                                                                                                                           │
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/click/core.py:1128 in __call__                                                                            │
│                                                                                                                                                                                                │
│   1125 │                                                                                                                                                                                       │
│   1126 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                                                                                                                         │
│   1127 │   │   """Alias for :meth:`main`."""                                                                                                                                                   │
│ ❱ 1128 │   │   return self.main(*args, **kwargs)                                                                                                                                               │
│   1129                                                                                                                                                                                         │
│   1130                                                                                                                                                                                         │
│   1131 class Command(BaseCommand):                                                                                                                                                             │
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/click/core.py:1053 in main                                                                                │
│                                                                                                                                                                                                │
│   1050 │   │   try:                                                                                                                                                                            │
│   1051 │   │   │   try:                                                                                                                                                                        │
│   1052 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                                                                                                                │
│ ❱ 1053 │   │   │   │   │   rv = self.invoke(ctx)                                                                                                                                               │
│   1054 │   │   │   │   │   if not standalone_mode:                                                                                                                                             │
│   1055 │   │   │   │   │   │   return rv                                                                                                                                                       │
│   1056 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                                                                                                                             │
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/click/core.py:1659 in invoke                                                                              │
│                                                                                                                                                                                                │
│   1656 │   │   │   │   super().invoke(ctx)                                                                                                                                                     │
│   1657 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                                                                                                                  │
│   1658 │   │   │   │   with sub_ctx:                                                                                                                                                           │
│ ❱ 1659 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                             │
│   1660 │   │                                                                                                                                                                                   │
│   1661 │   │   # In chain mode we create the contexts step by step, but after the                                                                                                              │
│   1662 │   │   # base command has been invoked.  Because at that point we do not                                                                                                               │
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/click/core.py:1395 in invoke                                                                              │
│                                                                                                                                                                                                │
│   1392 │   │   │   echo(style(message, fg="red"), err=True)                                                                                                                                    │
│   1393 │   │                                                                                                                                                                                   │
│   1394 │   │   if self.callback is not None:                                                                                                                                                   │
│ ❱ 1395 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                                                                                                              │
│   1396 │                                                                                                                                                                                       │
│   1397 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:                                                                                                │
│   1398 │   │   """Return a list of completions for the incomplete value. Looks                                                                                                                 │
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/click/core.py:754 in invoke                                                                               │
│                                                                                                                                                                                                │
│    751 │   │                                                                                                                                                                                   │
│    752 │   │   with augment_usage_errors(__self):                                                                                                                                              │
│    753 │   │   │   with ctx:                                                                                                                                                                   │
│ ❱  754 │   │   │   │   return __callback(*args, **kwargs)                                                                                                                                      │
│    755 │                                                                                                                                                                                       │
│    756 │   def forward(                                                                                                                                                                        │
│    757 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902                                                                                                           │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/cli.py:228 in finalize_dataloader                                                                                         │
│                                                                                                                                                                                                │
│   225 │   Formats .tsvs and runs a full data loader test.                                                                                                                                      │
│   226 │   """                                                                                                                                                                                  │
│   227 │   path_loader, path_data, _ = set_paths(loader=path_loader, data=path_data)                                                                                                            │
│ ❱ 228 │   _full_test(path_loader=path_loader, path_data=path_data, doi=doi, schema=schema,                                                                                                     │
│       clean_tsvs=True, in_phase_3=True)                                                                                                                                                        │
│   229                                                                                                                                                                                          │
│   230                                                                                                                                                                                          │
│   231 @sfaira_cli.command()                                                                                                                                                                    │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/cli.py:188 in _full_test                                                                                                  │
│                                                                                                                                                                                                │
│   185 │   │   dataloader_validator = DataloaderValidator(path_loader=path_loader, doi=doi,                                                                                                     │
│       schema=schema)                                                                                                                                                                           │
│   186 │   │   dataloader_validator.validate()                                                                                                                                                  │
│   187 │   │   dataloader_tester = DataloaderTester(path_loader, path_data, doi)                                                                                                                │
│ ❱ 188 │   │   dataloader_tester.test_dataloader(clean_tsvs=clean_tsvs, in_phase_3=in_phase_3)                                                                                                  │
│   189 │   else:                                                                                                                                                                                │
│   190 │   │   print('[bold red]The supplied DOI is malformed!')  # noqa: W605                                                                                                                  │
│   191                                                                                                                                                                                          │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/commands/test_dataloader.py:34 in test_dataloader                                                                         │
│                                                                                                                                                                                                │
│    31 │   │   Runs a predefined unit test on a given dataloader.                                                                                                                               │
│    32 │   │   """                                                                                                                                                                              │
│    33 │   │   self.doi_sfaira_repr = clean_doi(self.doi)                                                                                                                                       │
│ ❱  34 │   │   self._test_dataloader(clean_tsvs=clean_tsvs, in_phase_3=in_phase_3)                                                                                                              │
│    35 │                                                                                                                                                                                        │
│    36 │   def _get_ds(self):                                                                                                                                                                   │
│    37 │   │   return get_ds(doi_sfaira_repr=self.doi_sfaira_repr, path_data=self.path_data,                                                                                                    │
│       path_loader=self.path_loader)                                                                                                                                                            │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/commands/test_dataloader.py:131 in _test_dataloader                                                                       │
│                                                                                                                                                                                                │
│   128 │   │   │   │   print(f'[bold red]From {signal_raw} total counts before streamlining,                                                                                                    │
│       {signal_proc} are left after.')                                                                                                                                                          │
│   129 │   │   │   │   print('[bold red]Consider revising feature meta data.')                                                                                                                  │
│   130 │   │   │   │   sys.exit()                                                                                                                                                               │
│ ❱ 131 │   │   │   v.streamline_metadata(schema="cellxgene")                                                                                                                                    │
│   132 │   │   print("[bold blue]Completed testing of data loader, the data loader is now ready                                                                                                 │
│       for use.")                                                                                                                                                                               │
│   133 │   │   if in_phase_3:                                                                                                                                                                   │
│   134 │   │   │   print('[bold orange]Sfaira butler: "You data loader is finished!"')                                                                                                          │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/data/dataloaders/base/dataset.py:972 in streamline_metadata                                                               │
│                                                                                                                                                                                                │
│    969 │   │   │   self.adata.obs.index = index_old                                                                                                                                            │
│    970 │   │   for k in [x for x in adata_target_ids.obs_keys if x in                                                                                                                          │
│        adata_target_ids.ontology_constrained]:                                                                                                                                                 │
│    971 │   │   │   # Add remaining output columns for ontology-constrained meta data.                                                                                                          │
│ ❱  972 │   │   │   self.__impute_ontology_cols_obs(attr=k, adata_ids=adata_target_ids)                                                                                                         │
│    973 │   │   │   # Delete attribute-specific columns that are not desired.                                                                                                                   │
│    974 │   │   │   col_name = getattr(self._adata_ids, k) + self._adata_ids.onto_id_suffix                                                                                                     │
│    975 │   │   │   if not keep_id_obs and col_name in self.adata.obs.columns:                                                                                                                  │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/data/dataloaders/base/dataset.py:1274 in __impute_ontology_cols_obs                                                       │
│                                                                                                                                                                                                │
│   1271 │   │   │   # Original annotation (free text):                                                                                                                                          │
│   1272 │   │   │   original_present = col_original in self.adata.obs.columns                                                                                                                   │
│   1273 │   │   │   if original_present and not symbol_present and not id_present:  # 1)                                                                                                        │
│ ❱ 1274 │   │   │   │   self.project_free_to_ontology(attr=attr)                                                                                                                                │
│   1275 │   │   │   if symbol_present or id_present:  # 2)                                                                                                                                      │
│   1276 │   │   │   │   if symbol_present and not id_present:  # 2a)                                                                                                                            │
│   1277 │   │   │   │   │   self.__project_ontology_ids_obs(attr=attr, from_id=False,                                                                                                           │
│        adata_ids=adata_ids)                                                                                                                                                                    │
│                                                                                                                                                                                                │
│ /mnt/znas/icb_zstore01/groups/ml01/code/karin.hrovatin/sfaira/sfaira/data/dataloaders/base/dataset.py:1179 in project_free_to_ontology                                                         │
│                                                                                                                                                                                                │
│   1176 │   │   ontology_map = self.ontology_class_maps[attr]                                                                                                                                   │
│   1177 │   │   adata_fields = self._adata_ids                                                                                                                                                  │
│   1178 │   │   col_original = attr + adata_fields.onto_original_suffix                                                                                                                         │
│ ❱ 1179 │   │   labels_original = self.adata.obs[col_original].values                                                                                                                           │
│   1180 │   │   if ontology_map is not None:  # only if this was defined                                                                                                                        │
│   1181 │   │   │   labels_mapped = [                                                                                                                                                           │
│   1182 │   │   │   │   ontology_map[x] if x in ontology_map.keys()                                                                                                                             │
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/pandas/core/frame.py:3505 in __getitem__                                                                  │
│                                                                                                                                                                                                │
│    3502 │   │   if is_single_key:                                                                                                                                                              │
│    3503 │   │   │   if self.columns.nlevels > 1:                                                                                                                                               │
│    3504 │   │   │   │   return self._getitem_multilevel(key)                                                                                                                                   │
│ ❱  3505 │   │   │   indexer = self.columns.get_loc(key)                                                                                                                                        │
│    3506 │   │   │   if is_integer(indexer):                                                                                                                                                    │
│    3507 │   │   │   │   indexer = [indexer]                                                                                                                                                    │
│    3508 │   │   else:                                                                                                                                                                          │
│                                                                                                                                                                                                │
│ /home/icb/karin.hrovatin/miniconda3/envs/sfaira_workshop/lib/python3.8/site-packages/pandas/core/indexes/base.py:3623 in get_loc                                                               │
│                                                                                                                                                                                                │
│   3620 │   │   │   try:                                                                                                                                                                        │
│   3621 │   │   │   │   return self._engine.get_loc(casted_key)                                                                                                                                 │
│   3622 │   │   │   except KeyError as err:                                                                                                                                                     │
│ ❱ 3623 │   │   │   │   raise KeyError(key) from err                                                                                                                                            │
│   3624 │   │   │   except TypeError:                                                                                                                                                           │
│   3625 │   │   │   │   # If we have a listlike key, _check_indexing_error will raise                                                                                                           │
│   3626 │   │   │   │   #  InvalidIndexError. Otherwise we fall through and re-raise                                                                                                            │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'assay_sc_original'

[davidsebfischer]

Looks like faulty streamlining, your loader is most likely fine already! Will look into it.