🚸 Catch Django errors of death

[chaichontat]

Except INSTALLED_APPS error and show an understandable error message

This happens after I deleted ~/.lamin. The actual actionable message is hidden under an incomprehensible exception.

Edit by @Zethson : This should be resolved in laminlabs/lamin-cli#103

[falexwolf]

Agreed - there are a few Django related tracebacks that we're handling downstream. It's a bit dangerous to default to hiding tracebacks in these cases, but you're right that in all cases where the original traceback is not needed, it's super annoying.

I'll weigh pros & cons or see whether there is a danger-free solution to this.

[falexwolf]

@Zethson shared feedback from a Hackathon

Two people tried to initialize instances and then work with bionty. Since they didn't add --schema they got a horrible django error and just gave up and were annoyed/frustrated. (I helped them then)

https://laminlabs.slack.com/archives/C054T673Q8J/p1728474032224189?thread_ts=1728279777.837979&cid=C054T673Q8J

We have to figure out a way to catch this error.

[falexwolf]

A new version of this: AppRegistryNotReady: Models aren't loaded yet. was raised when installing a schema module ourprojects after connecting to an instance.

Internal Slack ref.

---------------------------------------------------------------------------
AppRegistryNotReady                       Traceback (most recent call last)
Cell In[6], line 4
      1 # track the notebook
      2 #ln.settings.transform.stem_uid = "ePfXkzW8MP5B"
      3 #ln.settings.transform.version = "1"
----> 4 ln.track()
      5 # define directories
      6 basedir_use = "/data/global_swarm/sites/cambridge"

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/lamindb/core/_context.py:338, in Context.track(self, transform, params, new_run, path)
    335 run = None
    336 if not new_run:  # try loading latest run by same user
    337     run = (
--> 338         Run.filter(
    339             transform=self._transform, created_by_id=ln_setup.settings.user.id
    340         )
    341         .order_by("-created_at")
    342         .first()
    343     )
    344     if run is not None:  # loaded latest run
    345         run.started_at = datetime.now(timezone.utc)  # update run time

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/lamindb/_record.py:246, in filter(cls, *queries, **expressions)
    243 if "_using_key" in expressions:
    244     _using_key = expressions.pop("_using_key")
--> 246 return QuerySet(model=cls, using=_using_key).filter(*queries, **expressions)

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/lamindb/_query_set.py:545, in QuerySet.filter(self, *queries, **expressions)
    543 expressions = process_expressions(self, expressions)
    544 if len(expressions) > 0:
--> 545     return super().filter(*queries, **expressions)
    546 else:
    547     return self

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/query.py:1436, in QuerySet.filter(self, *args, **kwargs)
   1431 """
   1432 Return a new QuerySet instance with the args ANDed to the existing
   1433 set.
   1434 """
   1435 self._not_support_combined_queries("filter")
-> 1436 return self._filter_or_exclude(False, args, kwargs)

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/query.py:1454, in QuerySet._filter_or_exclude(self, negate, args, kwargs)
   1452     clone._deferred_filter = negate, args, kwargs
   1453 else:
-> 1454     clone._filter_or_exclude_inplace(negate, args, kwargs)
   1455 return clone

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/query.py:1461, in QuerySet._filter_or_exclude_inplace(self, negate, args, kwargs)
   1459     self._query.add_q(~Q(*args, **kwargs))
   1460 else:
-> 1461     self._query.add_q(Q(*args, **kwargs))

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/sql/query.py:1546, in Query.add_q(self, q_object)
   1537 # For join promotion this case is doing an AND for the added q_object
   1538 # and existing conditions. So, any existing inner join forces the join
   1539 # type to remain inner. Existing outer joins can however be demoted.
   1540 # (Consider case where rel_a is LOUTER and rel_a__col=1 is added - if
   1541 # rel_a doesn't produce any rows, then the whole condition must fail.
   1542 # So, demotion is OK.
   1543 existing_inner = {
   1544     a for a in self.alias_map if self.alias_map[a].join_type == INNER
   1545 }
-> 1546 clause, _ = self._add_q(q_object, self.used_aliases)
   1547 if clause:
   1548     self.where.add(clause, AND)

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/sql/query.py:1577, in Query._add_q(self, q_object, used_aliases, branch_negated, current_negated, allow_joins, split_subq, check_filterable, summarize)
   1573 joinpromoter = JoinPromoter(
   1574     q_object.connector, len(q_object.children), current_negated
   1575 )
   1576 for child in q_object.children:
-> 1577     child_clause, needed_inner = self.build_filter(
   1578         child,
   1579         can_reuse=used_aliases,
   1580         branch_negated=branch_negated,
   1581         current_negated=current_negated,
   1582         allow_joins=allow_joins,
   1583         split_subq=split_subq,
   1584         check_filterable=check_filterable,
   1585         summarize=summarize,
   1586     )
   1587     joinpromoter.add_votes(needed_inner)
   1588     if child_clause:

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/sql/query.py:1427, in Query.build_filter(self, filter_expr, branch_negated, current_negated, can_reuse, allow_joins, split_subq, reuse_with_filtered_relation, check_filterable, summarize)
   1425 if not arg:
   1426     raise FieldError("Cannot parse keyword query %r" % arg)
-> 1427 lookups, parts, reffed_expression = self.solve_lookup_type(arg, summarize)
   1429 if check_filterable:
   1430     self.check_filterable(reffed_expression)

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/sql/query.py:1237, in Query.solve_lookup_type(self, lookup, summarize)
   1235             expression = Ref(annotation, expression)
   1236         return expression_lookups, (), expression
-> 1237 _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
   1238 field_parts = lookup_splitted[0 : len(lookup_splitted) - len(lookup_parts)]
   1239 if len(lookup_parts) > 1 and not field_parts:

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/sql/query.py:1700, in Query.names_to_path(self, names, opts, allow_many, fail_on_missing)
   1695             field = opts.get_field(filtered_relation.relation_name)
   1696 if field is not None:
   1697     # Fields that contain one-to-many relations with a generic
   1698     # model (like a GenericForeignKey) cannot generate reverse
   1699     # relations and therefore cannot be used for reverse querying.
-> 1700     if field.is_relation and not field.related_model:
   1701         raise FieldError(
   1702             "Field %r does not generate an automatic reverse "
   1703             "relation and therefore cannot be used for reverse "
   1704             "querying. If it is a GenericForeignKey, consider "
   1705             "adding a GenericRelation." % name
   1706         )
   1707     try:

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/utils/functional.py:57, in cached_property.__get__(self, instance, cls)
     55 if instance is None:
     56     return self
---> 57 res = instance.__dict__[self.name] = self.func(instance)
     58 return res

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/db/models/fields/related.py:113, in RelatedField.related_model(self)
    110 @cached_property
    111 def related_model(self):
    112     # Can't cache this property until all the models are loaded.
--> 113     apps.check_models_ready()
    114     return self.remote_field.model

File ~/miniforge3/envs/model_dev/lib/python3.9/site-packages/django/apps/registry.py:143, in Apps.check_models_ready(self)
    141 """Raise an exception if all models haven't been imported yet."""
    142 if not self.models_ready:
--> 143     raise AppRegistryNotReady("Models aren't loaded yet.")

AppRegistryNotReady: Models aren't loaded yet.

[chaichontat]

Another one:

  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamin_cli/__main__.py", line 271, in load
    return connect(entity, _reload_lamindb=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamindb_setup/core/cloud_sqlite_locker.py", line 230, in wrapper
    raise exc
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamindb_setup/core/cloud_sqlite_locker.py", line 225, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamindb_setup/_connect_instance.py", line 322, in connect
    raise e
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamindb_setup/_connect_instance.py", line 278, in connect
    check, msg = isettings._load_db()
                 ^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamindb_setup/core/_settings_instance.py", line 503, in _load_db
    disable_auto_connect(setup_django)(self)
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamindb_setup/_check_setup.py", line 48, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lamindb_setup/core/django.py", line 97, in setup_django
    django.setup(set_prefix=False)
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/django/apps/registry.py", line 116, in populate
    app_config.import_models()
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/django/apps/config.py", line 269, in import_models
    self.models_module = import_module(models_module_name)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/clinicore/models.py", line 10, in <module>
    from lnschema_core.models import (
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lnschema_core/models.py", line 893, in <module>
    class User(Record, CanCurate):
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/lnschema_core/models.py", line 571, in __new__
    new_class = super().__new__(cls, name, bases, attrs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chaichontat/miniforge3/envs/lamin/lib/python3.11/site-packages/django/db/models/base.py", line 134, in __new__
    raise RuntimeError(
RuntimeError: Model class lnschema_core.models.User doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

[Zethson]

How did you get that one?

[chaichontat]

Just lamin load an instance. lamin-dev.
pip dump:

bionty                    1.1.0
lamin-cli                 1.1.0
lamin-utils               0.13.10
laminci                   0.14.0         /Users/chaichontat/GitHub/laminci
lamindb                   1.1.0
lamindb-setup             1.1.0
wetlab                    1.1.0

[falexwolf]

This is the issue with the v1 migration of all schema modules.

It can happen if you have ourprojects < 1 and didn't re-connect. Or you have clinicore < 1.

Nothing else tries to import lnschema_core, which is gone from the stack.

[chaichontat]

Can you print the list of schema modules I have to update in case there's an error like this? Or any unknown error from Django that is not caught.

I don't think there's a way for a user to know if all schema versions are synced or not. Just pulling from supabase would be a solution to this.

[falexwolf]

These topics are all taken care of above >1 through proper constraints in pyproject.toml which the package manager will log as warnings or errors once we go to lamindb >2

The removal of lnschema_core together with ourprojects was particularly disruptive -- that's why the migration had notes about this

Had we thought of v1 in November when making the last <1 releases it wouldn't have happened because all packages would already have had the >1 incompat constraint -- but I didn't think of making "v1" so early, so this bumpiness was foreseeable