Bug fixes

0.10.5

🐛 Fixes

• Fix bug in fastapi-pagination #73
• Remove unnecessary Optional in List[Optional[T]] in return value for QuerySet.all() and Querysetproxy.all() return values #174
• Run tests coverage publish only on internal prs instead of all in github action.

Python style filter and order_by with field chain access

0.10.4

✨ Features

• Add Python style to filter and order_by with field access instead of dunder separated strings. #51
  • Accessing a field with attribute access (chain of dot notation) can be used to construct FilterGroups (ormar.and_ and ormar.or_)
  • Field access overloads set of python operators and provide a set of functions to allow same functionality as with dunder separated param names in **kwargs, that means that querying from sample model Track related to model Album now you have more options:
    • exact - exact match to value, sql column = <VALUE>
      • OLD: album__name__exact='Malibu'
      • NEW: can be also written as Track.album.name == 'Malibu
    • iexact - exact match sql column = <VALUE> (case insensitive)
      • OLD: album__name__iexact='malibu'
      • NEW: can be also written as Track.album.name.iexact('malibu')
    • contains - sql column LIKE '%<VALUE>%'
      • OLD: album__name__contains='Mal'
      • NEW: can be also written as Track.album.name % 'Mal')
      • NEW: can be also written as Track.album.name.contains('Mal')
    • icontains - sql column LIKE '%<VALUE>%' (case insensitive)
      • OLD: album__name__icontains='mal'
      • NEW: can be also written as Track.album.name.icontains('mal')
    • in - sql column IN (<VALUE1>, <VALUE2>, ...)
      • OLD: album__name__in=['Malibu', 'Barclay']
      • NEW: can be also written as Track.album.name << ['Malibu', 'Barclay']
      • NEW: can be also written as Track.album.name.in_(['Malibu', 'Barclay'])
    • isnull - sql column IS NULL (and sql column IS NOT NULL)
      • OLD: album__name__isnull=True (isnotnull album__name__isnull=False)
      • NEW: can be also written as Track.album.name >> None
      • NEW: can be also written as Track.album.name.is_null(True)
      • NEW: not null can be also written as Track.album.name.is_null(False)
      • NEW: not null can be also written as ~(Track.album.name >> None)
      • NEW: not null can be also written as ~(Track.album.name.is_null(True))
    • gt - sql column > <VALUE> (greater than)
      • OLD: position__gt=3
      • NEW: can be also written as Track.album.name > 3
    • gte - sql column >= <VALUE> (greater or equal than)
      • OLD: position__gte=3
      • NEW: can be also written as Track.album.name >= 3
    • lt - sql column < <VALUE> (lower than)
      • OLD: position__lt=3
      • NEW: can be also written as Track.album.name < 3
    • lte - sql column <= <VALUE> (lower equal than)
      • OLD: position__lte=3
      • NEW: can be also written as Track.album.name <= 3
    • startswith - sql column LIKE '<VALUE>%' (exact start match)
      • OLD: album__name__startswith='Mal'
      • NEW: can be also written as Track.album.name.startswith('Mal')
    • istartswith - sql column LIKE '<VALUE>%' (case insensitive)
      • OLD: album__name__istartswith='mal'
      • NEW: can be also written as Track.album.name.istartswith('mal')
    • endswith - sql column LIKE '%<VALUE>' (exact end match)
      • OLD: album__name__endswith='ibu'
      • NEW: can be also written as Track.album.name.endswith('ibu')
    • iendswith - sql column LIKE '%<VALUE>' (case insensitive)
      • OLD: album__name__iendswith='IBU'
      • NEW: can be also written as Track.album.name.iendswith('IBU')
• You can provide FilterGroups not only in filter() and exclude() but also in:
  • get()
  • get_or_none()
  • get_or_create()
  • first()
  • all()
  • delete()
• With FilterGroups (ormar.and_ and ormar.or_) you can now use:
  • & - as and_ instead of next level of nesting
  • | - as `or_' instead of next level of nesting
  • ~ - as negation of the filter group
• To combine groups of filters into one set of conditions use & (sql AND) and | (sql OR)

  # Following queries are equivalent:
  # sql: ( product.name = 'Test'  AND  product.rating >= 3.0 ) 
  
  # ormar OPTION 1 - OLD one
  Product.objects.filter(name='Test', rating__gte=3.0).get()
  
  # ormar OPTION 2 - OLD one
  Product.objects.filter(ormar.and_(name='Test', rating__gte=3.0)).get()
  
  # ormar OPTION 3 - NEW one (field access)
  Product.objects.filter((Product.name == 'Test') & (Product.rating >=3.0)).get()

• Same applies to nested complicated filters

  # Following queries are equivalent:
  # sql: ( product.name = 'Test' AND product.rating >= 3.0 ) 
  #       OR (categories.name IN ('Toys', 'Books'))
  
  # ormar OPTION 1 - OLD one
  Product.objects.filter(ormar.or_(
                            ormar.and_(name='Test', rating__gte=3.0), 
                            categories__name__in=['Toys', 'Books'])
                        ).get()
  
  # ormar OPTION 2 - NEW one (instead of nested or use `|`)
  Product.objects.filter(
                        ormar.and_(name='Test', rating__gte=3.0) | 
                        ormar.and_(categories__name__in=['Toys', 'Books'])
                        ).get()
  
  # ormar OPTION 3 - NEW one (field access)
  Product.objects.filter(
                        ((Product.name='Test') & (Product.rating >= 3.0)) | 
                        (Product.categories.name << ['Toys', 'Books'])
                        ).get()

• Now you can also use field access to provide OrderActions to order_by()
  • Order ascending:
    • OLD: Product.objects.order_by("name").all()
    • NEW: Product.objects.order_by(Product.name.asc()).all()
  • Order descending:
    • OLD: Product.objects.order_by("-name").all()
    • NEW: Product.objects.order_by(Product.name.desc()).all()
  • You can of course also combine different models and many order_bys:
    Product.objects.order_by([Product.category.name.asc(), Product.name.desc()]).all()

🐛 Fixes

• Not really a bug but rather inconsistency. Providing a filter with nested model i.e. album__category__name = 'AA'
  is checking if album and category models are included in select_related() and if not it's auto-adding them there.
  The same functionality was not working for FilterGroups (and_ and or_), now it works (also for python style filters which return FilterGroups).

One sided relations and more powerful save_related

0.10.3

✨ Features

• ForeignKey and ManyToMany now support skip_reverse: bool = False flag #118.
  If you set skip_reverse flag internally the field is still registered on the other
  side of the relationship so you can:

  • filter by related models fields from reverse model
  • order_by by related models fields from reverse model

  But you cannot:

  • access the related field from reverse model with related_name
  • even if you select_related from reverse side of the model the returned models won't be populated in reversed instance (the join is not prevented so you still can filter and order_by)
  • the relation won't be populated in dict() and json()
  • you cannot pass the nested related objects when populating from dict() or json() (also through fastapi). It will be either ignored or raise error depending on extra setting in pydantic Config.

• Model.save_related() now can save whole data tree in once #148
  meaning:

  • it knows if it should save main Model or related Model first to preserve the relation

  • it saves main Model if

    • it's not saved,
    • has no pk value
    • or save_all=True flag is set

    in those cases you don't have to split save into two calls (save() and save_related())

  • it supports also ManyToMany relations

  • it supports also optional Through model values for m2m relations

• Add possibility to customize Through model relation field names.

• By default Through model relation names default to related model name in lowercase.
  So in example like this:

  ... # course declaration ommited
  class Student(ormar.Model):
      class Meta:
          database = database
          metadata = metadata
  
      id: int = ormar.Integer(primary_key=True)
      name: str = ormar.String(max_length=100)
      courses = ormar.ManyToMany(Course)
  
  # will produce default Through model like follows (example simplified)
  class StudentCourse(ormar.Model):
      class Meta:
          database = database
          metadata = metadata
          tablename = "students_courses"
  
      id: int = ormar.Integer(primary_key=True)
      student = ormar.ForeignKey(Student) # default name
      course = ormar.ForeignKey(Course)  # default name

• To customize the names of fields/relation in Through model now you can use new parameters to ManyToMany:

  • through_relation_name - name of the field leading to the model in which ManyToMany is declared
  • through_reverse_relation_name - name of the field leading to the model to which ManyToMany leads to

  Example:

  ... # course declaration ommited
  class Student(ormar.Model):
      class Meta:
          database = database
          metadata = metadata
  
      id: int = ormar.Integer(primary_key=True)
      name: str = ormar.String(max_length=100)
      courses = ormar.ManyToMany(Course,
                                 through_relation_name="student_id",
                                 through_reverse_relation_name="course_id")
  
  # will produce default Through model like follows (example simplified)
  class StudentCourse(ormar.Model):
      class Meta:
          database = database
          metadata = metadata
          tablename = "students_courses"
  
      id: int = ormar.Integer(primary_key=True)
      student_id = ormar.ForeignKey(Student) # set by through_relation_name
      course_id = ormar.ForeignKey(Course)  # set by through_reverse_relation_name

Optimization & bug fixes, additional parameters in update methods

0.10.2

✨ Features

• Model.save_related(follow=False) now accept also two additional arguments: Model.save_related(follow=False, save_all=False, exclude=None).
  • save_all:bool -> By default (so with save_all=False) ormar only upserts models that are not saved (so new or updated ones),
    with save_all=True all related models are saved, regardless of saved status, which might be useful if updated
    models comes from api call, so are not changed in the backend.
  • exclude: Union[Set, Dict, None] -> set/dict of relations to exclude from save, those relation won't be saved even with follow=True and save_all=True.
    To exclude nested relations pass a nested dictionary like: exclude={"child":{"sub_child": {"exclude_sub_child_realtion"}}}. The allowed values follow
    the fields/exclude_fields (from QuerySet) methods schema so when in doubt you can refer to docs in queries -> selecting subset of fields -> fields.
• Model.update() method now accepts _columns: List[str] = None parameter, that accepts list of column names to update. If passed only those columns will be updated in database.
  Note that update() does not refresh the instance of the Model, so if you change more columns than you pass in _columns list your Model instance will have different values than the database!
• Model.dict() method previously included only directly related models or nested models if they were not nullable and not virtual,
  now all related models not previously visited without loops are included in dict(). This should be not breaking
  as just more data will be dumped to dict, but it should not be missing.
• QuerySet.delete(each=False, **kwargs) previously required that you either pass a filter (by **kwargs or as a separate filter() call) or set each=True now also accepts
  exclude() calls that generates NOT filter. So either each=True needs to be set to delete whole table or at least one of filter/exclude clauses.
• Same thing applies to QuerySet.update(each=False, **kwargs) which also previously required that you either pass a filter (by **kwargs or as a separate filter() call) or set each=True now also accepts
  exclude() calls that generates NOT filter. So either each=True needs to be set to update whole table or at least one of filter/exclude clauses.
• Same thing applies to QuerysetProxy.update(each=False, **kwargs) which also previously required that you either pass a filter (by **kwargs or as a separate filter() call) or set each=True now also accepts
  exclude() calls that generates NOT filter. So either each=True needs to be set to update whole table or at least one of filter/exclude clauses.

🐛 Fixes

• Fix improper relation field resolution in QuerysetProxy if fk column has different database alias.
• Fix hitting recursion error with very complicated models structure with loops when calling dict().
• Fix bug when two non-relation fields were merged (appended) in query result when they were not relation fields (i.e. JSON)
• Fix bug when during translation to dict from list the same relation name is used in chain but leads to different models
• Fix bug when bulk_create would try to save also property_field decorated methods and pydantic fields
• Fix wrong merging of deeply nested chain of reversed relations

💬 Other

• Performance optimizations
• Split tests into packages based on tested area

Add get_or_none, fix quoting sql keyword names in order_by queries

0.10.1

Features

• add get_or_none(**kwargs) method to QuerySet and QuerysetProxy. It is exact equivalent of get(**kwargs) but instead of raising ormar.NoMatch exception if there is no db record matching the criteria, get_or_none simply returns None.

Fixes

• Fix dialect dependent quoting of column and table names in order_by clauses not working
  properly in postgres.

Partial typing fix, add select_all, change model fields to instances

0.10.0

Breaking

• Dropped supported for long deprecated notation of field definition in which you use ormar fields as type hints i.e. test_field: ormar.Integger() = None
• Improved type hints -> mypy can properly resolve related models fields (ForeignKey and ManyToMany) as well as return types of QuerySet methods.
  Those mentioned are now returning proper model (i.e. Book) instead or ormar.Model type. There is still problem with reverse sides of relation and QuerysetProxy methods,
  to ease type hints now those return Any. Partially fixes #112.

Features

• add select_all(follow: bool = False) method to QuerySet and QuerysetProxy.
  It is kind of equivalent of the Model's load_all() method but can be used directly in a query.
  By default select_all() adds only directly related models, with follow=True also related models
  of related models are added without loops in relations. Note that it's not and end async model
  so you still have to issue get(), all() etc. as select_all() returns a QuerySet (or proxy)
  like fields() or order_by(). #131

Internals

• ormar fields are no longer stored as classes in Meta.model_fields dictionary
  but instead they are stored as instances.

Add default ordering, new aggr functions, new signals

0.9.9

Features

• Add possibility to change default ordering of relations and models.
  • To change model sorting pass orders_by = [columns] where columns: List[str] to model Meta class
  • To change relation order_by pass orders_by = [columns] where columns: List[str]
  • To change reverse relation order_by pass related_orders_by = [columns] where columns: List[str]
  • Arguments can be column names or -{col_name} to sort descending
  • In relations you can sort only by directly related model columns
    or for ManyToMany columns also Through model columns "{through_field_name}__{column_name}"
  • Order in which order_by clauses are applied is as follows:
    • Explicitly passed order_by() calls in query
    • Relation passed orders_by if exists
    • Model Meta class orders_by
    • Model primary key column asc (fallback, used if none of above provided)
• Add 4 new aggregated functions -> min, max, sum and avg that are their
  corresponding sql equivalents.
  • You can pass one or many column names including related columns.
  • As of now each column passed is aggregated separately (so sum(col1+col2) is not possible,
    you can have sum(col1, col2) and later add 2 returned sums in python)
  • You cannot sum and avg non numeric columns
  • If you aggregate on one column, the single value is directly returned as a result
  • If you aggregate on multiple columns a dictionary with column: result pairs is returned
• Add 4 new signals -> pre_relation_add, post_relation_add, pre_relation_remove and post_relation_remove
  • The newly added signals are emitted for ManyToMany relations (both sides)
    and reverse side of ForeignKey relation (same as QuerysetProxy is exposed).
  • Signals recieve following args: sender: Type[Model] - sender class,
    instance: Model - instance to which related model is added, child: Model - model being added,
    relation_name: str - name of the relation to which child is added,
    for add signals also passed_kwargs: Dict - dict of kwargs passed to add()

Changes

• Through models for ManyToMany relations are now instantiated on creation, deletion and update, so you can provide not only
  autoincrement int as a primary key but any column type with default function provided.
• Since Through models are now instantiated you can also subscribe to Through model
  pre/post save/update/delete signals
• pre_update signals receivers now get also passed_args argument which is a
  dict of values passed to update function if any (else empty dict)

Fixes

• pre_update signal now is sent before the extraction of values so you can modify the passed
  instance in place and modified fields values will be reflected in database
• bulk_update now works correctly also with UUID primary key column type

Fields encryption

0.9.8

Features

• Add possibility to encrypt the selected field(s) in the database
  • As minimum you need to provide encrypt_secret and encrypt_backend
  • encrypt_backend can be one of the ormar.EncryptBackends enum (NONE, FERNET, HASH, CUSTOM) - default: NONE
  • When custom backend is selected you need to provide your backend class that subclasses ormar.fields.EncryptBackend
  • You cannot encrypt primary_key column and relation columns (FK and M2M).
  • Provided are 2 backends: HASH and FERNET
    • HASH is a one-way hash (like for password), never decrypted on retrieval
    • FERNET is a two-way encrypt/decrypt backend
  • Note that in FERNET backend you loose filtering possibility altogether as part of the encrypted value is a timestamp.
  • Note that in HASH backend you can filter by full value but filters like contain will not work as comparison is make on encrypted values
  • Note that adding encrypt_backend changes the database column type to TEXT, which needs to be reflected in db either by migration or manual change

Fixes

• (Advanced/ Internal) Restore custom sqlalchemy types (by types.TypeDecorator subclass) functionality that ceased to working so process_result_value was never called

Isnull filter and complex filters (including or)

0.9.7

Features

• Add isnull operator to filter and exclude methods.

  album__name__isnull=True #(sql: album.name is null)
  album__name__isnull=False #(sql: album.name is not null))

• Add ormar.or_ and ormar.and_ functions that can be used to compose
  complex queries with nested conditions.
  Sample query:

  books = (
      await Book.objects.select_related("author")
      .filter(
          ormar.and_(
              ormar.or_(year__gt=1960, year__lt=1940),
              author__name="J.R.R. Tolkien",
          )
      )
      .all()
  )

  Check the updated docs in Queries -> Filtering and sorting -> Complex filters

Other

• Setting default on ForeignKey or ManyToMany raises and ModelDefinition exception as it is (and was) not supported

Add through fields, load_all() method and make through models optional

0.9.6

Important

• Through model for ManyToMany relations now becomes optional. It's not a breaking change
  since if you provide it everything works just fine as it used to. So if you don't want or need any additional
  fields on Through model you can skip it. Note that it's going to be created for you automatically and
  still has to be included in example in alembic migrations.
  If you want to delete existing one check the default naming convention to adjust your existing database structure.

  Note that you still need to provide it if you want to
  customize the Through model name or the database table name.

Features

• Add update method to QuerysetProxy so now it's possible to update related models directly from parent model
  in ManyToMany relations and in reverse ForeignKey relations. Note that update like in QuerySet update returns number of
  updated models and does not update related models in place on parent model. To get the refreshed data on parent model you need to refresh
  the related models (i.e. await model_instance.related.all())
• Add load_all(follow=False, exclude=None) model method that allows to load current instance of the model
  with all related models in one call. By default it loads only directly related models but setting
  follow=True causes traversing the tree (avoiding loops). You can also pass exclude parameter
  that works the same as QuerySet.exclude_fields() method.
• Added possibility to add more fields on Through model for ManyToMany relationships:
  • name of the through model field is the lowercase name of the Through class
  • you can pass additional fields when calling add(child, **kwargs) on relation (on QuerysetProxy)
  • you can pass additional fields when calling create(**kwargs) on relation (on QuerysetProxy)
    when one of the keyword arguments should be the through model name with a dict of values
  • you can order by on through model fields
  • you can filter on through model fields
  • you can include and exclude fields on through models
  • through models are attached only to related models (i.e. if you query from A to B -> only on B)
  • note that through models are explicitly loaded without relations -> relation is already populated in ManyToMany field.
  • note that just like before you cannot declare the relation fields on through model, they will be populated for you by ormar,
    but now if you try to do so ModelDefinitionError will be thrown
  • check the updated ManyToMany relation docs for more information

Other

• Updated docs and api docs
• Refactors and optimisations mainly related to filters, exclusions and order bys