wip: first draft of a unit test to reproduce this type of issue: #439

simpers · 2024-12-08T22:37:49Z

WIP of trying to reproduce an issue with many_to_many filters expressions trying to reference a parent's has_many relationship, the join-relationship.

Repository with the issue: SecretSanta Ash OpenSource

Contributor checklist

Bug fixes include regression tests
Features include unit/acceptance tests

1) test Thing.create! create works without associating other things (ThingTest) test/thing_test.exs:20 ** (Ash.Error.Unknown) Bread Crumbs: > Exception raised in: Things.get_by_id Unknown Error * ** (RuntimeError) Error while building reference: all_other_things.associated_at (ash_sql 0.2.38) lib/expr.ex:1796: AshSql.Expr.default_dynamic_expr/6 (ash_sql 0.2.38) lib/expr.ex:88: AshSql.Expr.default_dynamic_expr/6

zachdaniel · 2024-12-09T03:04:28Z

Please address the credo & format errors 😁

simpers · 2024-12-09T07:35:00Z

Please address the credo & format errors 😁

Of course! I just thought that wasn't important until after we know if it will be merged at all 😅

zachdaniel · 2024-12-09T11:35:37Z

Right right, it wasn't important just reflex sorry 🤣

zachdaniel · 2024-12-09T17:57:46Z

Going to merge this and see if I can tinker with it. I've found a fix for half of the issue, specifically using these types of relationships in exists/2, but there is still a problem loading it.

zachdaniel · 2024-12-09T19:18:41Z

Actually maybe I fixed it all? Can you please give main of ash_postgres and ash_sql a shot?

simpers · 2024-12-10T15:22:29Z

Actually maybe I fixed it all? Can you please give main of ash_postgres and ash_sql a shot?

$ mix test --stale test/secret_santa/groups/group_test.exs:46

Running the above I get this very verbose error now, when running the unit tests with both ash_sql and ash_postgres pointing to main:

1) test Groups.create! create works without inviting (SecretSanta.Groups.GroupTest)
     test/secret_santa/groups/group_test.exs:20
     ** (Ash.Error.Unknown)
     Bread Crumbs:
       > Exception raised in: SecretSanta.Users.UserProfile.read
       > Exception raised in: SecretSanta.Groups.Group.get_by_id

     Unknown Error

     * ** (Ecto.SubQueryError) the following exception happened when compiling a subquery.

         ** (Ecto.SubQueryError) the following exception happened when compiling a subquery.

             ** (Ecto.SubQueryError) the following exception happened when compiling a subquery.

                 ** (Ecto.QueryError) could not find named binding `as(1)` in query:

                 from u0 in SecretSanta.Users.UserProfile,
                   as: 0,
                   where: not is_nil(
                   type(
                     as(1).accepted_at,
                     {:parameterized, {Ash.Type.UtcDatetimeUsec.EctoType, precision: :microsecond, timezone: :utc}}
                   )
                 ),
                   select: %{
                   id: u0.id,
                   inserted_at: u0.inserted_at,
                   updated_at: u0.updated_at,
                   deleted_at: u0.deleted_at,
                   name: u0.name,
                   account_id: u0.account_id
                 }


             The subquery originated from the following query:

             from g0 in SecretSanta.Groups.Group,
               as: 1,
               join: u1 in SecretSanta.Groups.UserGroup,
               on: as(1).id == u1.group_id,
               join: u2 in subquery(from u0 in SecretSanta.Users.UserProfile,
               as: 0,
               where: not is_nil(
               type(
                 as(1).accepted_at,
                 {:parameterized, {Ash.Type.UtcDatetimeUsec.EctoType, precision: :microsecond, timezone: :utc}}
               )
             ),
               select: %{
               id: u0.id,
               inserted_at: u0.inserted_at,
               updated_at: u0.updated_at,
               deleted_at: u0.deleted_at,
               name: u0.name,
               account_id: u0.account_id
             }),
               on: u2.id == u1.user_id,
               join: a3 in SecretSanta.Accounts.Account,
               on: as(4).account_id == a3.id,
               join: u4 in SecretSanta.Groups.UserGroup,
               on: u4.group_id == g0.id and parent_as(0).id == u4.user_id,
               where: type(as(5).id, {:parameterized, {SecretSanta.NanoId.EctoType, []}}) ==
               type(^"gCRgJ7ZY", {:parameterized, {SecretSanta.NanoId.EctoType, []}}),
               select: %{result: 1}


         The subquery originated from the following query:

         from u0 in SecretSanta.Users.UserProfile,
           as: 0,
           join: u1 in SecretSanta.Groups.UserGroup,
           on: u1.user_id == u0.id,
           where: exists(
           subquery(
             #Ecto.Query<from g0 in SecretSanta.Groups.Group, as: 1, join: u1 in SecretSanta.Groups.UserGroup, on: as(1).id == u1.group_id, join: u2 in subquery(from u0 in SecretSanta.Users.UserProfile,
           as: 0,
           where: not is_nil(
           type(
             as(1).accepted_at,
             {:parameterized, {Ash.Type.UtcDatetimeUsec.EctoType, precision: :microsecond, timezone: :utc}}
           )
         ),
           select: %{
           id: u0.id,
           inserted_at: u0.inserted_at,
           updated_at: u0.updated_at,
           deleted_at: u0.deleted_at,
           name: u0.name,
           account_id: u0.account_id
         }), on: u2.id == u1.user_id, join: a3 in SecretSanta.Accounts.Account, on: as(4).account_id == a3.id, join: u4 in SecretSanta.Groups.UserGroup, on: u4.group_id == g0.id and parent_as(0).id == u4.user_id, where: type(as(5).id, {:parameterized, {SecretSanta.NanoId.EctoType, []}}) ==
           type(^"gCRgJ7ZY", {:parameterized, {SecretSanta.NanoId.EctoType, []}}), select: %{result: 1}>
           )
         ) or
           exists(
             subquery(
               #Ecto.Query<from g0 in SecretSanta.Groups.Group, as: 1, join: u1 in SecretSanta.Users.UserProfile, on: as(1).lead_santa_id == u1.id, join: a2 in SecretSanta.Accounts.Account, on: as(3).account_id == a2.id, join: u3 in SecretSanta.Groups.UserGroup, on: u3.group_id == g0.id and parent_as(0).id == u3.user_id, where: type(as(4).id, {:parameterized, {SecretSanta.NanoId.EctoType, []}}) ==
           type(^"gCRgJ7ZY", {:parameterized, {SecretSanta.NanoId.EctoType, []}}), select: %{result: 1}>
             )
           ),
           where: not is_nil(
           type(
             as(1).accepted_at,
             {:parameterized, {Ash.Type.UtcDatetimeUsec.EctoType, precision: :microsecond, timezone: :utc}}
           )
         ),
           where: u1.group_id == parent_as(0).id,
           select: %{
           id: u0.id,
           name: u0.name,
           account_id: u0.account_id,
           updated_at: u0.updated_at,
           deleted_at: u0.deleted_at,
           inserted_at: u0.inserted_at
         }


     The subquery originated from the following query:

     from g0 in SecretSanta.Groups.Group,
       as: 0,
       inner_lateral_join: u1 in subquery(from u0 in SecretSanta.Users.UserProfile,
       as: 0,
       join: u1 in SecretSanta.Groups.UserGroup,
       on: u1.user_id == u0.id,
       where: exists(
       subquery(
         #Ecto.Query<from g0 in SecretSanta.Groups.Group, as: 1, join: u1 in SecretSanta.Groups.UserGroup, on: as(1).id == u1.group_id, join: u2 in subquery(from u0 in SecretSanta.Users.UserProfile,
       as: 0,
       where: not is_nil(
       type(
         as(1).accepted_at,
         {:parameterized, {Ash.Type.UtcDatetimeUsec.EctoType, precision: :microsecond, timezone: :utc}}
       )
     ),
       select: %{
       id: u0.id,
       inserted_at: u0.inserted_at,
       updated_at: u0.updated_at,
       deleted_at: u0.deleted_at,
       name: u0.name,
       account_id: u0.account_id
     }), on: u2.id == u1.user_id, join: a3 in SecretSanta.Accounts.Account, on: as(4).account_id == a3.id, join: u4 in SecretSanta.Groups.UserGroup, on: u4.group_id == g0.id and parent_as(0).id == u4.user_id, where: type(as(5).id, {:parameterized, {SecretSanta.NanoId.EctoType, []}}) ==
       type(^"gCRgJ7ZY", {:parameterized, {SecretSanta.NanoId.EctoType, []}}), select: %{result: 1}>
       )
     ) or
       exists(
         subquery(
           #Ecto.Query<from g0 in SecretSanta.Groups.Group, as: 1, join: u1 in SecretSanta.Users.UserProfile, on: as(1).lead_santa_id == u1.id, join: a2 in SecretSanta.Accounts.Account, on: as(3).account_id == a2.id, join: u3 in SecretSanta.Groups.UserGroup, on: u3.group_id == g0.id and parent_as(0).id == u3.user_id, where: type(as(4).id, {:parameterized, {SecretSanta.NanoId.EctoType, []}}) ==
       type(^"gCRgJ7ZY", {:parameterized, {SecretSanta.NanoId.EctoType, []}}), select: %{result: 1}>
         )
       ),
       where: not is_nil(
       type(
         as(1).accepted_at,
         {:parameterized, {Ash.Type.UtcDatetimeUsec.EctoType, precision: :microsecond, timezone: :utc}}
       )
     ),
       where: u1.group_id == parent_as(0).id,
       select: %{
       id: u0.id,
       name: u0.name,
       account_id: u0.account_id,
       updated_at: u0.updated_at,
       deleted_at: u0.deleted_at,
       inserted_at: u0.inserted_at
     }),
       on: true,
       where: g0.id in ^["ruqHb728"],
       distinct: true,
       select: merge(u1, %{__lateral_join_source__: map(g0, [:id])})

       (elixir 1.17.3) lib/enum.ex:1829: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
       (elixir 1.17.3) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
       (ecto 3.12.5) lib/ecto/repo/queryable.ex:214: Ecto.Repo.Queryable.execute/4
       (ecto 3.12.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
       (ash_postgres 2.4.15) lib/data_layer.ex:994: AshPostgres.DataLayer.run_query_with_lateral_join/4
       (ash 3.4.44) lib/ash/actions/read/read.ex:2754: Ash.Actions.Read.run_query/4
       (ash 3.4.44) lib/ash/actions/read/read.ex:594: anonymous fn/8 in Ash.Actions.Read.do_read/5
       (ash 3.4.44) lib/ash/actions/read/read.ex:938: Ash.Actions.Read.maybe_in_transaction/3
       (ash 3.4.44) lib/ash/actions/read/read.ex:319: Ash.Actions.Read.do_run/3
       (ash 3.4.44) lib/ash/actions/read/read.ex:82: anonymous fn/3 in Ash.Actions.Read.run/3
       (ash 3.4.44) lib/ash/actions/read/read.ex:81: Ash.Actions.Read.run/3
       (ash 3.4.44) lib/ash/actions/read/relationships.ex:437: anonymous fn/2 in Ash.Actions.Read.Relationships.do_fetch_related_records/4
       (ash 3.4.44) lib/ash/actions/read/relationships.ex:65: Ash.Actions.Read.Relationships.fetch_related_records/4
     stacktrace:
       (elixir 1.17.3) lib/enum.ex:1829: Enum."-map_reduce/3-lists^mapfoldl/2-0-"/3
       (elixir 1.17.3) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
       (ecto 3.12.5) lib/ecto/repo/queryable.ex:214: Ecto.Repo.Queryable.execute/4
       (ecto 3.12.5) lib/ecto/repo/queryable.ex:19: Ecto.Repo.Queryable.all/3
       (ash_postgres 2.4.15) lib/data_layer.ex:994: AshPostgres.DataLayer.run_query_with_lateral_join/4
       (ash 3.4.44) lib/ash/actions/read/read.ex:2754: Ash.Actions.Read.run_query/4
       (ash 3.4.44) lib/ash/actions/read/read.ex:594: anonymous fn/8 in Ash.Actions.Read.do_read/5
       (ash 3.4.44) lib/ash/actions/read/read.ex:938: Ash.Actions.Read.maybe_in_transaction/3
       (ash 3.4.44) lib/ash/actions/read/read.ex:319: Ash.Actions.Read.do_run/3
       (ash 3.4.44) lib/ash/actions/read/read.ex:82: anonymous fn/3 in Ash.Actions.Read.run/3
       (ash 3.4.44) lib/ash/actions/read/read.ex:81: Ash.Actions.Read.run/3
       (ash 3.4.44) lib/ash/actions/read/relationships.ex:437: anonymous fn/2 in Ash.Actions.Read.Relationships.do_fetch_related_records/4
       (ash 3.4.44) lib/ash/actions/read/relationships.ex:65: Ash.Actions.Read.Relationships.fetch_related_records/4

zachdaniel · 2024-12-10T18:14:44Z

Mind giving ash_sql latest main a try?

simpers · 2024-12-12T08:48:16Z

Mind giving ash_sql latest main a try?

Tests seem to pass now! So I'll just write go over them, and write some more tests, to verify that my tests are valid too! But now it compiles and seems to run ✅

simpers · 2024-12-16T11:24:49Z

Unfortunately, it still persists :thinkies:

Commits:

AshPostgres:d836fa80deb19e748d7f743fbde4435c9820f49e
AshSql: 5cd1d14f79895dace2276377a6b56bb37946a430

I had to fix some other issues are renaming things back and forth when trying to fix this initially, and now I'm back to the same error:

10) test Groups.create! cannot read if not lead_santa and not guest (SecretSanta.Groups.GroupTest)
     test/secret_santa/groups/group_test.exs:173
     ** (Ash.Error.Unknown)
     Bread Crumbs:
       > Exception raised in: SecretSanta.Groups.Group.read
       > Exception raised in: SecretSanta.Groups.Group.create

     Unknown Error

     * ** (RuntimeError) Error while building reference: all_users.accepted_at
       (ash_sql 0.2.41) lib/expr.ex:1805: AshSql.Expr.default_dynamic_expr/6
       (ash_sql 0.2.41) lib/expr.ex:88: AshSql.Expr.default_dynamic_expr/6
       (ash_sql 0.2.41) lib/expr.ex:75: AshSql.Expr.default_dynamic_expr/6
       (ash_sql 0.2.41) lib/filter.ex:37: anonymous fn/2 in AshSql.Filter.add_filter_expression/2
       (elixir 1.17.3) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
       (ash_postgres 2.4.16) lib/data_layer.ex:3014: AshPostgres.DataLayer.filter/4
       (ash 3.4.46) lib/ash/query/query.ex:3119: Ash.Query.maybe_filter/3
       (ash 3.4.46) lib/ash/query/query.ex:2996: Ash.Query.data_layer_query/2
       (ash_sql 0.2.41) lib/join.ex:379: AshSql.Join.related_query/3
       (ash_sql 0.2.41) lib/join.ex:274: AshSql.Join.related_subquery/3
       (ash_sql 0.2.41) lib/aggregate.ex:171: anonymous fn/7 in AshSql.Aggregate.add_aggregates/6
       (elixir 1.17.3) lib/enum.ex:4858: Enumerable.List.reduce/3
       (elixir 1.17.3) lib/enum.ex:2585: Enum.reduce_while/3
       (ash_sql 0.2.41) lib/aggregate.ex:88: AshSql.Aggregate.add_aggregates/6
       (ash 3.4.46) lib/ash/query/query.ex:2991: Ash.Query.data_layer_query/2
       (ash 3.4.46) lib/ash/actions/read/read.ex:1104: Ash.Actions.Read.reselect_and_load/6
       (ash 3.4.46) lib/ash/actions/read/read.ex:310: Ash.Actions.Read.do_run/3
       (ash 3.4.46) lib/ash/actions/read/read.ex:82: anonymous fn/3 in Ash.Actions.Read.run/3
       (ash 3.4.46) lib/ash/actions/read/read.ex:81: Ash.Actions.Read.run/3
       (ash 3.4.46) lib/ash.ex:1835: Ash.load/3

I might try to disable the policies and reintroduce them one-by-one to see.

zachdaniel · 2024-12-16T12:54:34Z

Is this reproduced in the repo you shared? I can take a look.

simpers · 2024-12-16T13:56:47Z

Is this reproduced in the repo you shared? I can take a look.

Should be at the moment, yes:

mix test test/secret_santa/groups/group_test.exs

All tests fail with the same thing, since I added change load: [:all_users, :participants, ...] to the create action.

Otherwise it failed upon the next read with said loads.

zachdaniel · 2024-12-16T14:30:56Z

Alright, I've pushed something up to ash_sql main, we were missing a similar fix for certain kinds of aggregates. You still have other tests failing, though, so if those are related to some kind of incorrect query from ash_postgres you will have to lay that out in a separate issue.

One thing also to keep in mind is that if worse comes to worse you can always use a manual relationship as laid out here: https://hexdocs.pm/ash_postgres/manual-relationships.html

simpers · 2024-12-16T15:11:54Z

Alright, I've pushed something up to ash_sql main, we were missing a similar fix for certain kinds of aggregates. You still have other tests failing, though, so if those are related to some kind of incorrect query from ash_postgres you will have to lay that out in a separate issue.

One thing also to keep in mind is that if worse comes to worse you can always use a manual relationship as laid out here: https://hexdocs.pm/ash_postgres/manual-relationships.html

I don't know how valid my tests are other than that none could run due to the dynamic_expr issue 😅

But if that is fixed, I'll be able to move on and validate the rest of them. As far as I know, this is the only issue I had after trying to implement the filters. And I broke more of them by refactoring as part of figuring that out hah

fix: ran formatter and fixed credo warnings

d934ef4

zachdaniel merged commit f8daf7f into ash-project:main Dec 9, 2024
51 checks passed

simpers deleted the maybe-bug/m2m-parent-filter branch December 10, 2024 15:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wip: first draft of a unit test to reproduce this type of issue: #439

wip: first draft of a unit test to reproduce this type of issue: #439

simpers commented Dec 8, 2024 •

edited

Loading

zachdaniel commented Dec 9, 2024

simpers commented Dec 9, 2024

zachdaniel commented Dec 9, 2024

zachdaniel commented Dec 9, 2024

zachdaniel commented Dec 9, 2024

simpers commented Dec 10, 2024

zachdaniel commented Dec 10, 2024

simpers commented Dec 12, 2024

simpers commented Dec 16, 2024

zachdaniel commented Dec 16, 2024

simpers commented Dec 16, 2024

zachdaniel commented Dec 16, 2024

simpers commented Dec 16, 2024

wip: first draft of a unit test to reproduce this type of issue: #439

wip: first draft of a unit test to reproduce this type of issue: #439

Conversation

simpers commented Dec 8, 2024 • edited Loading

Contributor checklist

zachdaniel commented Dec 9, 2024

simpers commented Dec 9, 2024

zachdaniel commented Dec 9, 2024

zachdaniel commented Dec 9, 2024

zachdaniel commented Dec 9, 2024

simpers commented Dec 10, 2024

zachdaniel commented Dec 10, 2024

simpers commented Dec 12, 2024

simpers commented Dec 16, 2024

zachdaniel commented Dec 16, 2024

simpers commented Dec 16, 2024

zachdaniel commented Dec 16, 2024

simpers commented Dec 16, 2024

simpers commented Dec 8, 2024 •

edited

Loading