remove ordering by sys.id

[mattseddon]

Related to #477 and https://github.com/iterative/studio/issues/10635#issuecomment-2381829809 & https://github.com/iterative/studio/issues/10635#issuecomment-2406225017

All the context for this change is in #477 but the tl;dr is:

This behaviour is currently undocumented, untested and misunderstood. There is no way to support this behaviour with BigQuery and we have to find another way around datasets appearing ordered. One suggestion is to save the order along with the dataset information in the metastore (perhaps save the appropriate select statement as this would be extendable in the future).

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.19%. Comparing base (fce1dc3) to head (6947d0a).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #507      +/-   ##
==========================================
+ Coverage   87.15%   87.19%   +0.03%     
==========================================
  Files          92       92              
  Lines        9834     9824      -10     
  Branches     1348     1344       -4     
==========================================
- Hits         8571     8566       -5     
+ Misses        910      907       -3     
+ Partials      353      351       -2     

Flag	Coverage Δ
datachain	87.16% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cloudflare-workers-and-pages]

Deploying datachain-documentation with    Cloudflare Pages

Latest commit:	6947d0a
Status:	✅  Deploy successful!
Preview URL:	https://3d9e9d13.datachain-documentation.pages.dev
Branch Preview URL:	https://check-ordering.datachain-documentation.pages.dev

View logs

[ilongin]

We had a lot of discussions about this through the history of dvcx / datachain. Basically it boils down to this:

1. Preserving order by making sure inserted rows are in correct order is not how things are usually done. That's why we don't have this feature in BigQuery and probably in other DBs. Usually DB don't allow you to "mess" into how they internally store rows.
2. If you want to fetch rows in some order, just use order by in query....as you said, we can save ordering in some metadata and use it when doing query.

Why we still end up with 1) ? ... because 2) is not really working well with billions of rows and column oriented DBs like Clickhouse and BigQuery. Having to do order by on each select means whole DB must be reshuffled which in Clickhouse also leads to OOM errors if I remember correctly (we need to double check this though). Ordering and column databases don't play well with each other as those DBs are imagined to just go through bunch of rows as fast as possible and calculate something (by using small amount of columns as well), and ordering is not really in their use case ...

Anyway, I'm ok to approve this if product agrees, as this means we are removing feature which was requested explicitly by product some time ago.

[mattseddon]

Preserving order by making sure inserted rows are in correct order is not how things are usually done. That's why we don't have this feature in BigQuery and probably in other DBs. Usually DB don't allow you to "mess" into how they internally store rows.

A small note: Even if we insert rows in the correct order in BQ, there is no way to guarantee the order they are returned when we pull them out. We cannot generate a sequential ID in the same way we do for SQLite and ClickHouse AND ordering is not guaranteed unless an order by statement is provided.

[dreadatour]

Finally! Can’t wait for this to be merged; it’ll make things simpler from the code cleanliness perspective and faster in terms of performance.

Might be controversial in terms of user experience, but on the other hand, it is kind of intuitive that you should not rely on default ordering when you’re dealing with big distributed data.

If anyone needs ordering, they can use order_by before using the related query.

[shcheklein]

Do we need sys__id at all after this decision (removing ordering)?

[mattseddon]

Do we need sys__id at all after this decision (removing ordering)?

It is used in distributed processing.