Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider reworking cohorts as "create table on preview, commit on save" the same way downloads currently are #249

Open
Viqsi opened this issue Aug 26, 2024 · 0 comments
Assignees
Labels
design issue Something to noodle before any action taken low priority
Milestone

Comments

@Viqsi
Copy link
Member

Viqsi commented Aug 26, 2024

This is a possible optimization of cohort management that would be closely related to #229 (basically, more attempts at getting away from cohortInds getting superhuge and slow). Previews of downloads presently involve creating the download (which creates a table of structure not wholly dissimilar to that of a cohort) and then partially retrieving it; actually downloading involves getting all the contents of that table and marking it in diverDownloadTables as having been actually used.

Something similar could also be done for cohort creation as a possible optimization - in particular, it'd let us:

  • make creation after preview much faster (since we have the information already)
  • greatly cut down on cohortInds size and, along with Cache built kibblesets for later reuse #229, potentially eliminate it (as a very large table that gets large updates, it's a potential locking concern)
  • potentially allow for a cleaner separation between Application Structure and User Data (right now it's a little dubious)

Possible concerns:

  • adding a lot of tables might cause slowdowns in other areas
  • doing lots of previewing with no commitment will fill up the database (we can have ways around that tho - DROP TABLE can be pretty quick).
  • a little more proliferation of dynamic SQL
  • other stuff I may not have considered.

Still, something to noodle.

@Viqsi Viqsi added low priority design issue Something to noodle before any action taken labels Aug 26, 2024
@Viqsi Viqsi added this to the Aspirational milestone Aug 26, 2024
@Viqsi Viqsi self-assigned this Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design issue Something to noodle before any action taken low priority
Projects
None yet
Development

No branches or pull requests

2 participants
@Viqsi and others