You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is a possible optimization of cohort management that would be closely related to #229 (basically, more attempts at getting away from cohortInds getting superhuge and slow). Previews of downloads presently involve creating the download (which creates a table of structure not wholly dissimilar to that of a cohort) and then partially retrieving it; actually downloading involves getting all the contents of that table and marking it in diverDownloadTables as having been actually used.
Something similar could also be done for cohort creation as a possible optimization - in particular, it'd let us:
make creation after preview much faster (since we have the information already)
greatly cut down on cohortInds size and, along with Cache built kibblesets for later reuse #229, potentially eliminate it (as a very large table that gets large updates, it's a potential locking concern)
potentially allow for a cleaner separation between Application Structure and User Data (right now it's a little dubious)
Possible concerns:
adding a lot of tables might cause slowdowns in other areas
doing lots of previewing with no commitment will fill up the database (we can have ways around that tho - DROP TABLE can be pretty quick).
a little more proliferation of dynamic SQL
other stuff I may not have considered.
Still, something to noodle.
The text was updated successfully, but these errors were encountered:
This is a possible optimization of cohort management that would be closely related to #229 (basically, more attempts at getting away from cohortInds getting superhuge and slow). Previews of downloads presently involve creating the download (which creates a table of structure not wholly dissimilar to that of a cohort) and then partially retrieving it; actually downloading involves getting all the contents of that table and marking it in diverDownloadTables as having been actually used.
Something similar could also be done for cohort creation as a possible optimization - in particular, it'd let us:
Possible concerns:
Still, something to noodle.
The text was updated successfully, but these errors were encountered: