Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data entry redux #5671

Merged
merged 4 commits into from
Oct 10, 2024
Merged

Data entry redux #5671

merged 4 commits into from
Oct 10, 2024

Conversation

JorisGoosen
Copy link
Contributor

Speed up the filtered data entry model massively by not making gigantic jsons for everything.

This will add "named filters" to jasp, which can be linked to jaspControls.
We could later extend on this functionality for analyses etc.

It also will make the filtered date entry model and r-side directly interface with the dataset and thus force the component to always be linked to a computed column.

@koenderks
Copy link
Contributor

Don't forget the classical workflow (I see now that you adjusted only the Bayesian workflow?)

@JorisGoosen
Copy link
Contributor Author

Its a draft and non-functional yet, I just opened it to have an easy way to see all the changes.

I still need to integrate the computed column into the dataentry thing all the way. Hopefully It'll be done next week

@JorisGoosen
Copy link
Contributor Author

Only jaspAudit has some changes related to this PR

@JorisGoosen
Copy link
Contributor Author

@koenderks @boutinb this is ready for review

for koen: I think it still works, maybe you can confirm :p
Should be much quicker for large datasets alround now.

@koenderks
Copy link
Contributor

koenderks commented Oct 9, 2024

I'm currently trying this but I get:

image

I don't think this has to do with your changes, right? It is only in the dev module, not the non-dev audit. I can only test this once I can access the workflow, so I'll wait a bit.

@JorisGoosen
Copy link
Contributor Author

You can build development right @koenderks ?
If you checkout this jasp-desktop branch (dataEntryRedux) and run git submodule update it ought to have the changes in jaspAudit as well. No need for development modules if you can build jasp.

This error of yours looks like you might need to clean up the buildfolder a bit?
(Unless you can build and run development with audit because then there is some other problem)

@koenderks
Copy link
Contributor

Nice, it works fine. However, for a brief moment (long enough to make a screenshot however) you get this visual error:

image

Furthermore, I notice that the selected column is immediately made and visible but the auditResult column is not?

image

Works fast with dataset of 100000 rows!

@JorisGoosen
Copy link
Contributor Author

Nice, it works fine. However, for a brief moment (long enough to make a screenshot however) you get this visual error:

This is caused by the way Audit right now sets the filter-code and "auditResult" columnname only when you click "Fill Table". Only then is auditResult created. But selected isnt filled yet, so the filter returns nothing... So the error/warning.
Then on such an error I simply let it run again.
Now selected is filled and thus the next filter-return it shows stuff in the table.

This could be avoided by filling selected earlier.

Also, the initial values of the component are taking from bookValue but because they should only be set on the selected rows I have to wait until the filter returns to know which rows. So:

  • Fill table, creates auditResults and sets filter to something like "selected > 0"
  • Filter runs, but selected is empty so gets nothing. Mentions this to user.
  • analysiscode runs, fills selected but gets no values from performAudit.
  • filter runs again because of the earlier not-getting-anything
  • now table knows which rows it should show.
  • first time it gets a usable filter it takes "initialValues" and sets it to the rows
  • runs analysiscode again and now it gets all the info

Furthermore, I notice that the selected column is immediately made and visible but the auditResult column is not?

Yes, the computed column is now created by the FilteredDataEntry thing. If you want it created earlier you should set colName earlier.

Works fast with dataset of 100000 rows!

Right? :D

For  https://github.com/jasp-stats/jasp-test-release/issues/2515

Allow for filters to be created by JASPControls etc

this should help speeding up jasp quite a bit for large datasets in analyses that use filtering in their gui.
So, Audit

the filters didnt get a dataset id by me and it seems they should.
for tomorrow

Ok DataSet is now provided to a component

probably works now, but Audit needs to be slightly rewritten to get the info from the filter directly
and also, how does it get the data from the filter?

dit dan?

make it compile and use test module again

postmerge fixes

change some lambdas

use testmodule with special case for filtered data entry

Dont allow translaters to write R code for the filter

link computed column creation through to listmodelfiltereddataentry

commit jaspAudit with some mods in dataEntryRedux branch

add submodules

Use dataEntryRedux branches of jaspBase, jaspTools and jaspAudit

fix some bugs with filter by name and error propagation

more

show filterwarning if some problem and actually rerun filter on datasetchanges

showing and setting data now works

initial values are used as, shocking, initial values

dont ad a queue full of identical requests, not the prettiest solution...

now the column gets created by jasp...

now it inits correctly

add meta for loading the filtered data

its called 'loadFilteredData', an object with 'filter' and 'column' as fields

recursive update of options with regards to stuff in meta added, can be used later for more things

should allow for loading from a jaspfile?

loading from jasp file seems to work

make sure to properly load non-conflicting filters by name

use updated vewrsion jaspAudit

use master of jaspBase and jaspTools

return of the "show analysis" button

add special field for checking whether a column is free or owned my analysis

Avoid crashing
Copy link
Contributor

@boutinb boutinb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just added a commit to set the connections in the right object.

@JorisGoosen JorisGoosen merged commit 0618cfd into development Oct 10, 2024
1 check failed
@JorisGoosen JorisGoosen deleted the dataEntryRedux branch October 10, 2024 12:52
RensDofferhoff pushed a commit to RensDofferhoff/jasp-desktop that referenced this pull request Oct 15, 2024
* Filtered data entry Redux

For  https://github.com/jasp-stats/jasp-test-release/issues/2515

Allow for filters to be created by JASPControls etc

this should help speeding up jasp quite a bit for large datasets in analyses that use filtering in their gui.
So, Audit

the filters didnt get a dataset id by me and it seems they should.
for tomorrow

Ok DataSet is now provided to a component

probably works now, but Audit needs to be slightly rewritten to get the info from the filter directly
and also, how does it get the data from the filter?

dit dan?

make it compile and use test module again

postmerge fixes

change some lambdas

use testmodule with special case for filtered data entry

Dont allow translaters to write R code for the filter

link computed column creation through to listmodelfiltereddataentry

commit jaspAudit with some mods in dataEntryRedux branch

add submodules

Use dataEntryRedux branches of jaspBase, jaspTools and jaspAudit

fix some bugs with filter by name and error propagation

more

show filterwarning if some problem and actually rerun filter on datasetchanges

showing and setting data now works

initial values are used as, shocking, initial values

dont ad a queue full of identical requests, not the prettiest solution...

now the column gets created by jasp...

now it inits correctly

add meta for loading the filtered data

its called 'loadFilteredData', an object with 'filter' and 'column' as fields

recursive update of options with regards to stuff in meta added, can be used later for more things

should allow for loading from a jaspfile?

loading from jasp file seems to work

make sure to properly load non-conflicting filters by name

use updated vewrsion jaspAudit

use master of jaspBase and jaspTools

return of the "show analysis" button

add special field for checking whether a column is free or owned my analysis

Avoid crashing

* take into account how enabled the tableview is

* use audit without enable disabled

* Set connections in TableView not in Bound object
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants