group by showprogress too chatty

[ethanbsmith]

hate to nitpick on such a great package.

i'm seeing things like this: Processed 5332 groups out of 5332. 100% done. Time elapsed: 3s. ETA: 0s.

progress on a 3 second op is getting annoying. maybe up the threshold significantly (like at least 30 seconds) or have the threshold as a configurable option?

[joshhwuu]

The current threshold was made to be consistent with fread and fwrite (2 3 seconds IIRC), but since group by is probably more common of an operation, an option to configure the threshold may be a good idea (maybe even globally for all showProgress?

@tdhock WDYT? I recall that you expressed a preference to using 3 seconds

[tdhock]

you can turn it off using the showProgress argument/option, from ?data.table

       showProgress = getOption("datatable.showProgress", interactive())]
...
showProgress: 'TRUE' shows progress indicator with estimated time to
          completion for lengthy "by" operations.

if showProgress is set to numeric value, we could use that value as the threshold in seconds, instead of the default of 3 seconds.

[joshhwuu]

if showProgress is set to numeric value, we could use that value as the threshold in seconds, instead of the default of 3 seconds.

What about options(showProgress = ...)? Would we want to support this with fread/fwrite as well, as they both read showProgress as an option

[ethanbsmith]

you can turn it off using the showProgress argument/option, from ?data.table

that requires hunting down all the scenarios where this is happening and changing code. a global solution would be much preferred

[joshhwuu]

you can turn it off using the showProgress argument/option, from ?data.table

that requires hunting down all the scenarios where this is happening and changing code. a global solution would be much preferred

You can use options("datatable.showProgress" = FALSE), it'll set globally

[ethanbsmith]

for some reason i thought it was decided against having a global option to govern this. maybe i misread, or something changed. either way, that will solve this for me.

[ethanbsmith]

resolved as far as im concerned. your call whether u want to close this.

[joshhwuu]

This could be a good feature to think about if more people share the sentiment that a 3s threshold is too little, I don't think it hurts to keep the issue up

[MichaelChirico]

The request makes sense to me -- now we just have progress on/off, exposing some more customization is OK. I also worry adding several new options might be overkill, so the suggestion to overload the datatable.show.progress argument here makes sense to me. Here are the magic numbers we use for showProgress=TRUE now:

data.table/src/dogroups.c

Line 76 in 2a70e2f

double nextTime = (showProgress) ? startTime+3 : 0; // wait 3 seconds before printing progress

data.table/src/fwrite.c

Line 601 in 2a70e2f

double nextTime = startTime+2; // start printing progress meter in 2 sec if not completed by then

data.table/src/freadR.c

Line 681 in 2a70e2f

if (eta<3 || p>50) return;

One downside (upside?) to the change would be forcing the onset time to be the same for fwrite (currently it's 2 where the others are 3).

I also note the extra magic number for fread: IIUC if the initial estimate of progress is >50%, no bar is shown.