qsv_cumsum gives zero

[ggrothendieck]

Using 2.2.1 this gives all zeros in the cumsum column. Maybe qsv_cumsum never actually got implemented? other?

>type aa.csv
a,b
1,2
3,4
> qsv luau map cumsum "qsv_cumsum(a)" aa.csv
a,b,cumsum
1,2,0
3,4,0

[jqnatividad]

The correct syntax to use luau qsv_cumsum is:

qsv luau map cumsum 'qsv_cumsum("total", a)' aa.csv
a,b,cumsum
1,2,2
3,4,6

The qsv_cumsum luau helper function requires two parameters. See:

qsv/src/cmd/luau.rs

Lines 2218 to 2248 in 4bb5556

	// this is a helper function that calculates the cumulative sum of a numeric column.
	// if the input cannot be converted to a number, it returns 0 for that row.
	//
	// qsv_cumsum(name, value)
	// name: identifier for this cumulative sum (allows multiple sums to run in parallel)
	// value: the numeric value to add to the cumulative sum
	// returns: the cumulative sum up to the current row for the named sum
	//
	let qsv_cumsum = luau.create_function(|_, (name, value): (String, mlua::Value)| {
	// Get static cumulative sums using thread_local storage
	thread_local! {
	static CUMSUMS: RefCell<HashMap<String, f64>> = RefCell::new(HashMap::new());
	}
	// Convert input value to number, defaulting to 0.0 if conversion fails
	let num = match value {
	Value::Number(n) => n,
	Value::Integer(i) => i as f64,
	Value::String(s) => s.to_string_lossy().parse::<f64>().unwrap_or_default(),
	_ => 0.0,
	};
	// Update cumulative sum for this name
	CUMSUMS.with(|cs| {
	let mut sums = cs.borrow_mut();
	let sum = sums.entry(name).or_insert(0.0);
	*sum += num;
	Ok(*sum)
	})
	})?;
	luau.globals().set("qsv_cumsum", qsv_cumsum)?;

Moving this issue to discussions so other folks with a similar question can easily find it...

[jqnatividad]

Further, for multiple cum_sums:

qsv luau map cumsum_a,cumsum_b '{qsv_cumsum("cumsum_a", a),qsv_cumsum("cumsum_b", b)}' aa.csv | qsv table
a   b   cumsum_a  cumsum_b
1   2   1         2
3   4   4         6

Note how the two qsv_cumsums are enclosed in braces so it returns a tuple which is mapped to the respective columns in the order they're listed in the luau command.

[ggrothendieck]

Is this linked to in the documentation? Suggest making that argument optional.

[jqnatividad]

It's "documented" inside the source code as you can see in the docstring above :).

And yeah, I'll need to have an expanded Luau section in the Wiki...

Will also look into making the name parameter optional...

[ggrothendieck]

Also is accumulate as suggested on April 29, 2024 here #1760 being implemented? That one is particularly useful because it replaces numerous loops not covered by cumsum and others. For example, if you want to do a cumsum but reset it every time a zero is encounted then accumuilate can do that.

[jqnatividad]

@ggrothendieck I implemented the other requested helper functions but not accumulate.

Your description of it went a bit over my head.

accumulate
Has three arguments. A column, a function of two arguments and an optional initial value. If y = accum(x, f, init)then y[1] = init and for i >1 we have y[i] = f(y[i-1], x[i]). The default for init is x[1]. Note that if f is +, *, or, and, max or min we get the above cum... functions.

Can you give a simpler definition with an illustrative example?

[jqnatividad]

The name argument for all cumulative helper fns (cum_) are now optional.
You only need to specify a name to avoid collisions when using the same cum_ helper multiple times in a single operation.

See #2540

[ggrothendieck]

It is a generalized loop where the value of the prior iteration is fed back into the current iteration. That is, it is basically a loop iterating over the elements of a vector x such that the output vector y is recursively given by y[i] = f(y[i-1], x[i]). To start off y[1] is x[1] if init is missing and f(init, x[1]) if init is present.

For example, if the function passed to it is addition and init is missing or equal to 0 then then y[i] = x[1] + x[2] + ... + x[i] .

The lua Functional module may have some relevant code as well.

Here is some lua code and some tests.

-- x is input vector, f is function of two args, init is optional
-- output is vector same length as x
function accumulate(x, f, init) do
  y = {}
  for i = 1, #x do
    if (i == 1) then
      if (init) then y[1] = f(init, x[1]) else y[1] = x[1] end
    else
      y[i] = f(y[i-1], x[i])
    end
  end
  return y end
end

-- tests

print("cumulative sum")
x = {3, 5, 4, 1}
add = function(x, y) do return x + y end end
y = accumulate(x, add)
for i = 1, #y do print(x[i], y[i]) end

print("\ncumulative sum with reset once sum exceeds 7")
add2 = function(x, y) do 
  if x > 7 then z = y else z = x+y end
  return z end
end
y = accumulate(x, add2)
for i = 1, #y do print(x[i], y[i]) end

giving

cumulative sum
3       3
5       8
4       12
1       13

cumulative sum with reset once sum exceeds 7
3       3
5       8
4       4  <-- note how it reset here since 8 > 7
1       5	

[ggrothendieck]

Also note that accumulate is related to fold in Rust except that fold only gives the the last value in the vector that accumulate returns.

[jqnatividad]

accumulate has been implemented. See #2536

[ggrothendieck]

I tried this and got the indicated error for every line of input. What should I be doing here?

> qsv --version
qsv 3.0.0-mimalloc-Luau 0.660;-4-4;3.11 GiB-8.04 GiB-1.02 GiB-3.89 GiB (x86_64-pc-windows-msvc compiled with Rust 1.84.1) compiled

> qsv luau map cum "qsv_accumulate(Tax, function(x, y) return x + y end)" us-states.csv
AL,4,"<ERROR> _IDX: 1 error(1): CallbackError { traceback: ""stack traceback:\n\t[C]: in function 'qsv_accumulate'\n\t[string \""src\\cmd\\luau.rs:808:18\""]:3: in ?"", cause: FromLuaConversionError { from: ""nil"", to: ""String"", message: Some(""expected string or number"") } }"
...snip...

[jqnatividad]

Be sure to enclose the column name in quotes:

> qsv luau map cum 'qsv_accumulate("Tax", function(x, y) return x + y end)' us-states.csv

[jqnatividad]

Just added column name validation to "qsv_accumulate" so the runtime error is less cryptic.
0b8f727

[ggrothendieck]

Using the same qsv version as before this is what I got. I put the line in a file in case the Windows cmd line was messing up.

>qsv --version
qsv 3.0.0-mimalloc-Luau 0.660;-4-4;3.11 GiB-7.68 GiB-743.02 MiB-3.89 GiB (x86_64-pc-windows-msvc compiled with Rust 1.84.1) compiled

>type acc.qsv
qsv_accumulate("Tax", function(x, y) return x + y end)

> qsv luau map cum -n acc.qsv us-states.csv
State,Tax,"<ERROR> _IDX: 1 error(1): RuntimeError(""[string \""src\\cmd\\luau.rs:808:18\""]:3: attempt to index nil with 'qsv'\nstack traceback:\n\t[C]: in ?\n\t[string \""src\\cmd\\luau.rs:808:18\""]:3: in ?"")"
AL,4,"<ERROR> _IDX: 2 error(2): RuntimeError(""[string \""src\\cmd\\luau.rs:808:18\""]:3: attempt to index nil with 'qsv'\nstack traceback:\n\t[C]: in ?\n\t[string \""src\\cmd\\luau.rs:808:18\""]:3: in ?"")"
...snip...

[jqnatividad]

@ggrothendieck , I would rename acc.qsv to acc.luau and issue the command as follows:

> qsv luau map cum acc.luau us-states.csv

There is no need for the -n (--no-headers) option...

[ggrothendieck]

Now it does not give any errors but the accumulate output is missing.

>type acc.luau
qsv_accumulate("Tax", function(x, y) return x + y end)

> qsv luau map cum acc.luau us-states.csv
State,Tax,cum
AL,4,
AK,0,
AZ,5.6,

[jqnatividad]

Yep! That creates a debug build of qsv.

To create an optimized release build:

git clone https://github.com/dathere/qsv.git
cd qsv
CARGO_BUILD_RUSTFLAGS='-C target-cpu=native' cargo build  --release --locked -F feature_capable,luau
copy target\release\qsv.exe %userprofile%\.cargo\bin
cargo clean

[ggrothendieck]

It now works. Why does one need return in return qsv_accumulate(...) but not in a number of other contexts? How does one know when return is needed and when not needed? I suppose one could always use return but there is an advantage to keeping the cmd line as short as possible.

> qsv luau map tmp "Tax+tax" us-states.csv
State,Tax,tmp
AL,4,8
AK,0,0
AZ,5.6,11.2
...snip...

> qsv luau map tmp "math.sqrt(Tax)" us-states.csv
State,Tax,tmp
AL,4,2
AK,0,0
AZ,5.6,2.3664319132398464
...snip...

[jqnatividad]

There is a very simple heuristic in luau where it looks for a "return" in the luau expression. If none is found, it prepends a return automatically.

qsv/src/cmd/luau.rs

Lines 413 to 418 in b3eb647

	// if the main script is a single expression, we need to prepend a return statement
	let mut main_script = if luau_script.contains("return") {
	String::new()
	} else {
	String::from("return ")
	};

Since the function arg in the example above contains a return in the function body, it doesn't prepend the return.

We can improve the heuristic so its a bit smarter.

[ggrothendieck]

Thanks. BTW I noticed that my qsv does not have sqlp. Do I just add it like this? Is there some way to ensure that I get everything?

cargo build  --locked -F feature_capable,luau,sqlp

[jqnatividad]

The feature you want to add is polars. It gates the sqlp, joinp and pivotp commands and the accelerated polars mode of the count command.

cargo build --locked -F feature_capable,luau,polars

or if you want to enable all features:

cargo build --locked -F all_features

FYI, if you're using qsv in production, an optimized release build is much, much faster.

CARGO_BUILD_RUSTFLAGS='-C target-cpu=native' cargo build --locked --release -F all_features

[jqnatividad]

This now works upstream after some bug fixing and making the name arg optional for all luau accumulator helper functions:

>type aa.csv
a,b
1,2
3,4
> qsv luau map cumsum "qsv_cumsum(a)" aa.csv
a,b,cumsum
1,2,1
3,4,4

This fix will be generally available in the v3.1.0 release.