Run unit tests in AppVeyor

[nemequ]

There is a nuget package for glib, so it shouldn't be too hard to get the unit tests running. Getting them to run successfully will, of course, be more work.

[jibsen]

The unit tests are all running on AppVeyor now, the remaining issues are:

• Dead2/zlib-ng#55, which is fixed upstream.
• Stack usage of ms-compress (Stack usage coderforlife/ms-compress#25), for now fixed by increasing stack space.
• lzham fails 32-bit threads test. I believe this is due to memory usage -- either we could change the options for the codec or run it with fewer threads.

[nemequ]

I just pulled in the latest zlib-ng, so that should be fixed.

LZHAM's memory usage shouldn't be too bad… Does this only happen on AppVeyor, or do you see it locally as well? I'd really like to see a backtrace on one of those crashes, maybe it's possible to set up AppVeyor to print one out… We already only spawn n_processors threads, I don't think decreasing that is a good idea (actually, it should really be at least doubled).

This is all in wip/msvc-unit-tests, right? We should merge the changes in to master (except for actually enabling the tests, until they all work).

[jibsen]

LZHAM's memory usage shouldn't be too bad… Does this only happen on AppVeyor, or do you see it locally as well?

Locally as well. If I do squash -c lzham file file.out, it uses 668 MB of memory. 32-bit processes are limited to 2 GB of memory by default, which matches the fact that it works if I limit the test to 2 threads, but fails with 4.

Here are the lzham log numbers if you want to see where it fails:

lzham_log_error: 5000
lzham_log_error: 5005
lzham_log_error: 9002
lzham_log_error: 7005
lzham_log_error: 5000
lzham_log_error: 5005
lzham_log_error: 9002
lzham_log_error: 7005
lzham_log_error: 6025
lzham_log_error: 5000
lzham_log_error: 5005
lzham_log_error: 9002
lzham_log_error: 7005
lzham_log_error: 6025

Wouldn't it be possible to set the dictionary size to perhaps 512k through the codec options for the threads test?

[nemequ]

Wouldn't it be possible to set the dictionary size to perhaps 512k through the codec options for the threads test?

It should be; I didn't actually expose that functionality in the plugin, but I'll do so soon. That said…

The current default for Squash is 258 MiB (which is what lzhamtest does on x86). Before we do anything here I'd like to wait to see what happens with richgel999/lzham_codec#14. If the "default" is lowered to something a bit more reasonable, we can avoid special-casing lzham in the unit test.

[jibsen]

I rebased and the tests are completing now, not sure what was causing the issue on AppVeyor then. It looks like the VM has only 2 cores, so it should be able to run the lzham test. It is still failing on my machine with 4 cores though.

[nemequ]

Okay, everything looks good, but I have questions about two commits:

• d8ac4b0: Why is this necessary? The files are opened in binary mode, so I would think that the whole \n vs. \r\n thing shouldn't be an issue, and we should be getting back exactly what we put in.
• 95e2375: This seems very fishy. I added this recently (99b9bc1) because, on Windows, if vsnprintf fails it doesn't return the number of bytes which would have been required to successfully write the data, it returns -1.

[jibsen]

• I guess I fixed the issue twice -- I changed the mode to binary in d40b3ac and then also removed the newline to be certain.
• Perhaps I fixed the wrong bug then, it returns SQUASH_FAILED because size == written.

[nemequ]

Perhaps I fixed the wrong bug then, it returns SQUASH_FAILED because size == written.

I just pushed a commit which adds a unit test for strings over 512 bytes long. Should be helpful here.

[nemequ]

Merged all but those two commits, thanks!