Common issues with tests failing on Windows

So far we have been running all our tests on Linux, so we haven't been taking into account some of the differences when running on Windows. Below is a list of common issues which cause tests to fail on Windows:

Path separators

In various tests we do things like:

# Get output of command-line interaction
assert "Wrote output to ./blah.txt" in output

This will fail on Windows, as the path to the file will be represented as .\blah.txt.

To fix this, use a proper os.path.join command to get the path, and then check that, for example:

# Get output of command-line interaction
output_path = os.path.join('.', 'blah.txt')
assert f"Wrote output to {output_path}" in output

Not closing SQLite connections

Windows is far more sensitive to files that are left open, and will refuse to delete them. If you have code like:

# Create SQLite file through Pepys
with sqlite3.connect('blah.db') as connection:
    connection.execute('SELECT * FROM Sensors;')

os.remove('blah.db')

Then it will fail on Windows, as using sqlite3.connect as a context manager does not close the connection when you exit the with block. Instead, you should switch to managing the connection manually - eg:

# Create SQLite file through Pepys
connection = sqlite3.connect('blah.db')
connection.execute('SELECT * FROM Sensors;')
connection.close()

os.remove('blah.db')

Querying without using session_scope()

Similarly, because of the way that SQLAlchemy manages connections, if you access data_store.session without being inside a with data_store.session_scope() block then SQLAlchemy will keep the session open, and therefore keep the SQLite file open - and if you try to delete the file you will get an error.

Therefore, make sure you always use a with data_store.session_scope() block if you're accessing the data in a test.

Line endings

On Linux the end of a line is signified by \n (known as LF) and on Windows the end of a line is \r\n (known as CRLF). This causes a few issues when running tests on Windows.

1. We do some checks to make sure we're reporting the file size and hash correctly for some of the test data files. By default when Git checks out files on Windows it replaces \n with \r\n, so the size and hash of these files changes. I have fixed this by adding a .gitattributes file in the root of the repo, which instructs Git not to do that to files in tests\sample_data.

2. We sometimes read in a file, do some modifications, then write it out again and compare the files. Python also automatically adapts to Windows-style line endings, so when we read a file with LF endings it will then write it out again with CRLF endings. This means the size won't match, as a load of CR characters will have been added. To fix this, we can specify newline="\n" in the open call when we open the file for writing, which forces it to write with LF-only line endings.

Checking os.stat mode

os.stat mode entries (that give chmod-style permissions) are a bit wonky on Windows, and some things (like read-only attributes) aren't reported here. It's best to just skip these checks on Windows.