Better testing isolation

[di]

In #1425 it was discovered that a test was failing due to a configuration file in the user's home directory. Some tests also populate the global scope and fail the second time they are run.

By using pytest's tmpdir fixture and setting some environmental variables, it should be possible to improve the isolation of tests (at the expense of adding a bit more runtime to the test suite).

An example of how it's done from the pip test suite: https://github.com/pypa/pip/blob/db127330d4e071c84c5e1c107f60f8229e7d68ce/tests/conftest.py#L79-L148

[pganssle]

This issue is pretty big and amorphous, so in general I've found that makes it hard to divide up work (particularly during sprints).

If at some point someone (maybe me) wants to go through and figure out all the places where this can/needs to be done, we can possibly break this up into smaller issues.

[di]

I think this issue is pretty self contained, essentially we just need to add tmpdir and isolate fixtures and set them to be used automatically.

We can likely reuse the same pattern that pip uses, so the hard work would be paring down anything pip-specific from the linked example, and adding anything setuptools-specific.

[pganssle]

I don't understand, we use tmpdir in several places - by "use automatically" do you mean that every test is automatically run in a separate temporary directory? In the past we've used it like this.

If this can be done globally then you are right, we don't need to break it into smaller issues.

[di]

by "use automatically" do you mean that every test is automatically run in a separate temporary directory?

Yes.

[pganssle]

@di Would this be a good first issue for sprinters?

[di]

Yes I think it could be!

[jaraco]

In #1548, someone has worked on a patch. In this patch, they cause a temporary directory to be created for each test, but it's not used anywhere, so I don't think it has any effect. Is that because an 'isolate' fixture is missing?