Remove Config objects

[ejnnr]

Very much WIP (e.g. I haven't adjusted tests yet, abstractions are probably broken, and I'll probably also want to get rid of ScriptConfig). But I (perhaps naively) think most of the work is done, and I tested that things still work manually a bit (see e.g. notebooks/simple_demo.ipynb)

This will probably just replace #29 but I wanted to make a separate PR in case we decide to not use these changes but still use #29.

[ejnnr]

Mostly done now (tests and the example notebook pass, and I've removed every single config class). Also cleaned up a few other things, so this PR ended up pretty huge. I think the main thing that would be good to look at @VRehnberg is notebooks/simple_demo.ipynb and check that the interface seems fine (we can make minor improvements later).

The other thing worth checking might be how WaNet is handled---I've removed all the logic for automagically ensuring the validation set uses the same control grid, or loading from disk. Basically, I think it's really hard for us to always handle those things correctly now that we just accept arbitrary pytorch Datasets (e.g. the user could pass in their own custom dataset, which wraps a BackdoorDataset). So I think we should let the user handle this, I've added a warning to the docstring, and tried to make it somewhat harder to forget loading the control grid by having a mandatory init argument. We can also have a notebook example that shows how to handle everything correctly. Still, this remains a bit of a footgun.

Of course you're welcome to check other things too, but I don't have concrete guesses for which parts are most likely to be buggy unfortunately.

[ejnnr]

Also note that configs aren't saved to disk anymore, see #32 I don't think we actually need this at the moment but would definitely be nice to add back in a much simpler version

[VRehnberg]

The interface is nice. Two things to note:

• The test_tampering.py fails for me (type mismatch when calculating the loss I believe) but it fails on main for me as well (different error) and I don't think this PR introduces the issue. Still, perhaps double check that.
• I'm a bit unsure how you've been imagining that WanetBackdoor instances best be used right now (see comment).

[VRehnberg]

Suggested change

	Within a single process, just make sure you only initialize WanetBackdoor once
	and then use that everywhere.
	Within a single process, just make sure you only initialize a fresh WanetBackdoor once
	and then reuse its warping pattern everywhere.

Surely you'll want to have different instances here to control p_backdoor and p_noise individually for different validation sets? Or am I missing something?

[VRehnberg]

I've added #34 for an example of how I'd imagine you'd be using these backdoors.

[ejnnr]

Yeah you're right of course, thanks for #34!

[ejnnr]

Hm, the tampering test passes for me both here and on main. Since it already fails for you on main anyway, I'll merge this PR now, but we should figure out what's going on probably. Can you paste the stack trace you're getting?

You're totally right about the need for multiple WaNet backdoor instances, I'll merge #34 or some version of it first and then merge into main.

[VRehnberg]

Here's the stacktrace https://gist.github.com/VRehnberg/f3e5089fc03083808d1923b3df4bbc11

[VRehnberg]

It's the torch.nn.functional.binary_cross_entropy that doesn't accept that the targets of data.TamperingDataset("diamonds") are boolean. MWE:

>>> import torch
>>> torch.nn.functional.binary_cross_entropy(torch.rand(10), (torch.rand(10) > 0.5).to(torch.bool))
*** RuntimeError: Found dtype Bool but expected Float
>>> torch.nn.functional.binary_cross_entropy(torch.rand(10), (torch.rand(10) > 0.5).to(torch.float32))
tensor(1.2316)

[VRehnberg]

The labels are boolean from the source and an additional all statement in tampering.py.

>>> datasets.load_dataset("redwoodresearch/diamonds-seed0", split="validation")["measurements"][0]
[True, True, True]

[ejnnr]

Oh I think I know what's going on. Your MWE errors out on CPU for me but passes on MPS. Since pytorch lightning picks MPS by default over CPU if available, the tests pass for me. Guessing that Oliver was also on MPS (or maybe it also works on CUDA).

[VRehnberg]

As long as it also works with floats on MPS (CUDA errors out on booleans same as CPU). I'll create a PR that adds a typecast somewhere.