Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Config objects #30

Merged
merged 27 commits into from
Mar 4, 2024
Merged
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
460ff9b
Export AnomalyDetector
ejnnr Feb 28, 2024
dbae3bf
Make tasks more flexible
ejnnr Feb 29, 2024
f16b9ca
Iterating on tasks
ejnnr Feb 29, 2024
9073a85
Mostly fix tests
ejnnr Feb 29, 2024
54c34a6
[WIP] Remove configs
ejnnr Mar 1, 2024
51e6a25
Remove unused DatasetConfigs
ejnnr Mar 1, 2024
48f8292
Rename task file
ejnnr Mar 1, 2024
79b51ec
WIP on removing ScriptConfig and TrainConfig
ejnnr Mar 2, 2024
bdd56fb
Remove backdoor loading/storing logic
ejnnr Mar 2, 2024
62e618a
Remove TrainConfig
ejnnr Mar 2, 2024
94c54ed
Adjust abstractions
ejnnr Mar 2, 2024
4c7e0c2
Remove loggers
ejnnr Mar 3, 2024
6809a7e
Fix bugs and tests
ejnnr Mar 3, 2024
6f0e472
Move save_path and max_batch_size arguments
ejnnr Mar 3, 2024
ae98812
Remove another unused file
ejnnr Mar 3, 2024
31a7993
Remove more unused code
ejnnr Mar 3, 2024
f0dacc5
Minor improvements and remove TODOs
ejnnr Mar 3, 2024
0267bd1
Fix demo notebook
ejnnr Mar 3, 2024
975289e
Add WaNet warning
ejnnr Mar 3, 2024
1b82635
Update gitignore
ejnnr Mar 3, 2024
35220aa
Update documentation somewhat
ejnnr Mar 3, 2024
f9ab02b
Remove simple_parsing dependency
ejnnr Mar 3, 2024
80463e2
Merge remote-tracking branch 'origin/main' into no-configs
ejnnr Mar 4, 2024
d61c676
Adjust tampering/LM code to no-config style
ejnnr Mar 4, 2024
565f456
Add convenience method to clone WanetBackdoor instance
VRehnberg Mar 4, 2024
2c1b38c
Minor changes to WaNet cloning
ejnnr Mar 4, 2024
f7e9300
Merge pull request #34 from VRehnberg/wanet-partial-clone-method
ejnnr Mar 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Minor improvements and remove TODOs
ejnnr committed Mar 3, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit f0dacc5123be402f3e258476f9dbffb69a1ad7d9
10 changes: 3 additions & 7 deletions src/cupbearer/detectors/anomaly_detector.py
Original file line number Diff line number Diff line change
@@ -92,11 +92,7 @@ def eval(
self,
# Don't need train_dataset here, but e.g. adversarial abstractions need it,
# and in general there's no reason to deny detectors access to it during eval.
# TODO: I think we can/should remove this and require detectors to handle
# anything involving training data during training (now that they get access
# to untrusted data then).
train_dataset: Dataset,
test_dataset: MixedData,
dataset: MixedData,
batch_size: int = 1024,
histogram_percentile: float = 95,
save_path: Path | str | None = None,
@@ -105,10 +101,10 @@ def eval(
):
# Check this explicitly because otherwise things can break in weird ways
# when we assume that anomaly labels are included.
assert isinstance(test_dataset, MixedData), type(test_dataset)
assert isinstance(dataset, MixedData), type(dataset)

test_loader = DataLoader(
test_dataset,
dataset,
batch_size=batch_size,
# For some methods, such as adversarial abstractions, it might matter how
# normal/anomalous data is distributed into batches. In that case, we want
3 changes: 1 addition & 2 deletions src/cupbearer/scripts/eval_detector.py
Original file line number Diff line number Diff line change
@@ -14,8 +14,7 @@ def main(
detector.set_model(task.model)

detector.eval(
train_dataset=task.trusted_data,
test_dataset=task.test_data,
dataset=task.test_data,
pbar=pbar,
save_path=save_path,
batch_size=batch_size,
2 changes: 1 addition & 1 deletion src/cupbearer/scripts/train_classifier.py
Original file line number Diff line number Diff line change
@@ -62,7 +62,6 @@ def main(
trainer_kwargs["callbacks"] = callbacks

# Define metrics logger
# TODO: make adjustable and set config correctly
if "logger" not in trainer_kwargs:
if wandb:
metrics_logger = loggers.WandbLogger(project="cupbearer")
@@ -72,6 +71,7 @@ def main(
"model": repr(model),
"train_data": repr(train_loader.dataset),
"batch_size": train_loader.batch_size,
"lr": lr,
}
)
elif path:
5 changes: 0 additions & 5 deletions src/cupbearer/tasks/backdoor_detection.py
Original file line number Diff line number Diff line change
@@ -20,11 +20,6 @@ def backdoor_detection(
"this is probably unintentional."
)

# TODO: for WaNet, we currently expect the user to load the control grid.
# (Otherwise we'd have to always take in a path here, and also when working
# in a notebook it might just be easier to pass in the existing backdoor object.)
# But we should somehow check somewhere that it's loaded to avoid silent errors.

return Task.from_base_data(
model=model,
train_data=train_data,