Anomaly Detector for ALeRCE broker

Welcome to the ALeRCE anomaly detector framework. This is the main repository where you can find all the resources associated to our article: Alert Classification for the ALeRCE Broker System: The Anomaly Detector.

Our methodology has been published and is available through the ALeRCE Broker, a Chilean-led platform that processes the alert stream from the Zwicky Transient Facility (ZTF).

In our work we look for the most promising outlier detection algorithms that aim to find transient, periodic and stochastic anomalous sources within the ZTF data stream. The ALeRCE anomaly detector framework consists of crossvalidating six anomaly detection algorithms for each of these three classes using the ALeRCE light curve features.

Following the ALeRCE taxonomy, we consider four transient subclasses, five stochastic subclasses, and five periodic subclasses. We evaluate each algorithm by considering each subclass as the anomaly class.

About our work

We provide a Machine and Deep Learning-based framework for anomaly detection. Our methodology is inspired by the ALeRCE's light curve (LC) classifier and follows a hierarchical approach. The light curves are categorized into three main classes: transient, stochastic, and periodic, at the top level. For each class, a distinct anomaly detection model is constructed, utilizing only information about the known objects (i.e., inliers) for training. During testing, in order to assign the light curve to one of the anomaly detectors and compute the anomaly score, we use the probabilities, as given by ALeRCE's LC classifier, indicating whether the light curve corresponds to a transient, stochastic, or periodic nature.

Pipeline

Figure 1. Methodology for training and evaluation of the anomaly detection algorithms. We split the data into a training set and test set composed by 80% and 20% of the data, respectively The training set is subdivided into transient stochastic, and periodic data. For each of these classes, we choose each subclass as the outlier class. The outlier class is removed from the training set and added to the test set (TS2). Then, an anomaly detection algorithm is trained using the remaining objects of each of the classes, and is evaluated using TS2.

Results

See the notebooks (/presentation/notebooks/*) for usage examples with these models.

	Transient				Stochastic					Periodic
Method	SLSN	SNII	SNIa	SNIbc	AGN	Blazar	CV/Nova	QSO	YSO	CEP	DSCT	E	RRL	LPV
IForest	0.640 ±0.014	0.721 ±0.021	0.428 ±0.032	0.490 ±0.038	0.573 ±0.017	0.710 ±0.009	0.975 ±0.001	0.468 ±0.016	0.913 ±0.003	0.359 ±0.007	0.295 ±0.012	0.469 ±0.021	0.549 ±0.033	0.971 ±0.007
OCSVM	0.577 ±0.014	0.587 ±0.014	0.434 ±0.021	0.492 ±0.011	0.532 ±0.008	0.443 ±0.002	0.909 ±0.001	0.517 ±0.005	0.792 ±0.005	0.432 ±0.004	0.557 ±0.005	0.555 ±0.003	0.539 ±0.004	0.943 ±0.001
AE	0.736 ±0.022	0.807 ±0.021	0.438 ±0.015	0.537 ±0.019	0.701 ±0.010	0.762 ±0.006	0.980 ±0.016	0.443 ±0.004	0.990 ±0.001	0.564 ±0.024	0.367 ±0.015	0.864 ±0.009	0.907 ±0.015	0.996 ±0.000
VAE	0.669 ±0.015	0.690 ±0.023	0.404 ±0.018	0.522 ±0.025	0.596 ±0.007	0.597 ±0.010	0.849 ±0.028	0.500 ±0.009	0.795 ±0.009	0.442 ±0.010	0.417 ±0.007	0.561 ±0.007	0.451 ±0.006	0.936 ±0.007
Deep SVDD	0.644 ±0.043	0.690 ±0.043	0.475 ±0.040	0.507 ±0.040	0.496 ±0.025	0.607 ±0.044	0.932 ±0.015	0.411 ±0.008	0.901 ±0.022	0.707 ±0.027	0.482 ±0.054	0.636 ±0.055	0.774 ±0.068	0.785 ±0.025
MCDSVDD (Ours)	0.686 ±0.051	0.828 ±0.024	0.624 ±0.039	0.584 ±0.032	0.706 ±0.069	0.512 ±0.113	0.770 ±0.127	0.483 ±0.080	0.854 ±0.041	0.858 ±0.025	0.819 ±0.015	0.945 ±0.006	0.953 ±0.003	0.953 ±0.008

Table 1. Evaluation of the performance of each model when applied to each of the ALeRCE top level taxonomy (transient, stochastic, periodic). Each row represents a different outlier detection algorithm, and each column represents the subclass considered as outlier. The performance is evaluated using the cross–validation AUROC scores.

As shown in Table 1, the best performance was achieved for transient and periodic sources using a modified version of the Deep Support Vector Data Description (MCDSVDD) neural network. However, for stochastic sources, the best results were obtained by calculating the reconstruction error of an autoencoder (AE) neural network.

Features

• Six anomaly detection algorithms were implemented in order to compare their performances in finding outliers (/src/models/*)
  • Isolation Forest (/src/models/IForest.py)
  • One-class Support Vector Machine (/src/models/*)
  • Autoencoder (/src/models/Autoencoder.py)
  • Variational Autoencoder (/src/models/VariationalAutoencoder.py)
  • Deep Support Vector Data Description (/src/models/DeepSVDD.py)
  • Multi-Class Deep SVDD (/src/models/ClasSVDD.py)
• A new scheme of training and evaluation methodology
• Validated framework in a real-world scenario by selecting the 10 sources with the highest outlier score per each of the 15 classes predicted by the ALeRCE light curve classifier (presentation/notebooks/Results analysis.ipynb)
• Methods trained using the ZTF alert stream and benefit from the ALeRCE LC classifier
• Predefined experiments to reproduce publication results (presentation/notebooks/*)
• Data preprocessing, saving and reading (/src/preprocessing/*)
• Dockerfile and scripts for building (build_container.sh) and run (run_container.sh) the Anomaly Detector container

Implementation tree

📦AnomalyALeRCE
 ┣ 📂data
 ┃ ┣ 📜 get_data.sh: download data (raw and preprocessed) from gdrive
 ┃ ┗ 📜 README.md: instructions how to use get_data.sh 
 ┣ 📂 experiments: experiments folder
 ┃ ┗ 📜 launch_experiments.sh: script for running all the published version of the experiments
 ┣ 📂 presentation: everything that depends on the model code (i.e., experiments, plots and figures)
 ┃ ┣ 📂 figures
 ┃ ┗ 📂 notebooks: util notebooks to visualize and analyze the results
 ┣ 📂 src: Model source code
 ┃ ┗ 📂 preprocessing: functions related to data manipulation
 ┃ ┃ ┣ 📜 ALeRCE_LC.py: data preparation functions inspired on ALeRCE LC classifier
 ┃ ┃ ┣ 📜 create_dataloaders.py: create the dataloaders for the ML and DL models
 ┃ ┃ ┣ 📜 ALeRCE_LC.py: data preparation functions inspired on ALeRCE LC classifier
 ┃ ┃ ┣ 📜 data_utils.py: general functions to standardize, among others
 ┃ ┃ ┗ 📜 main.py: main script containing functions to format data
 ┃ ┗ 📂 models: Anomaly Detector models architectures
 ┃ ┃ ┣ 📜 main.py: build the network with the selected classifier
 ┃ ┃ ┣ 📜 Autoencoder.py: Autoencoder
 ┃ ┃ ┣ 📜 ClasSVDD.py: Multi-Class Deep SVDD (ours)
 ┃ ┃ ┣ 📜 DeepSVDD.py: Deep Support Vector Data Description
 ┃ ┃ ┗ 📜 VariationalAutoencoder.py: Variational autoencoder
 ┃ ┗ 📂 utils.py: universal functions to use on different modules
 ┃ ┣ 📜 train.py: functions related to the training stage of the models
 ┃ ┗ 📜 evaluate.py: functions related to the inference stage of the models
 ┣ 📜 .gitignore: files that should not be considered during a GitHub push
 ┣ 📜 .dockerignore: files to exclude when building a Docker container.
 ┣ 📜 build_container.sh: script to build the Anomaly Detector Docker image
 ┣ 📜 run_container.sh: script to run the Anomaly Detector Docker image (up container)
 ┣ 📜 Dockerfile: Docker image definition
 ┣ 📜 requirements.txt: python dependencies
 ┣ 📜 main.py: code for running the models
 ┗ 📜 README.md: what you are currently reading

Setup the enviroment

The easiest way to reproduce our results is to set up an environment by using Docker. Docker provides the ability to package and run an application in a loosely isolated environment called a container. We create a a container identical to the one used to perform the experiments.

Install Docker by following the instructions here.

The Dockerfile contains all the configurations required to run the container for the Anomaly Detector framework. You don't need to modify it directly. Instead, use the provided scripts build_container.sh and run_container.sh to handle the container setup for you.

Building the container

To build the container, run the following command:

  bash build_container.sh

This command creates a virtual machine named anomalydetector that includes all the necessary dependencies, such as Python, PyTorch, and others.

Running the Container

To run the Anomaly Detector container, use the following command:

  bash run_container.sh

This script automatically finds the anomalydetector container and runs it on top of your kernel. If GPUs are available, the script makes them visible inside the container.

By default, the run_container.sh script opens ports 8888 and 6006 for Jupyter Notebook and TensorBoard, respectively. To access them, use the following commands:

For Jupyter Notebook

jupyter notebook --ip 0.0.0.0

For TensorBoard

tensorboard --logdir <my-logs-folder> --host 0.0.0.0

Setup locally

If you prefer not to use Docker, you can install the required packages locally. The requirements.txt file contains all the necessary dependencies. To install them, use the following command in your local Python environment:

pip install -r requirements.txt

Dependencies

• Python == 3.8.10
• Torch == 2.0.1
• Other packages (see requirements.txt for details).

Reproducibility

For reproducibility of our reported results run, we provide a script (presentation/experiments/launch_experiments.bash) that can be used to run all the models at once. It is important to note that this script includes the learning rate and embedding dimensions that were used in the paper. Additionally, the order in which the experiments are run is crucial, as the DEEPSVDD and MCDSVDD models rely on the pretrained weights of the AE model. To preprocess the data used in the paper, download the data following the next steps.

src data

bash get_data.sh ztf-processed

### To use your own ALeRCE's ZTF data, you should first run (src/preprocessing/main.py). Following the steps below:

cd src/preprocess

python3 main.py --labels_file 'your_path1' --features_file  'your_path2' --features_list  'your_path3'

Train models

If you want to train a model with your own data subsets, we provide a script (main.py) where you can run models as:

python3 main.py --model ae --hierClass Transient --outlier YSO  --all_outliers False --lr 5e-4 --z_dim 32

In the given case, the outlier class corresponds to the transient YSO. If you want to detect all possible outliers within the Transient hierarchical class, you should set the parameter (--all_outliers) to true.

python3 main.py --model ae --hierClass Transient --all_outliers True --lr 5e-4 --z_dim 32

Models (--model) available are Autoencoder (ae), Variational Autoencoder (vae), IForest (iforest), OCSVM (ocsvm), Deep Support Vector Data Description (deepsvdd) and our proposed method Multi-Class Deep SVDD (classvdd). Hierarchical classes (--hierClass) available are [Transient, Stchastic, Periodic].

Contributing 🤝

Contributions are always welcome!

Issues and featuring can be directly published in this repository via Pull Request. Look at this tutorial for more information about pull requests.

Cite us

@article{Perez-Carrasco_2023,
doi = {10.3847/1538-3881/ace0c1},
url = {https://dx.doi.org/10.3847/1538-3881/ace0c1},
year = {2023},
month = {sep},
publisher = {The American Astronomical Society},
volume = {166},
number = {4},
pages = {151},
author = {Manuel Perez-Carrasco and Guillermo Cabrera-Vives and Lorena Hernandez-García and F. Förster and Paula Sanchez-Saez and Alejandra M. Muñoz Arancibia and Javier Arredondo and Nicolás Astorga and Franz E. Bauer and Amelia Bayo and M. Catelan and Raya Dastidar and P. A. Estévez and Paulina Lira and Giuliano Pignata},
title = {Alert Classification for the ALeRCE Broker System: The Anomaly Detector},
journal = {The Astronomical Journal},
abstract = {Astronomical broker systems, such as Automatic Learning for the Rapid Classification of Events (ALeRCE), are currently analyzing hundreds of thousands of alerts per night, opening up an opportunity to automatically detect anomalous unknown sources. In this work, we present the ALeRCE anomaly detector, composed of three outlier detection algorithms that aim to find transient, periodic, and stochastic anomalous sources within the Zwicky Transient Facility data stream. Our experimental framework consists of cross-validating six anomaly detection algorithms for each of these three classes using the ALeRCE light-curve features. Following the ALeRCE taxonomy, we consider four transient subclasses, five stochastic subclasses, and six periodic subclasses. We evaluate each algorithm by considering each subclass as the anomaly class. For transient and periodic sources the best performance is obtained by a modified version of the deep support vector data description neural network, while for stochastic sources the best results are obtained by calculating the reconstruction error of an autoencoder neural network. Including a visual inspection step for the 10 most promising candidates for each of the 15 ALeRCE subclasses, we detect 31 bogus candidates (i.e., those with photometry or processing issues) and seven potential astrophysical outliers that require follow-up observations for further analysis.}
      year={2023},
      eprint={2308.05011},
      archivePrefix={arXiv},
      primaryClass={cs.LG}}