examples/jupyter_notebook/tutorial_log_anomaly_detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Tutorial: Log Anomaly Detection Using LogAI\n",
    "\n",
    "This is an example to show how to use LogAI to conduct log anomaly detection analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Load Data\n",
    "\n",
    "You can use `OpensetDataLoader` to load a sample open log dataset. Here we use HealthApp dataset from\n",
    "[LogHub](https://zenodo.org/record/3227177#.Y1M3LezML0o) as an example.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from logai.dataloader.openset_data_loader import OpenSetDataLoader, OpenSetDataLoaderConfig\n",
    "\n",
    "#File Configuration\n",
    "filepath = os.path.join(\"..\", \"datasets\", \"HealthApp_2000.log\") # Point to the target HealthApp.log dataset\n",
    "\n",
    "dataset_name = \"HealthApp\"\n",
    "data_loader = OpenSetDataLoader(\n",
    "    OpenSetDataLoaderConfig(\n",
    "        dataset_name=dataset_name,\n",
    "        filepath=filepath)\n",
    ")\n",
    "\n",
    "logrecord = data_loader.load_data()\n",
    "\n",
    "logrecord.to_dataframe().head(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Preprocess\n",
    "\n",
    "In preprocessing step user can retrieve and replace any regex strings and clean the raw loglines. This\n",
    "can be very useful to improve information extraction of the unstructured part of logs,\n",
    " as well as generate more structured attributes with domain knowledge.\n",
    "\n",
    "Here in the example, we use the below regex to retrieve IP addresses."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.preprocess.preprocessor import PreprocessorConfig, Preprocessor\n",
    "from logai.utils import constants\n",
    "\n",
    "loglines = logrecord.body[constants.LOGLINE_NAME]\n",
    "attributes = logrecord.attributes\n",
    "\n",
    "preprocessor_config = PreprocessorConfig(\n",
    "    custom_replace_list=[\n",
    "        [r\"\\d+\\.\\d+\\.\\d+\\.\\d+\", \"<IP>\"],   # retrieve all IP addresses and replace with <IP> tag in the original string.\n",
    "    ]\n",
    ")\n",
    "\n",
    "preprocessor = Preprocessor(preprocessor_config)\n",
    "\n",
    "clean_logs, custom_patterns = preprocessor.clean_log(\n",
    "    loglines\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Parsing\n",
    "\n",
    "After preprocessing, we call auto-parsing algorithms to automatically parse the cleaned logs.\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.log_parser import LogParser, LogParserConfig\n",
    "from logai.algorithms.parsing_algo.drain import DrainParams\n",
    "\n",
    "# parsing\n",
    "parsing_algo_params = DrainParams(\n",
    "    sim_th=0.5, depth=5\n",
    ")\n",
    "\n",
    "log_parser_config = LogParserConfig(\n",
    "    parsing_algorithm=\"drain\",\n",
    "    parsing_algo_params=parsing_algo_params\n",
    ")\n",
    "\n",
    "parser = LogParser(log_parser_config)\n",
    "parsed_result = parser.parse(clean_logs)\n",
    "\n",
    "parsed_loglines = parsed_result['parsed_logline']"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Time-series Anomaly Detection\n",
    "\n",
    "Here we show an example to conduct time-series anomaly detection with parsed logs."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Feature Extraction\n",
    "\n",
    "After parsing the logs and get log templates, we can extract timeseries features by coverting\n",
    "these parsed loglines into counter vectors."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.feature_extractor import FeatureExtractorConfig, FeatureExtractor\n",
    "\n",
    "config = FeatureExtractorConfig(\n",
    "    group_by_time=\"15min\",\n",
    "    group_by_category=['parsed_logline', 'Action', 'ID'],\n",
    ")\n",
    "\n",
    "feature_extractor = FeatureExtractor(config)\n",
    "\n",
    "timestamps = logrecord.timestamp['timestamp']\n",
    "parsed_loglines = parsed_result['parsed_logline']\n",
    "counter_vector = feature_extractor.convert_to_counter_vector(\n",
    "    log_pattern=parsed_loglines,\n",
    "    attributes=attributes,\n",
    "    timestamps=timestamps\n",
    ")\n",
    "\n",
    "counter_vector.head(5)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Anomaly Detection\n",
    "\n",
    "With the generated `counter_vcetor`, you can use `AnomalyDetector` to detect timeseries anomalies.\n",
    "Here we use an algorithm in Merlion library called `DynamicBaseLine`."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.analysis.anomaly_detector import AnomalyDetector, AnomalyDetectionConfig\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "\n",
    "counter_vector[\"attribute\"] = counter_vector.drop(\n",
    "                [\n",
    "                    constants.LOG_COUNTS,\n",
    "                    constants.LOG_TIMESTAMPS,\n",
    "                    constants.EVENT_INDEX\n",
    "                ],\n",
    "                axis=1\n",
    "            ).apply(\n",
    "                lambda x: \"-\".join(x.astype(str)), axis=1\n",
    "            )\n",
    "\n",
    "attr_list = counter_vector[\"attribute\"].unique()\n",
    "\n",
    "anomaly_detection_config = AnomalyDetectionConfig(\n",
    "    algo_name='dbl'\n",
    ")\n",
    "\n",
    "res = pd.DataFrame()\n",
    "for attr in attr_list:\n",
    "    temp_df = counter_vector[counter_vector[\"attribute\"] == attr]\n",
    "    if temp_df.shape[0] >= constants.MIN_TS_LENGTH:\n",
    "        train, test = train_test_split(\n",
    "            temp_df[[constants.LOG_TIMESTAMPS, constants.LOG_COUNTS]],\n",
    "            shuffle=False,\n",
    "            train_size=0.3\n",
    "        )\n",
    "        anomaly_detector = AnomalyDetector(anomaly_detection_config)\n",
    "        anomaly_detector.fit(train)\n",
    "        anom_score = anomaly_detector.predict(test)\n",
    "        res = res.append(anom_score)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Get anomalous datapoints\n",
    "anomalies = counter_vector.iloc[res[res>0].index]\n",
    "anomalies.head(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Semantic Anomaly Detection\n",
    "\n",
    "We can also use the log template for semantic based anomaly detection. In this approach, we retrieve\n",
    "the semantic features from the logs. This includes two parts: vectorizing the unstructured log templates\n",
    "and encoding the structured log attributes."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Vectorization for unstructured loglines\n",
    "\n",
    "Here we use `word2vec` to vectorize unstructured part of the logs. The output will be a list of\n",
    "numeric vectors that representing the semantic features of these log templates."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.log_vectorizer import VectorizerConfig, LogVectorizer\n",
    "\n",
    "vectorizer_config = VectorizerConfig(\n",
    "    algo_name = \"word2vec\"\n",
    ")\n",
    "\n",
    "vectorizer = LogVectorizer(\n",
    "    vectorizer_config\n",
    ")\n",
    "\n",
    "# Train vectorizer\n",
    "vectorizer.fit(parsed_loglines)\n",
    "\n",
    "# Transform the loglines into features\n",
    "log_vectors = vectorizer.transform(parsed_loglines)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Categorical Encoding for log attributes\n",
    "\n",
    "We also do categorical encoding for log attributes to convert the strings into numerical representations."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.categorical_encoder import CategoricalEncoderConfig, CategoricalEncoder\n",
    "\n",
    "encoder_config = CategoricalEncoderConfig(name=\"label_encoder\")\n",
    "\n",
    "encoder = CategoricalEncoder(encoder_config)\n",
    "\n",
    "attributes_encoded = encoder.fit_transform(attributes)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Feature Extraction\n",
    "\n",
    "Then we extract and concate the semantic features for both the unstructured and structured part of logs.\n"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from logai.information_extraction.feature_extractor import FeatureExtractorConfig, FeatureExtractor\n",
    "\n",
    "timestamps = logrecord.timestamp['timestamp']\n",
    "\n",
    "config = FeatureExtractorConfig(\n",
    "    max_feature_len=100\n",
    ")\n",
    "\n",
    "feature_extractor = FeatureExtractor(config)\n",
    "\n",
    "_, feature_vector = feature_extractor.convert_to_feature_vector(log_vectors, attributes_encoded, timestamps)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Anomaly Detection\n",
    "\n",
    "With the extracted log semantic feature set, we can perform anomaly detection to find the abnormal\n",
    "logs. Here we use `isolation_forest` as an example."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, test = train_test_split(feature_vector, train_size=0.7, test_size=0.3)\n",
    "\n",
    "from logai.algorithms.anomaly_detection_algo.isolation_forest import IsolationForestParams\n",
    "from logai.analysis.anomaly_detector import AnomalyDetectionConfig, AnomalyDetector\n",
    "\n",
    "algo_params = IsolationForestParams(\n",
    "    n_estimators=10,\n",
    "    max_features=100\n",
    ")\n",
    "config = AnomalyDetectionConfig(\n",
    "    algo_name='isolation_forest',\n",
    "    algo_params=algo_params\n",
    ")\n",
    "\n",
    "anomaly_detector = AnomalyDetector(config)\n",
    "anomaly_detector.fit(train)\n",
    "res = anomaly_detector.predict(test)\n",
    "# obtain the anomalous datapoints\n",
    "anomalies = res[res==1]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Check the corresponding loglines"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "loglines.iloc[anomalies.index].head(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "#### Check the corresponding attributes"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "attributes.iloc[anomalies.index].head(5)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}