explainers/notebooks/mli-byor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLI BYOR: Custom Explainers\n",
    "This notebook is a demo of MLI **bring your own explainer recipe** (BYOR) Python API.\n",
    "\n",
    "**Ad-hoc OOTB and/or custom explainer run** scenario:\n",
    "* **Upload** interpretation recipe.\n",
    "    * Determine recipe upload job **status**.\n",
    "* **Run** ad-hoc recipe run job.\n",
    "    * Determine ad-hoc recipe job **status**.\n",
    "* **Get** explainer type for given job.\n",
    "* **List** explanation types for given explainer run.\n",
    "* **List** explanation representations (formats) URLs for given explanation type.\n",
    "* **Download** explanation representation from ^ URL.    \n",
    "* **Download** interpretation recipe job result **data**.\n",
    "\n",
    "**Interpretation explainers run** scenario:\n",
    "* **List** available/compatible explainers.\n",
    "* **Choose** subset or all ^ compatible explainers.\n",
    "* **Run** interpretation job.\n",
    "   * Determine **status** of interpretation job.\n",
    "   * Determine **status** per explainer job (running within the scope of interpretation).\n",
    "* **List** explainer types which were ran within interpretation.\n",
    "* **List** explainer runs for given explainer type within the interpretation.\n",
    "* **List** explanation types for given explainer run.\n",
    "* **List** explanation representations (formats) URLs for given explanation type.\n",
    "* **Download** explanation representation from ^ URL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Virtual Environment and Dependencies\n",
    "Prepare and activate virtual environment for this notebook by running:\n",
    "```\n",
    ". .env/bin/activate\n",
    "pip install ipykernel\n",
    "ipython kernel install --user --name=dai\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pprint\n",
    "import time\n",
    "from random import randint\n",
    "from h2oaicore.mli.oss.commons import MimeType, MliJobStatus\n",
    "from h2oaicore.messages import (\n",
    "    CommonDaiExplainerParameters,\n",
    "    CommonExplainerParameters,\n",
    "    DatasetReference,\n",
    "    Explainer,\n",
    "    ExplainerDescriptor,\n",
    "    ExplainerJobStatus,\n",
    "    ExplainerRunJob,\n",
    "    ExplainersRunJob,\n",
    "    InterpretationJob,\n",
    "    ModelReference,\n",
    ")\n",
    "\n",
    "import h2o\n",
    "import h2oai_client\n",
    "from h2oai_client import Client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Connect** to DAI server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect to Driverless AI server - make sure to use the same \n",
    "# user name and password as when signing in through the GUI\n",
    "hostname = '127.0.0.1'\n",
    "address = 'http://' + hostname + ':12345'\n",
    "username = 'h2oai'\n",
    "password = 'h2oai'\n",
    "\n",
    "# h2oai = Client(\"http://localhost:12345\", \"h2oai\", \"h2oai\")\n",
    "h2oai = Client(address = address, username = username, password = password)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload \n",
    "Upload BYOR interpretation [recipe](http://192.168.59.141/mli-byor/mli_byor_foo.py) to Driverless AI server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# URL of the recipe to upload\n",
    "\n",
    "# Custom Morris sensitivity analysis\n",
    "URL_BYOR_EXPLAINER = \"https://h2o-public-test-data.s3.amazonaws.com/recipes/explainers/morris_sensitivity_explainer.py\"\n",
    "BYOR_EXPLAINER_NAME = \"Morris Sensitivity Analysis\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Upload recipe** to DAI server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3782c148-017b-11eb-a66d-207918bc8e4b'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recipe_job_key = h2oai.create_custom_recipe_from_url(URL_BYOR_EXPLAINER)\n",
    "recipe_job_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'created': 1601291428.8856423,\n",
      " 'entity': {'data_file': '',\n",
      "            'data_files': [],\n",
      "            'datas': [],\n",
      "            'explainers': [],\n",
      "            'fpath': None,\n",
      "            'key': '3782c148-017b-11eb-a66d-207918bc8e4b',\n",
      "            'models': [],\n",
      "            'name': 'morris_sensitivity_explainer',\n",
      "            'pretransformers': [],\n",
      "            'scorers': [],\n",
      "            'transformers': [],\n",
      "            'type': '',\n",
      "            'url': 'https://h2o-public-test-data.s3.amazonaws.com/recipes/explainers/morris_sensitivity_explainer.py'},\n",
      " 'error': '',\n",
      " 'message': 'MorrisSensitivityLeExplainer:\\nInstalling Class Packages',\n",
      " 'progress': 1,\n",
      " 'status': 0}\n"
     ]
    }
   ],
   "source": [
    "recipe_job = h2oai._wait_for_recipe_load(recipe_job_key)\n",
    "pprint.pprint(recipe_job.dump())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploaded recipe ID: False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer'\n"
     ]
    }
   ],
   "source": [
    "if recipe_job.entity.explainers:\n",
    "    uploaded_explainer_id:str = recipe_job.entity.explainers[0].id    \n",
    "else:\n",
    "    # explainer already deployed (look it up)\n",
    "    explainers = h2oai.list_explainers(\n",
    "        experiment_types=None, \n",
    "        explanation_scopes=None,\n",
    "        dai_model_key=None,\n",
    "        keywords=None,\n",
    "        explainer_filter=[]\n",
    "    )\n",
    "    uploaded_explainer_id = [explainer.id for explainer in explainers if BYOR_EXPLAINER_NAME == explainer.name][0]\n",
    "\n",
    "print(f\"Uploaded recipe ID: {uploaded_explainer_id}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Driverless AI **model** and **dataset**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model  : 34edc42e-0152-11eb-a66d-207918bc8e4b\n",
      "Dataset: dataset_33ab9348-0152-11eb-a66d-207918bc8e4b\n",
      "Target : default payment next month\n"
     ]
    }
   ],
   "source": [
    "# *) hardcoded DAI keys\n",
    "# DATASET_KEY = \"f12f69b4-475b-11ea-bf67-9cb6d06b189b\"\n",
    "# MODEL_KEY = \"f268e364-475b-11ea-bf67-9cb6d06b189b\"\n",
    "\n",
    "# *) lookup compatible DAI keys\n",
    "compatible_models = h2oai.list_explainable_models(\n",
    "    explainer_id=uploaded_explainer_id, offset=0, size=30\n",
    ")\n",
    "if compatible_models.models:\n",
    "    MODEL_KEY = compatible_models.models[0].key\n",
    "    DATASET_KEY = compatible_models.models[0].parameters.dataset.key\n",
    "else:\n",
    "    raise RuntimeError(\"No compatible models found: please train an IID regression/binomial experiment\")\n",
    "\n",
    "target_col = h2oai.get_model_summary(key=MODEL_KEY).parameters.target_col\n",
    "    \n",
    "print(f\"Model  : {MODEL_KEY}\\nDataset: {DATASET_KEY}\\nTarget : {target_col}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# List Explainers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List custom and OOTB recipes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-sensitivity-analysis',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'SaExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'SA explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-disparate-impact-analysis',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'DiaExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.dia_explainer.DiaExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DIA explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.permutation_feat_imp_absolute_explainer.AbsolutePermutationFeatureImportanceExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Absolute permutation-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-partial-dependence',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'PartialDependenceExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-individual-conditional-explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'IndividualConditionalExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.dai_pd_ice_explainer.DaiPdIceExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DAI PD/ICE explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ReportExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_markdown_explainer.MockMarkdownVegaExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Markdown with Vega explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-partial-dependence',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'PartialDependenceExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-individual-conditional-explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'IndividualConditionalExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_pd_explainer.MockPartialDependenceExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock PartialDependence explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_dt_explainer.MockDecisionTreeExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock DecisionTree explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ReportExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_markdown_pd_explainer.MockMarkdownPdExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Markdown explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_featimp_explainer.MockFeatureImportanceExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Feature Importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-scatter-plot',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalScatterPlotExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_scatter_plot_explainer.MockScatterPlotExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock ScatterPlot explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates_n_shapleys_explainer.MainSurrogatesAndShapleysExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogates and Shapleys explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.permutation_feat_imp_relative_explainer.RelativePermutationFeatureImportanceExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 'name': 'Relative permutation-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.orig_kernel_shap_explainer.OriginalKernelShapExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Original Kernel SHAP explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_pd_explainer.RandomForestPartialDependenceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF PD/ICE explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_feat_imp_explainer.RandomForestFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.original_feat_imp_explainer.OriginalFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Original feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalDtExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.dt_surrogate_explainer.DecisionTreeSurrogateExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DT Surrogate explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.klime_explainer.KLimeExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'k-LIME explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_loco_explainer.RandomForestLocoExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF LOCO explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.text.nlp_tokenizer_explainer.NlpTokenizerExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'NLP Tokenizer explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.text.nlp_loco_explainer.NlpLocoExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'NLP LOCO explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-auto-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'AutoReportExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.autoreport_explainer.AutoReportExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'AutoReport'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.transformed_feat_imp_explainer.TransformedFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Transformed feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.transformed_shapley_explainer.TransformedShapleyFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Transformed Shapley-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Morris Sensitivity Analysis'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_setup_explainer_28f31b80_contentexplainer.TestMockExSetupExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception setup() explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_fit_explainer_1b629753_contentexplainer.TestMockExFitExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception fit() explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_explain_explainer_4cbf4487_contentexplainer.TestMockExExplainExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception explain() explainer'}\n"
     ]
    }
   ],
   "source": [
    "# list available server recipes\n",
    "explainers = h2oai.list_explainers(\n",
    "    experiment_types=None, \n",
    "    explanation_scopes=None,\n",
    "    dai_model_key=None,\n",
    "    keywords=None,\n",
    "    explainer_filter=[]\n",
    ")\n",
    "\n",
    "for e in explainers:\n",
    "    pprint.pprint(e.dump())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-sensitivity-analysis',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'SaExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'SA explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-disparate-impact-analysis',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'DiaExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.dia_explainer.DiaExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DIA explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.permutation_feat_imp_absolute_explainer.AbsolutePermutationFeatureImportanceExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Absolute permutation-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-partial-dependence',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'PartialDependenceExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-individual-conditional-explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'IndividualConditionalExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.dai_pd_ice_explainer.DaiPdIceExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DAI PD/ICE explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ReportExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_markdown_explainer.MockMarkdownVegaExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Markdown with Vega explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-partial-dependence',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'PartialDependenceExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-individual-conditional-explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'IndividualConditionalExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_pd_explainer.MockPartialDependenceExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock PartialDependence explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_dt_explainer.MockDecisionTreeExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock DecisionTree explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ReportExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_markdown_pd_explainer.MockMarkdownPdExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Markdown explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_featimp_explainer.MockFeatureImportanceExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Feature Importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-scatter-plot',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalScatterPlotExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_scatter_plot_explainer.MockScatterPlotExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock ScatterPlot explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates_n_shapleys_explainer.MainSurrogatesAndShapleysExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogates and Shapleys explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.permutation_feat_imp_relative_explainer.RelativePermutationFeatureImportanceExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 'name': 'Relative permutation-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.orig_kernel_shap_explainer.OriginalKernelShapExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Original Kernel SHAP explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_pd_explainer.RandomForestPartialDependenceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF PD/ICE explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_feat_imp_explainer.RandomForestFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.original_feat_imp_explainer.OriginalFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Original feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalDtExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.dt_surrogate_explainer.DecisionTreeSurrogateExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DT Surrogate explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.klime_explainer.KLimeExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'k-LIME explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_loco_explainer.RandomForestLocoExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF LOCO explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.text.nlp_tokenizer_explainer.NlpTokenizerExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'NLP Tokenizer explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.text.nlp_loco_explainer.NlpLocoExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'NLP LOCO explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-auto-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'AutoReportExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.autoreport_explainer.AutoReportExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'AutoReport'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.transformed_feat_imp_explainer.TransformedFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Transformed feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.transformed_shapley_explainer.TransformedShapleyFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Transformed Shapley-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Morris Sensitivity Analysis'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_setup_explainer_28f31b80_contentexplainer.TestMockExSetupExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception setup() explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_fit_explainer_1b629753_contentexplainer.TestMockExFitExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception fit() explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_explain_explainer_4cbf4487_contentexplainer.TestMockExExplainExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception explain() explainer'}\n"
     ]
    }
   ],
   "source": [
    "# list server recipes for given experiment type\n",
    "explainers = h2oai.list_explainers(\n",
    "    experiment_types=[\"binomial\"], \n",
    "    explanation_scopes=None,\n",
    "    dai_model_key=None,\n",
    "    keywords=None,\n",
    "    explainer_filter=[]\n",
    ")\n",
    "\n",
    "for e in explainers:\n",
    "    pprint.pprint(e.dump())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-sensitivity-analysis',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'SaExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'SA explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-disparate-impact-analysis',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'DiaExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.dia_explainer.DiaExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DIA explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.permutation_feat_imp_absolute_explainer.AbsolutePermutationFeatureImportanceExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Absolute permutation-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-partial-dependence',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'PartialDependenceExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-individual-conditional-explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'IndividualConditionalExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.dai_pd_ice_explainer.DaiPdIceExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DAI PD/ICE explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ReportExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_markdown_explainer.MockMarkdownVegaExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Markdown with Vega explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-partial-dependence',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'PartialDependenceExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-individual-conditional-explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'IndividualConditionalExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_pd_explainer.MockPartialDependenceExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock PartialDependence explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_dt_explainer.MockDecisionTreeExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock DecisionTree explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ReportExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_markdown_pd_explainer.MockMarkdownPdExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Markdown explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_featimp_explainer.MockFeatureImportanceExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock Feature Importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-scatter-plot',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalScatterPlotExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.mock.mock_scatter_plot_explainer.MockScatterPlotExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Mock ScatterPlot explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates_n_shapleys_explainer.MainSurrogatesAndShapleysExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogates and Shapleys explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.permutation_feat_imp_relative_explainer.RelativePermutationFeatureImportanceExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 'name': 'Relative permutation-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.orig_kernel_shap_explainer.OriginalKernelShapExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Original Kernel SHAP explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_pd_explainer.RandomForestPartialDependenceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF PD/ICE explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_feat_imp_explainer.RandomForestFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.original_feat_imp_explainer.OriginalFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Original feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalDtExplanation',\n",
      "                   'scope': 'local'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.dt_surrogate_explainer.DecisionTreeSurrogateExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'DT Surrogate explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.klime_explainer.KLimeExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'k-LIME explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_loco_explainer.RandomForestLocoExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Surrogate RF LOCO explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.text.nlp_tokenizer_explainer.NlpTokenizerExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'NLP Tokenizer explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'local-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'LocalFeatImpExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.text.nlp_loco_explainer.NlpLocoExplainer',\n",
      " 'keywords': ['run-by-default'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'NLP LOCO explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-auto-report',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'AutoReportExplanation',\n",
      "                   'scope': 'global'},\n",
      "                  {'category': None,\n",
      "                   'explanation_type': 'global-work-dir-archive',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'WorkDirArchiveExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.autoreport_explainer.AutoReportExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'AutoReport'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.transformed_feat_imp_explainer.TransformedFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Transformed feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'local-proxy_explanation',\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'ProxyExplanation',\n",
      "                   'scope': 'local'}],\n",
      " 'id': 'h2oaicore.mli.byor.recipes.transformed_shapley_explainer.TransformedShapleyFeatureImportanceExplainer',\n",
      " 'keywords': ['run-by-default', 'proxy-explainer'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Transformed Shapley-based feature importance explainer'}\n",
      "{'can_explain': ['regression', 'binomial'],\n",
      " 'explanation_scopes': ['global_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-feature-importance',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalFeatImpExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer',\n",
      " 'keywords': [],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Morris Sensitivity Analysis'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_setup_explainer_28f31b80_contentexplainer.TestMockExSetupExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception setup() explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_fit_explainer_1b629753_contentexplainer.TestMockExFitExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception fit() explainer'}\n",
      "{'can_explain': ['regression', 'binomial', 'multinomial'],\n",
      " 'explanation_scopes': ['global_scope', 'local_scope'],\n",
      " 'explanations': [{'category': None,\n",
      "                   'explanation_type': 'global-decision-tree',\n",
      "                   'formats': [],\n",
      "                   'has_local': None,\n",
      "                   'name': 'GlobalDtExplanation',\n",
      "                   'scope': 'global'}],\n",
      " 'id': 'False_test_mock_ex_explain_explainer_4cbf4487_contentexplainer.TestMockExExplainExplainer',\n",
      " 'keywords': ['mock'],\n",
      " 'model_types': ['iid'],\n",
      " 'name': 'Test Mock Exception explain() explainer'}\n"
     ]
    }
   ],
   "source": [
    "# list server recipes compatible with given DAI model\n",
    "explainers = h2oai.list_explainers(\n",
    "    dai_model_key=MODEL_KEY, \n",
    "    experiment_types=None,\n",
    "    explanation_scopes=None,\n",
    "    keywords=None,\n",
    "    explainer_filter=[]\n",
    ")\n",
    "\n",
    "for e in explainers:\n",
    "    pprint.pprint(e.dump())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ad-hoc Run of Built-in Explainer Recipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run OOTB explainer recipe shipped w/ DAI server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sa_explainer_id = [explainer.id for explainer in explainers if \"SA explainer\" == explainer.name][0]\n",
    "sa_explainer_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'common_params': {'target_col': 'default payment next month',\n",
       "  'weight_col': None,\n",
       "  'prediction_col': None,\n",
       "  'drop_cols': None,\n",
       "  'sample_num_rows': None},\n",
       " 'model': {'key': '34edc42e-0152-11eb-a66d-207918bc8e4b', 'display_name': ''},\n",
       " 'dataset': {'key': 'dataset_33ab9348-0152-11eb-a66d-207918bc8e4b',\n",
       "  'display_name': ''},\n",
       " 'validset': {'key': None, 'display_name': ''},\n",
       " 'testset': {'key': None, 'display_name': ''},\n",
       " 'use_raw_features': False,\n",
       " 'config_overrides': None,\n",
       " 'sequential_execution': True,\n",
       " 'debug_model_errors': False,\n",
       " 'debug_model_errors_class': ''}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# prepare explaination parameters\n",
    "explanation_params=Client.build_common_dai_explainer_params(\n",
    "    target_col=target_col,\n",
    "    model_key=MODEL_KEY,\n",
    "    dataset_key=DATASET_KEY,\n",
    ")\n",
    "explanation_params.dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running OOTB explainer: h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'explainer_job_keys': ['423581b8-017b-11eb-a66d-207918bc8e4b'],\n",
       " 'mli_key': '423581b6-017b-11eb-a66d-207918bc8e4b',\n",
       " 'created': 1601291446.921501,\n",
       " 'duration': 0,\n",
       " 'status': -1,\n",
       " 'progress': 0.0}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run explainer\n",
    "explainer_id = sa_explainer_id\n",
    "\n",
    "print(f\"Running OOTB explainer: {explainer_id}\")\n",
    "\n",
    "run_job = h2oai.run_explainers(\n",
    "    explainers=[Explainer(\n",
    "        explainer_id=explainer_id,\n",
    "        explainer_params=\"\",\n",
    "    )],\n",
    "    params=explanation_params,\n",
    ")\n",
    "run_job.dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291446.8943293,\n",
      "                   'duration': 4.786961078643799,\n",
      "                   'entity': {'can_explain': ['regression', 'binomial'],\n",
      "                              'explanation_scopes': ['global_scope'],\n",
      "                              'explanations': [{'category': 'DAI MODEL',\n",
      "                                                'explanation_type': 'global-work-dir-archive',\n",
      "                                                'formats': ['application/zip'],\n",
      "                                                'has_local': None,\n",
      "                                                'name': 'SA Data ZIP Archive',\n",
      "                                                'scope': 'global'},\n",
      "                                               {'category': 'DAI MODEL',\n",
      "                                                'explanation_type': 'global-sensitivity-analysis',\n",
      "                                                'formats': ['text/plain'],\n",
      "                                                'has_local': None,\n",
      "                                                'name': 'Sensitivity Analysis '\n",
      "                                                        '(SA)',\n",
      "                                                'scope': 'global'}],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer',\n",
      "                              'keywords': ['run-by-default'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'SA explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Explainer 423581b8-017b-11eb-a66d-207918bc8e4b '\n",
      "                              'run successfully finished',\n",
      "                   'progress': 1.0,\n",
      "                   'status': 0},\n",
      " 'explainer_job_key': '423581b8-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '423581b6-017b-11eb-a66d-207918bc8e4b'}\n"
     ]
    }
   ],
   "source": [
    "# wait for explainer to finish\n",
    "explainer_job_statuses = h2oai.wait_for_explainers(run_job.mli_key)\n",
    "for job_status in explainer_job_statuses:\n",
    "    pprint.pprint(job_status.dump())\n",
    "    \n",
    "mli_key = job_status.mli_key\n",
    "explainer_job_key = job_status.explainer_job_key\n",
    "explainer_job = job_status.explainer_job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Recipe Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['regression', 'binomial']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get recipe result FORMATS/TYPES (representations of recipe output)\n",
    "explainer_job.entity.can_explain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['global_scope']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "explainer_job.entity.explanation_scopes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'category': 'DAI MODEL',\n",
      " 'explanation_type': 'global-work-dir-archive',\n",
      " 'formats': ['application/zip'],\n",
      " 'has_local': None,\n",
      " 'name': 'SA Data ZIP Archive',\n",
      " 'scope': 'global'}\n",
      "{'category': 'DAI MODEL',\n",
      " 'explanation_type': 'global-sensitivity-analysis',\n",
      " 'formats': ['text/plain'],\n",
      " 'has_local': None,\n",
      " 'name': 'Sensitivity Analysis (SA)',\n",
      " 'scope': 'global'}\n"
     ]
    }
   ],
   "source": [
    "for explanation in explainer_job.entity.explanations: pprint.pprint(explanation.dump())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Explanation global-work-dir-archive:\n",
      "  application/zip:\n",
      "    http://127.0.0.1:12345/files/h2oai/mli_experiment_423581b6-017b-11eb-a66d-207918bc8e4b/explainer_h2oaicore_mli_byor_recipes_sa_explainer_SaExplainer_423581b8-017b-11eb-a66d-207918bc8e4b/global_work_dir_archive/application_zip/explanation.zip\n",
      "Explanation global-sensitivity-analysis:\n",
      "  text/plain:\n",
      "    http://127.0.0.1:12345/files/h2oai/mli_experiment_423581b6-017b-11eb-a66d-207918bc8e4b/explainer_h2oaicore_mli_byor_recipes_sa_explainer_SaExplainer_423581b8-017b-11eb-a66d-207918bc8e4b/global_sensitivity_analysis/text_plain/explanation.txt\n"
     ]
    }
   ],
   "source": [
    "# choose the most suitable format (if more than one) and get the result\n",
    "BASE_URL = f\"{address}/files/\"\n",
    "\n",
    "for explanation in explainer_job.entity.explanations:\n",
    "    for e_format in explanation.formats:\n",
    "        server_path: str = h2oai.get_explainer_result_url_path(\n",
    "            mli_key=mli_key,\n",
    "            explainer_job_key=explainer_job_key,\n",
    "            explanation_type=explanation.explanation_type,\n",
    "            explanation_format=e_format\n",
    "        )\n",
    "        print(f\"Explanation {explanation.explanation_type}:\\n  {e_format}:\\n    {BASE_URL}{server_path}\")\n",
    "        \n",
    "        download_dir = \"/tmp\"\n",
    "        h2oai.download(server_path, download_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r-- 1 dvorka dvorka 1842252 Sep 28 13:10 /tmp/explanation.zip\r\n"
     ]
    }
   ],
   "source": [
    "!ls -l {download_dir}/explanation.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "URL from above can be used to **download** choosen recipe result representation. Explainer log can be downloaded from:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://127.0.0.1:12345/files/h2oai/mli_experiment_423581b6-017b-11eb-a66d-207918bc8e4b/explainer_h2oaicore_mli_byor_recipes_sa_explainer_SaExplainer_423581b8-017b-11eb-a66d-207918bc8e4b/log/explainer_run_423581b8-017b-11eb-a66d-207918bc8e4b.log\n"
     ]
    }
   ],
   "source": [
    "server_path = h2oai.get_explainer_run_log_url_path(\n",
    "    mli_key=mli_key,\n",
    "    explainer_job_key=explainer_job_key,\n",
    ")\n",
    "print(f\"{BASE_URL}{server_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ad-hoc Run of Custom Explainer Recipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running previously uploaded custom explainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running CUSTOM explainer: False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'explainer_job_keys': ['46260112-017b-11eb-a66d-207918bc8e4b'],\n",
       " 'mli_key': '46260110-017b-11eb-a66d-207918bc8e4b',\n",
       " 'created': 1601291453.5125122,\n",
       " 'duration': 0,\n",
       " 'status': -1,\n",
       " 'progress': 0.0}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run custom explainer - use previously uploaded recipe ID\n",
    "if uploaded_explainer_id:\n",
    "    explainer_id = uploaded_explainer_id\n",
    "else:\n",
    "    # explainer has been uploaded before\n",
    "    explainers = h2oai.list_explainers(\n",
    "        time_series=False, \n",
    "        dai_model_key=MODEL_KEY, \n",
    "        experiment_types=None,\n",
    "        explanation_scopes=None,\n",
    "        keywords=None,\n",
    "    )\n",
    "    for e in explainers:\n",
    "        if e.name == BYOR_EXPLAINER_NAME:\n",
    "            explainer_id = e.id\n",
    "\n",
    "print(f\"Running CUSTOM explainer: {explainer_id}\")\n",
    "\n",
    "run_job = h2oai.run_explainers(\n",
    "    explainers=[\n",
    "        Explainer(explainer_id=explainer_id, explainer_params=None)\n",
    "    ],\n",
    "    params=Client.build_common_dai_explainer_params(\n",
    "        target_col=target_col,\n",
    "        model_key=MODEL_KEY,\n",
    "        dataset_key=DATASET_KEY,\n",
    "    ),\n",
    ")\n",
    "run_job.dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291453.4908412,\n",
      "                   'duration': 3.8387792110443115,\n",
      "                   'entity': {'can_explain': ['regression', 'binomial'],\n",
      "                              'explanation_scopes': ['global_scope'],\n",
      "                              'explanations': [{'category': 'CUSTOM',\n",
      "                                                'explanation_type': 'global-feature-importance',\n",
      "                                                'formats': ['application/vnd.h2oai.json+datatable.jay',\n",
      "                                                            'application/json'],\n",
      "                                                'has_local': None,\n",
      "                                                'name': 'Morris Sensitivity '\n",
      "                                                        'Analysis',\n",
      "                                                'scope': 'global'}],\n",
      "                              'id': 'False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer',\n",
      "                              'keywords': [],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Morris Sensitivity Analysis'},\n",
      "                   'error': '',\n",
      "                   'message': 'Explainer 46260112-017b-11eb-a66d-207918bc8e4b '\n",
      "                              'run successfully finished',\n",
      "                   'progress': 1.0,\n",
      "                   'status': 0},\n",
      " 'explainer_job_key': '46260112-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '46260110-017b-11eb-a66d-207918bc8e4b'}\n"
     ]
    }
   ],
   "source": [
    "# wait for explainer to finish\n",
    "explainer_job_statuses = h2oai.wait_for_explainers(run_job.mli_key)\n",
    "for job_status in explainer_job_statuses:\n",
    "    pprint.pprint(job_status.dump())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://127.0.0.1:12345/files/h2oai/mli_experiment_46260110-017b-11eb-a66d-207918bc8e4b/explainer_False_morris_sensitivity_explainer_d935205d_contentexplainer_MorrisSensitivityLeExplainer_46260112-017b-11eb-a66d-207918bc8e4b/global_feature_importance/application_json/explanation.json\n"
     ]
    }
   ],
   "source": [
    "server_path = h2oai.get_explainer_result_url_path(\n",
    "    mli_key=job_status.mli_key,\n",
    "    explainer_job_key=job_status.explainer_job_key,\n",
    "    explanation_type='global-feature-importance',\n",
    "    explanation_format='application/json'\n",
    ")\n",
    "print(f\"{BASE_URL}{server_path}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "URL from above can be used to **download** choosen **custom** recipe result representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explain (Model) with All Compatible or Selected Explainers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.dia_explainer.DiaExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.permutation_feat_imp_absolute_explainer.AbsolutePermutationFeatureImportanceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.dai_pd_ice_explainer.DaiPdIceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.mock.mock_markdown_explainer.MockMarkdownVegaExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.mock.mock_pd_explainer.MockPartialDependenceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.mock.mock_dt_explainer.MockDecisionTreeExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.mock.mock_markdown_pd_explainer.MockMarkdownPdExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.mock.mock_featimp_explainer.MockFeatureImportanceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.mock.mock_scatter_plot_explainer.MockScatterPlotExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.surrogates_n_shapleys_explainer.MainSurrogatesAndShapleysExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.permutation_feat_imp_relative_explainer.RelativePermutationFeatureImportanceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.orig_kernel_shap_explainer.OriginalKernelShapExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.surrogates.rf_pd_explainer.RandomForestPartialDependenceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.surrogates.rf_feat_imp_explainer.RandomForestFeatureImportanceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.surrogates.original_feat_imp_explainer.OriginalFeatureImportanceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.surrogates.dt_surrogate_explainer.DecisionTreeSurrogateExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.surrogates.klime_explainer.KLimeExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.surrogates.rf_loco_explainer.RandomForestLocoExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.text.nlp_tokenizer_explainer.NlpTokenizerExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.text.nlp_loco_explainer.NlpLocoExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.autoreport_explainer.AutoReportExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.transformed_feat_imp_explainer.TransformedFeatureImportanceExplainer',\n",
       " 'h2oaicore.mli.byor.recipes.transformed_shapley_explainer.TransformedShapleyFeatureImportanceExplainer',\n",
       " 'False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer',\n",
       " 'False_test_mock_ex_setup_explainer_28f31b80_contentexplainer.TestMockExSetupExplainer',\n",
       " 'False_test_mock_ex_fit_explainer_1b629753_contentexplainer.TestMockExFitExplainer',\n",
       " 'False_test_mock_ex_explain_explainer_4cbf4487_contentexplainer.TestMockExExplainExplainer']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get IDs of previously listed recipes\n",
    "explainer_ids = [explainer.id for explainer in explainers]\n",
    "explainer_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All explainers:\n",
      "['h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer', 'h2oaicore.mli.byor.recipes.dia_explainer.DiaExplainer', 'h2oaicore.mli.byor.recipes.permutation_feat_imp_absolute_explainer.AbsolutePermutationFeatureImportanceExplainer', 'h2oaicore.mli.byor.recipes.dai_pd_ice_explainer.DaiPdIceExplainer', 'h2oaicore.mli.byor.recipes.mock.mock_markdown_explainer.MockMarkdownVegaExplainer', 'h2oaicore.mli.byor.recipes.mock.mock_pd_explainer.MockPartialDependenceExplainer', 'h2oaicore.mli.byor.recipes.mock.mock_dt_explainer.MockDecisionTreeExplainer', 'h2oaicore.mli.byor.recipes.mock.mock_markdown_pd_explainer.MockMarkdownPdExplainer', 'h2oaicore.mli.byor.recipes.mock.mock_featimp_explainer.MockFeatureImportanceExplainer', 'h2oaicore.mli.byor.recipes.mock.mock_scatter_plot_explainer.MockScatterPlotExplainer', 'h2oaicore.mli.byor.recipes.surrogates_n_shapleys_explainer.MainSurrogatesAndShapleysExplainer', 'h2oaicore.mli.byor.recipes.permutation_feat_imp_relative_explainer.RelativePermutationFeatureImportanceExplainer', 'h2oaicore.mli.byor.recipes.orig_kernel_shap_explainer.OriginalKernelShapExplainer', 'h2oaicore.mli.byor.recipes.surrogates.rf_pd_explainer.RandomForestPartialDependenceExplainer', 'h2oaicore.mli.byor.recipes.surrogates.rf_feat_imp_explainer.RandomForestFeatureImportanceExplainer', 'h2oaicore.mli.byor.recipes.surrogates.original_feat_imp_explainer.OriginalFeatureImportanceExplainer', 'h2oaicore.mli.byor.recipes.surrogates.dt_surrogate_explainer.DecisionTreeSurrogateExplainer', 'h2oaicore.mli.byor.recipes.surrogates.klime_explainer.KLimeExplainer', 'h2oaicore.mli.byor.recipes.surrogates.rf_loco_explainer.RandomForestLocoExplainer', 'h2oaicore.mli.byor.recipes.text.nlp_tokenizer_explainer.NlpTokenizerExplainer', 'h2oaicore.mli.byor.recipes.text.nlp_loco_explainer.NlpLocoExplainer', 'h2oaicore.mli.byor.recipes.autoreport_explainer.AutoReportExplainer', 'h2oaicore.mli.byor.recipes.transformed_feat_imp_explainer.TransformedFeatureImportanceExplainer', 'h2oaicore.mli.byor.recipes.transformed_shapley_explainer.TransformedShapleyFeatureImportanceExplainer', 'False_morris_sensitivity_explainer_d935205d_contentexplainer.MorrisSensitivityLeExplainer', 'False_test_mock_ex_setup_explainer_28f31b80_contentexplainer.TestMockExSetupExplainer', 'False_test_mock_ex_fit_explainer_1b629753_contentexplainer.TestMockExFitExplainer', 'False_test_mock_ex_explain_explainer_4cbf4487_contentexplainer.TestMockExExplainExplainer']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'explainer_job_keys': ['49ec5ea4-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5ea6-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5ea8-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eaa-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eac-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eae-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eb0-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eb2-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eb4-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eb6-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eb8-017b-11eb-a66d-207918bc8e4b',\n",
       "  '49ec5eba-017b-11eb-a66d-207918bc8e4b'],\n",
       " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b',\n",
       " 'created': 1601291460.452971,\n",
       " 'duration': 0,\n",
       " 'status': -1,\n",
       " 'progress': 0.0}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run explainers: list of IDs OR empty list\n",
    "#   - empty explainer IDs list means \"run all model COMPATIBLE explainers with default parameters\")\n",
    "\n",
    "print(f\"All explainers:\\n{explainer_ids}\")\n",
    "run_job: ExplainersRunJob = h2oai.run_explainers(\n",
    "    explainers=[],\n",
    "    params=Client.build_common_dai_explainer_params(\n",
    "        target_col=target_col,\n",
    "        model_key=MODEL_KEY,\n",
    "        dataset_key=DATASET_KEY,\n",
    "    ),\n",
    ")\n",
    "run_job.dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'progress': 0.1,\n",
       " 'h2oprogress': {'progress': 0.1, 'msg': '', 'done': False},\n",
       " 'mliprogress': {'progress': 0.1, 'msg': '', 'done': False},\n",
       " 'status': -1,\n",
       " 'error': '',\n",
       " 'message': 'Explainers running',\n",
       " 'entity': {'key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b',\n",
       "  'description': '49ec5ea2-017b-11eb-a66d-207918bc8e4b',\n",
       "  'tmp_dir': './tmp',\n",
       "  'scoring_package_path': '',\n",
       "  'binned_list': [],\n",
       "  'labels': [],\n",
       "  'parameters': {'dai_model': {'key': '34edc42e-0152-11eb-a66d-207918bc8e4b',\n",
       "    'display_name': ''},\n",
       "   'dataset': {'key': 'dataset_33ab9348-0152-11eb-a66d-207918bc8e4b',\n",
       "    'display_name': ''},\n",
       "   'target_col': 'default payment next month',\n",
       "   'prediction_col': None,\n",
       "   'use_raw_features': False,\n",
       "   'nfolds': -1,\n",
       "   'klime_cluster_col': '',\n",
       "   'weight_col': None,\n",
       "   'drop_cols': None,\n",
       "   'sample': False,\n",
       "   'sample_num_rows': -1,\n",
       "   'qbin_cols': [],\n",
       "   'qbin_count': -1,\n",
       "   'lime_method': '',\n",
       "   'dt_tree_depth': -1,\n",
       "   'config_overrides': '',\n",
       "   'vars_to_pdp': -1,\n",
       "   'dia_cols': [],\n",
       "   'testset': {'key': None, 'display_name': ''},\n",
       "   'debug_model_errors': False,\n",
       "   'debug_model_errors_class': ''},\n",
       "  'training_duration': 0.7639961242675781,\n",
       "  'log_file_path': '',\n",
       "  'lime_rc_csv_path': '',\n",
       "  'shapley_rc_csv_path': '',\n",
       "  'shapley_orig_rc_csv_path': '',\n",
       "  'subtask_keys': [{'name': '49ec5ea4-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5ea4-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5ea6-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5ea6-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5ea8-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5ea8-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eaa-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eaa-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eac-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eac-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eae-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eae-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eb0-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eb0-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eb2-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eb2-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eb4-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eb4-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eb6-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eb6-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eb8-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eb8-017b-11eb-a66d-207918bc8e4b']},\n",
       "   {'name': '49ec5eba-017b-11eb-a66d-207918bc8e4b',\n",
       "    'value': ['49ec5eba-017b-11eb-a66d-207918bc8e4b']}],\n",
       "  'dai_target_transformation': '',\n",
       "  'prediction_label': ''},\n",
       " 'created': 1601291459.8144472}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check interpretation job status (legacy RPC API)\n",
    "i_job: InterpretationJob = h2oai.get_interpretation_job(run_job.mli_key)\n",
    "# note per-explainer (subtask) ID and display name\n",
    "i_job.dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b',\n",
       " 'explainer_job_key': '49ec5ea4-017b-11eb-a66d-207918bc8e4b',\n",
       " 'explainer_job': {'progress': 0,\n",
       "  'status': -1,\n",
       "  'error': '',\n",
       "  'message': 'Entering sequential execution gate...',\n",
       "  'entity': {'id': 'h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer',\n",
       "   'name': 'SA explainer',\n",
       "   'model_types': ['iid'],\n",
       "   'can_explain': ['regression', 'binomial'],\n",
       "   'explanation_scopes': ['global_scope'],\n",
       "   'explanations': [],\n",
       "   'keywords': ['run-by-default']},\n",
       "  'created': 1601291459.8144472,\n",
       "  'duration': 0.6814770698547363,\n",
       "  'child_explainers_job_keys': []}}"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check particular sub-job status (existing RPC API reused)\n",
    "h2oai.get_explainer_job_status(run_job.mli_key, run_job.explainer_job_keys[0]).dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291459.8144472,\n",
      "                   'duration': 0.6814770698547363,\n",
      "                   'entity': {'can_explain': ['regression', 'binomial'],\n",
      "                              'explanation_scopes': ['global_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.sa_explainer.SaExplainer',\n",
      "                              'keywords': ['run-by-default'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'SA explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5ea4-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291459.8862343,\n",
      "                   'duration': 0.6159448623657227,\n",
      "                   'entity': {'can_explain': ['regression', 'binomial'],\n",
      "                              'explanation_scopes': ['global_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.dia_explainer.DiaExplainer',\n",
      "                              'keywords': ['run-by-default'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'DIA explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5ea6-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': ['49ec5eaa-017b-11eb-a66d-207918bc8e4b',\n",
      "                                                 '49ec5eac-017b-11eb-a66d-207918bc8e4b',\n",
      "                                                 '49ec5eae-017b-11eb-a66d-207918bc8e4b',\n",
      "                                                 '49ec5eb0-017b-11eb-a66d-207918bc8e4b',\n",
      "                                                 '49ec5eb2-017b-11eb-a66d-207918bc8e4b',\n",
      "                                                 '49ec5eb4-017b-11eb-a66d-207918bc8e4b',\n",
      "                                                 '49ec5eb6-017b-11eb-a66d-207918bc8e4b',\n",
      "                                                 '49ec5eb8-017b-11eb-a66d-207918bc8e4b'],\n",
      "                   'created': 1601291459.927952,\n",
      "                   'duration': 0.5900819301605225,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.surrogates_n_shapleys_explainer.MainSurrogatesAndShapleysExplainer',\n",
      "                              'keywords': ['run-by-default'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Surrogates and Shapleys explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5ea8-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291459.9655921,\n",
      "                   'duration': 0.5583939552307129,\n",
      "                   'entity': {'can_explain': ['regression', 'binomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_pd_explainer.RandomForestPartialDependenceExplainer',\n",
      "                              'keywords': ['run-by-default', 'proxy-explainer'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Surrogate RF PD/ICE explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eaa-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291460.0036705,\n",
      "                   'duration': 0.5254826545715332,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_feat_imp_explainer.RandomForestFeatureImportanceExplainer',\n",
      "                              'keywords': ['run-by-default', 'proxy-explainer'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Surrogate RF feature importance '\n",
      "                                      'explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eac-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291460.0425496,\n",
      "                   'duration': 0.4986147880554199,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.surrogates.original_feat_imp_explainer.OriginalFeatureImportanceExplainer',\n",
      "                              'keywords': ['run-by-default', 'proxy-explainer'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Original feature importance explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eae-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291460.0819962,\n",
      "                   'duration': 0.46430420875549316,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.surrogates.dt_surrogate_explainer.DecisionTreeSurrogateExplainer',\n",
      "                              'keywords': ['run-by-default'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'DT Surrogate explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eb0-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   'created': 1601291460.125687,\n",
      "                   'duration': 0.4347710609436035,\n",
      "                   'entity': {'can_explain': ['regression', 'binomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.surrogates.klime_explainer.KLimeExplainer',\n",
      "                              'keywords': ['run-by-default', 'proxy-explainer'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'k-LIME explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eb2-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291460.2780929,\n",
      "                   'duration': 0.28701353073120117,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.surrogates.rf_loco_explainer.RandomForestLocoExplainer',\n",
      "                              'keywords': ['run-by-default', 'proxy-explainer'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Surrogate RF LOCO explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eb4-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291460.3306832,\n",
      "                   'duration': 0.2857487201690674,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.transformed_feat_imp_explainer.TransformedFeatureImportanceExplainer',\n",
      "                              'keywords': ['run-by-default', 'proxy-explainer'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Transformed feature importance '\n",
      "                                      'explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eb6-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291460.3682995,\n",
      "                   'duration': 0.2549896240234375,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.transformed_shapley_explainer.TransformedShapleyFeatureImportanceExplainer',\n",
      "                              'keywords': ['run-by-default', 'proxy-explainer'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'Transformed Shapley-based feature '\n",
      "                                      'importance explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Entering sequential execution gate...',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eb8-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n",
      "{'explainer_job': {'child_explainers_job_keys': [],\n",
      "                   'created': 1601291460.4067953,\n",
      "                   'duration': 0.1488208770751953,\n",
      "                   'entity': {'can_explain': ['regression',\n",
      "                                              'binomial',\n",
      "                                              'multinomial'],\n",
      "                              'explanation_scopes': ['global_scope',\n",
      "                                                     'local_scope'],\n",
      "                              'explanations': [],\n",
      "                              'id': 'h2oaicore.mli.byor.recipes.dai_pd_ice_explainer.DaiPdIceExplainer',\n",
      "                              'keywords': ['run-by-default'],\n",
      "                              'model_types': ['iid'],\n",
      "                              'name': 'DAI PD/ICE explainer'},\n",
      "                   'error': '',\n",
      "                   'message': 'Waiting for a worker.',\n",
      "                   'progress': 0,\n",
      "                   'status': -1},\n",
      " 'explainer_job_key': '49ec5eba-017b-11eb-a66d-207918bc8e4b',\n",
      " 'mli_key': '49ec5ea2-017b-11eb-a66d-207918bc8e4b'}\n"
     ]
    }
   ],
   "source": [
    "# check sub-jobs statuses (existing RPC API reused)\n",
    "job_statuses = h2oai.get_explainer_job_statuses(run_job.mli_key, run_job.explainer_job_keys)\n",
    "for js in job_statuses:\n",
    "    pprint.pprint(js.dump())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<h2oai_client.messages.ExplainerJobStatus at 0x7fc29c8a0b00>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c8a07f0>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c8a0dd8>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c8a0c88>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c84b438>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c84b128>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c84b710>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c84b828>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c84b240>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c84b9b0>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c84ba58>,\n",
       " <h2oai_client.messages.ExplainerJobStatus at 0x7fc29c8a0f98>]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# wait for ALL explainers to finish\n",
    "explainer_statuses=h2oai.wait_for_explainers(run_job.mli_key)\n",
    "explainer_statuses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Explanation global-work-dir-archive:\n",
      "  application/zip:\n",
      "    http://127.0.0.1:12345/files/h2oai/mli_experiment_49ec5ea2-017b-11eb-a66d-207918bc8e4b/explainer_h2oaicore_mli_byor_recipes_sa_explainer_SaExplainer_49ec5ea4-017b-11eb-a66d-207918bc8e4b/global_work_dir_archive/application_zip/explanation.zip\n",
      "Explanation global-sensitivity-analysis:\n",
      "  text/plain:\n",
      "    http://127.0.0.1:12345/files/h2oai/mli_experiment_49ec5ea2-017b-11eb-a66d-207918bc8e4b/explainer_h2oaicore_mli_byor_recipes_sa_explainer_SaExplainer_49ec5ea4-017b-11eb-a66d-207918bc8e4b/global_sensitivity_analysis/text_plain/explanation.txt\n",
      "\n",
      "Downloaded explanations in /tmp/interpretation_run_388854:\n",
      "total 8\n",
      "drwxr-xr-x 2 dvorka dvorka 4096 Sep 28 13:16 explanation_753563\n",
      "drwxr-xr-x 2 dvorka dvorka 4096 Sep 28 13:16 explanation_809124\n"
     ]
    }
   ],
   "source": [
    "# download explanation type in desired format\n",
    "DOWNLOAD_DIR = f\"/tmp/interpretation_run_{randint(0,1_000_000)}\"\n",
    "\n",
    "explainer_job_key = explainer_statuses[0].explainer_job_key\n",
    "explanations = h2oai.list_explainer_results(explainer_job_key=explainer_job_key).explanations\n",
    "# explanations\n",
    "for explanation in explanations:\n",
    "    # explantion's formats\n",
    "    for explanation_format in explanation.formats:\n",
    "        # format's URL\n",
    "        result_path: str = h2oai.get_explainer_result_url_path(\n",
    "            mli_key=run_job.mli_key,\n",
    "            explainer_job_key=explainer_job_key,\n",
    "            explanation_type=explanation.explanation_type,\n",
    "            explanation_format=explanation_format,\n",
    "        )\n",
    "            \n",
    "        # where to download\n",
    "        EXPLANATION_DIR = f\"{DOWNLOAD_DIR}/explanation_{randint(0,1_000_000)}\"\n",
    "        os.makedirs(EXPLANATION_DIR, exist_ok=True)\n",
    "        \n",
    "        # download\n",
    "        h2oai.download(result_path, EXPLANATION_DIR)\n",
    "\n",
    "        print(\n",
    "            f\"Explanation {explanation.explanation_type}:\\n\"\n",
    "            f\"  {explanation_format}:\\n\"\n",
    "            f\"    {BASE_URL}{result_path}\"\n",
    "        )\n",
    "        \n",
    "print(f\"\\nDownloaded explanations in {DOWNLOAD_DIR}:\")\n",
    "!ls -l {DOWNLOAD_DIR}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "h2oai",
   "language": "python",
   "name": "h2oai"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}