notebooks/training_with_hdfs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0e814db9",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# HugeCTR training with HDFS example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfa44b99",
   "metadata": {},
   "source": [
    "## Overview:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7166fb3",
   "metadata": {},
   "source": [
    "In version v3.4, we introduced the support of HDFS. Users can now move their data and model files from HDFS to local filesystem through our API to do HugeCTR training. And after training, users can choose to dump the trained parameeters and optimizer states into HDFS. In this example notebook, we are going to demonstrate the end to end procedure of training with HDFS."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bcaa8ee",
   "metadata": {},
   "source": [
    "## Table of contents:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f2fa417",
   "metadata": {},
   "source": [
    "-  [Installation](#1)\n",
    "   * [Get HugeCTR from NGC](#11)\n",
    "   * [Build HugeCTR from Source Code](#12)\n",
    "-  [Hadoop Preparation](#2)\n",
    "   * [Download and Install Hadoop](#21)\n",
    "   * [HDFS Conifiguration](#22)\n",
    "   * [Launch HDFS cluster](#23)\n",
    "-  [Wide&Deep Demo](#3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "025dd2e5",
   "metadata": {},
   "source": [
    "<a id=\"1\"></a>\n",
    "## 1. Installation\n",
    "\n",
    "<a id=\"11\"></a>\n",
    "### 1.1 Get HugeCTR from NGC\n",
    "The HugeCTR Python module is preinstalled in the [Merlin Training Container](https://ngc.nvidia.com/catalog/containers/nvidia:merlin:merlin-training): `nvcr.io/nvidia/merlin/merlin-training:22.03`.\n",
    "\n",
    "You can check the existence of required libraries by running the following Python code after launching this container.\n",
    "```bash\n",
    "$ python3 -c \"import hugectr\"\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d45548b1",
   "metadata": {},
   "source": [
    "<a id=\"12\"></a>\n",
    "### 1.2 Build HugeCTR from Source Code\n",
    "\n",
    "If you want to build HugeCTR from the source code instead of using the NGC container, please refer to the [How to Start Your Development](../../docs/hugectr_contributor_guide.md#how-to-start-your-development)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "479ba8e7",
   "metadata": {},
   "source": [
    "<a id=\"2\"></a>\n",
    "## 2. Hadoop Preparation\n",
    "\n",
    "<a id=\"21\"></a>\n",
    "### 2.1 Download and Install Hadoop\n",
    "\n",
    "Download JDK first:\n",
    "```bash\n",
    "wget https://download.java.net/java/GA/jdk16.0.2/d4a915d82b4c4fbb9bde534da945d746/7/GPL/openjdk-16.0.2_linux-x64_bin.tar.gz\n",
    "tar -zxvf openjdk-16.0.2_linux-x64_bin.tar.gz\n",
    "mv jdk-16.0.2 /usr/local\n",
    "```\n",
    "\n",
    "Set Java Environmental variables:\n",
    "```bash\n",
    "export JAVA_HOME=/usr/local/jdk-16.0.2\n",
    "export JRE_HOME=${JAVA_HOME}/jre\n",
    "export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib\n",
    "export PATH=.:${JAVA_HOME}/bin:$PATH\n",
    "```\n",
    "\n",
    "Download and install Hadoop:\n",
    "```bash\n",
    "wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz\n",
    "tar -zxvf hadoop-3.3.1.tar.gz\n",
    "mv hadoop-3.3.1 /usr/local\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d6d48b7",
   "metadata": {},
   "source": [
    "<a id=\"22\"></a>\n",
    "### 2.2 HDFS configuration\n",
    "Set Hadoop Environmental variables:\n",
    "```bash\n",
    "export HDFS_NAMENODE_USER=root\n",
    "export HDFS_DATANODE_USER=root\n",
    "export HDFS_SECONDARYNAMENODE_USER=root\n",
    "export YARN_RESOURCEMANAGER_USER=root\n",
    "export YARN_NODEMANAGER_USER=root\n",
    "echo ‘export JAVA_HOME=/usr/local/jdk-16.0.2’ >> /usr/local/hadoop/etc/hadoop/hadoop-env.sh\n",
    "```\n",
    "\n",
    "`core-site.xml` config:\n",
    "\n",
    "```xml\n",
    "<property>\n",
    "    <name>fs.defaultFS</name>\n",
    "    <value>hdfs://namenode:9000</value>\n",
    "</property>\n",
    "<property>\n",
    "    <name>hadoop.tmp.dir</name>\n",
    "    <value>/usr/local/hadoop/tmp</value>\n",
    "</property>\n",
    "```\n",
    "\n",
    "`hdfs-site.xml` for name node:\n",
    "```xml\n",
    "<property>\n",
    "     <name>dfs.replication</name>\n",
    "     <value>4</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.namenode.name.dir</name>\n",
    "     <value>file:/usr/local/hadoop/hdfs/name</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>         \n",
    "     <value>true</value>\n",
    "</property>\n",
    "<property>\n",
    "    <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>\n",
    "    <value>NEVER</value>\n",
    "</property>\n",
    "```\n",
    "\n",
    "`hdfs-site.xml` for data node:\n",
    "```xml\n",
    "<property>\n",
    "     <name>dfs.replication</name>\n",
    "     <value>4</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.datanode.data.dir</name>\n",
    "     <value>file:/usr/local/hadoop/hdfs/data</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>         \n",
    "     <value>true</value>\n",
    "</property>\n",
    "<property>\n",
    "    <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>\n",
    "    <value>NEVER</value>\n",
    "</property>\n",
    "```\n",
    "\n",
    "`workers` for all node:\n",
    "```bash\n",
    "worker1\n",
    "worker2\n",
    "worker3\n",
    "worker4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e4e0b59",
   "metadata": {},
   "source": [
    "<a id=\"23\"></a>\n",
    "### 2.3 Launch HDFS Cluster\n",
    "\n",
    "Enable ssh connection:\n",
    "```bash\n",
    "ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa\n",
    "cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys\n",
    "/etc/init.d/ssh start\n",
    "```\n",
    "\n",
    "For Name Node:\n",
    "```bash\n",
    "/usr/local/hadoop/bin/hdfs namenode -format\n",
    "```\n",
    "\n",
    "For Data Node:\n",
    "```bash\n",
    "/usr/local/hadoop/bin/hdfs datanode -format\n",
    "```\n",
    "\n",
    "Inside Name Node:\n",
    "```bash\n",
    "/usr/local/hadoop/sbin/start-dfs.sh\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f26b776f",
   "metadata": {},
   "source": [
    "<a id=\"3\"></a>\n",
    "## 3. Wide&Deep Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "399717d1",
   "metadata": {},
   "source": [
    "In your `nvcr.io/nvidia/merlin/merlin-training:22.03` docker:\n",
    "Make sure you have installed hadoop and set the proper environmental variables as instructed in the last sesstion.\n",
    "\n",
    "When compile HugeCTR, please make sure you set `DENABLE_HDFS` as `ON`.\n",
    "\n",
    "Then run `export CLASSPATH=$(hadoop classpath --glob)` first to link the required .jar\n",
    "Then, make sure that we have the model files your hadoop cluster and provide the correct links to the model files.\n",
    "\n",
    "Now you can run the sample below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "02510bae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_with_hdfs.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_with_hdfs.py\n",
    "import hugectr\n",
    "from mpi4py import MPI\n",
    "from hugectr.data import DataSource, DataSourceParams\n",
    "\n",
    "data_source_params = DataSourceParams(\n",
    "    use_hdfs = True, # Whether use HDFS to save model files\n",
    "    namenode = 'namenode', # HDFS namenode hostname\n",
    "    port = 9000, # HDFS port\n",
    "    \n",
    "    hdfs_train_source = '/data/train/',\n",
    "    hdfs_train_filelist = '/data/file_list.txt',\n",
    "    hdfs_eval_source = '/data/val/',\n",
    "    hdfs_eval_filelist = '/data/file_list_test.txt',\n",
    "    \n",
    "    # Make sure the followings are the correct path to the model files in your hadoop cluster\n",
    "    hdfs_dense_model = '/model/wdl/_dense_1000.model',\n",
    "    hdfs_dense_opt_states = '/model/wdl/_opt_dense_1000.model',\n",
    "    hdfs_sparse_model = ['/model/wdl/0_sparse_1000.model', '/model/wdl/1_sparse_1000.model'],\n",
    "    hdfs_sparse_opt_states = ['/model/wdl/0_opt_sparse_1000.model', '/model/wdl/1_opt_sparse_1000.model'],         \n",
    "                                              \n",
    "    local_train_source = './wdl_norm/train/', # Make sure this is the correct path to your train source\n",
    "    local_train_filelist = './wdl_norm/file_list.txt', # Make sure this is the correct path to your train filelist\n",
    "    local_eval_source = './wdl_norm/val/', # Make sure this is the correct path to your validation source\n",
    "    local_eval_filelist = './wdl_norm/file_list_test.txt', # Make sure this is the correct path to your validation filelist\n",
    "    local_dense_model = './_dense_1000.model',\n",
    "    local_dense_opt_states = './_opt_dense_1000.model',\n",
    "    local_sparse_model = ['./0_sparse_1000.model', './1_sparse_1000.model'],\n",
    "    local_sparse_opt_states = ['./0_opt_sparse_1000.model', './1_opt_sparse_1000.model'],\n",
    "    \n",
    "    hdfs_model_home = '/model/wdl/',\n",
    "    local_model_home = '/model/wdl/'\n",
    ")\n",
    "\n",
    "solver = hugectr.CreateSolver(max_eval_batches = 1280,\n",
    "                              batchsize_eval = 1024,\n",
    "                              batchsize = 1024,\n",
    "                              lr = 0.001,\n",
    "                              vvgpu = [[0]],\n",
    "                              repeat_dataset = True)\n",
    "reader = hugectr.DataReaderParams(data_reader_type = hugectr.DataReaderType_t.Norm,\n",
    "                                  source = [data_source_params.local_train_filelist],\n",
    "                                  eval_source = data_source_params.local_eval_filelist,\n",
    "                                  check_type = hugectr.Check_t.Sum)\n",
    "optimizer = hugectr.CreateOptimizer(optimizer_type = hugectr.Optimizer_t.Adam,\n",
    "                                    update_type = hugectr.Update_t.Global,\n",
    "                                    beta1 = 0.9,\n",
    "                                    beta2 = 0.999,\n",
    "                                    epsilon = 0.0000001)\n",
    "model = hugectr.Model(solver, reader, optimizer)\n",
    "model.add(hugectr.Input(label_dim = 1, label_name = \"label\",\n",
    "                        dense_dim = 13, dense_name = \"dense\",\n",
    "                        data_reader_sparse_param_array = \n",
    "                        # the total number of slots should be equal to data_generator_params.num_slot\n",
    "                        [hugectr.DataReaderSparseParam(\"wide_data\", 2, True, 1),\n",
    "                        hugectr.DataReaderSparseParam(\"deep_data\", 1, True, 26)]))\n",
    "model.add(hugectr.SparseEmbedding(embedding_type = hugectr.Embedding_t.LocalizedSlotSparseEmbeddingHash, \n",
    "                            workspace_size_per_gpu_in_mb = 69,\n",
    "                            embedding_vec_size = 1,\n",
    "                            combiner = \"sum\",\n",
    "                            sparse_embedding_name = \"sparse_embedding2\",\n",
    "                            bottom_name = \"wide_data\",\n",
    "                            optimizer = optimizer))\n",
    "model.add(hugectr.SparseEmbedding(embedding_type = hugectr.Embedding_t.LocalizedSlotSparseEmbeddingHash, \n",
    "                            workspace_size_per_gpu_in_mb = 1074,\n",
    "                            embedding_vec_size = 16,\n",
    "                            combiner = \"sum\",\n",
    "                            sparse_embedding_name = \"sparse_embedding1\",\n",
    "                            bottom_name = \"deep_data\",\n",
    "                            optimizer = optimizer))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Reshape,\n",
    "                            bottom_names = [\"sparse_embedding1\"],\n",
    "                            top_names = [\"reshape1\"],\n",
    "                            leading_dim=416))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Reshape,\n",
    "                            bottom_names = [\"sparse_embedding2\"],\n",
    "                            top_names = [\"reshape2\"],\n",
    "                            leading_dim=1))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Concat,\n",
    "                            bottom_names = [\"reshape1\", \"dense\"],\n",
    "                            top_names = [\"concat1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"concat1\"],\n",
    "                            top_names = [\"fc1\"],\n",
    "                            num_output=1024))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc1\"],\n",
    "                            top_names = [\"relu1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Dropout,\n",
    "                            bottom_names = [\"relu1\"],\n",
    "                            top_names = [\"dropout1\"],\n",
    "                            dropout_rate=0.5))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"dropout1\"],\n",
    "                            top_names = [\"fc2\"],\n",
    "                            num_output=1024))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc2\"],\n",
    "                            top_names = [\"relu2\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Dropout,\n",
    "                            bottom_names = [\"relu2\"],\n",
    "                            top_names = [\"dropout2\"],\n",
    "                            dropout_rate=0.5))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"dropout2\"],\n",
    "                            top_names = [\"fc3\"],\n",
    "                            num_output=1))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Add,\n",
    "                            bottom_names = [\"fc3\", \"reshape2\"],\n",
    "                            top_names = [\"add1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.BinaryCrossEntropyLoss,\n",
    "                            bottom_names = [\"add1\", \"label\"],\n",
    "                            top_names = [\"loss\"]))\n",
    "model.compile()\n",
    "model.summary()\n",
    "\n",
    "model.load_dense_weights('', data_source_params)\n",
    "model.load_dense_optimizer_states('', data_source_params)\n",
    "model.load_sparse_weights(['', ''], data_source_params)\n",
    "model.load_sparse_optimizer_states(['', ''], data_source_params)\n",
    "\n",
    "model.fit(max_iter = 1020, display = 200, eval_interval = 500, snapshot = 1000, data_source_params = data_source_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b29f1042",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HugeCTR Version: 3.3\n",
      "====================================================Model Init=====================================================\n",
      "[HCTR][09:00:54][WARNING][RK0][main]: The model name is not specified when creating the solver.\n",
      "[HCTR][09:00:54][INFO][RK0][main]: Global seed is 1285686508\n",
      "[HCTR][09:00:55][INFO][RK0][main]: Device to NUMA mapping:\n",
      "  GPU 0 ->  node 0\n",
      "[HCTR][09:00:56][WARNING][RK0][main]: Peer-to-peer access cannot be fully enabled.\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Start all2all warmup\n",
      "[HCTR][09:00:56][INFO][RK0][main]: End all2all warmup\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Using All-reduce algorithm: NCCL\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Device 0: Tesla V100-PCIE-32GB\n",
      "[HCTR][09:00:56][INFO][RK0][main]: num of DataReader workers: 12\n",
      "[HCTR][09:00:56][INFO][RK0][main]: max_vocabulary_size_per_gpu_=6029312\n",
      "[HCTR][09:00:56][INFO][RK0][main]: max_vocabulary_size_per_gpu_=5865472\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Graph analysis to resolve tensor dependency\n",
      "===================================================Model Compile===================================================\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][09:01:00][INFO][RK0][main]: Starting AUC NCCL warm-up\n",
      "[HCTR][09:01:00][INFO][RK0][main]: Warm-up done\n",
      "[HCTR][09:01:00][INFO][RK0][main]: ===================================================Model Summary===================================================\n",
      "label                                   Dense                         Sparse                        \n",
      "label                                   dense                          wide_data,deep_data           \n",
      "(None, 1)                               (None, 13)                              \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "Layer Type                              Input Name                    Output Name                   Output Shape                  \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "LocalizedSlotSparseEmbeddingHash        wide_data                     sparse_embedding2             (None, 1, 1)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "LocalizedSlotSparseEmbeddingHash        deep_data                     sparse_embedding1             (None, 26, 16)                \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding1             reshape1                      (None, 416)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding2             reshape2                      (None, 1)                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Concat                                  reshape1                      concat1                       (None, 429)                   \n",
      "                                        dense                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            concat1                       fc1                           (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc1                           relu1                         (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Dropout                                 relu1                         dropout1                      (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            dropout1                      fc2                           (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc2                           relu2                         (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Dropout                                 relu2                         dropout2                      (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            dropout2                      fc3                           (None, 1)                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Add                                     fc3                           add1                          (None, 1)                     \n",
      "                                        reshape2                                                                                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "BinaryCrossEntropyLoss                  add1                          loss                                                        \n",
      "                                        label                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "2022-02-23 09:01:00,548 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "[HDFS][INFO]: Read file /model/wdl/_dense_1000.model successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/_opt_dense_1000.model successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading dense opt states: \n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse model: /model/wdl/0_sparse_1000.model\n",
      "[HDFS][INFO]: Read file /model/wdl/0_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/0_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/0_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Start to upload embedding table file to GPUs, total loop_num: 128\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse model: /model/wdl/1_sparse_1000.model\n",
      "[HDFS][INFO]: Read file /model/wdl/1_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/1_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/1_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Start to upload embedding table file to GPUs, total loop_num: 518\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse optimizer states: /model/wdl/0_opt_sparse_1000.model\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse optimizer states: /model/wdl/1_opt_sparse_1000.model\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Done\n",
      "=====================================================Model Fit=====================================================\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Use non-epoch mode with number of iterations: 1020\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Training batchsize: 1024, evaluation batchsize: 1024\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Evaluation interval: 500, snapshot interval: 1000\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Dense network trainable: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Sparse embedding sparse_embedding1 trainable: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Sparse embedding sparse_embedding2 trainable: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Use mixed precision: False, scaler: 1.000000, use cuda graph: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: lr: 0.001000, warmup_steps: 1, end_lr: 0.000000\n",
      "[HCTR][09:01:02][INFO][RK0][main]: decay_start: 0, decay_steps: 1, decay_power: 2.000000\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Training source file: ./wdl_norm/file_list.txt\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Evaluation source file: ./wdl_norm/file_list_test.txt\n",
      "[HCTR][09:01:04][INFO][RK0][main]: Iter: 200 Time(200 iters): 1.12465s Loss: 0.632464 lr:0.001\n",
      "[HCTR][09:01:05][INFO][RK0][main]: Iter: 400 Time(200 iters): 1.03567s Loss: 0.612515 lr:0.001\n",
      "[HCTR][09:01:06][INFO][RK0][main]: Evaluation, AUC: 0.499877\n",
      "[HCTR][09:01:06][INFO][RK0][main]: Eval Time for 1280 iters: 0.647875s\n",
      "[HCTR][09:01:06][INFO][RK0][main]: Iter: 600 Time(200 iters): 1.68717s Loss: 0.625102 lr:0.001\n",
      "[HCTR][09:01:07][INFO][RK0][main]: Iter: 800 Time(200 iters): 1.03752s Loss: 0.608092 lr:0.001\n",
      "[HCTR][09:01:08][INFO][RK0][main]: Iter: 1000 Time(200 iters): 1.03691s Loss: 0.688194 lr:0.001\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Evaluation, AUC: 0.500383\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Eval Time for 1280 iters: 0.650671s\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Dump hash table from GPU0\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write hash table <key,value> pairs to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Dump hash table from GPU0\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write hash table <key,value> pairs to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Dumping sparse weights to files, successful\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:10][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:10][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:11][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:11][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Dumping sparse optimzer states to files, successful\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/_dense_1000.model successfully!\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Dumping dense weights to HDFS, successful\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/_opt_dense_1000.model successfully!\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Dumping dense optimizer states to HDFS, successful\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Finish 1020 iterations with batchsize: 1024 in 9.82s.\n"
     ]
    }
   ],
   "source": [
    "!python train_with_hdfs.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc771003",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}