tfoptflow/pwcnet_train_lg-6-2-cyclic-chairsthingsmix.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PWC-Net-large model training (with cyclical learning rate schedule)\n",
    "=======================================================\n",
    "\n",
    "In this notebook, we:\n",
    "- Use a PWC-Net-large model (with dense and residual connections), 6 level pyramid, uspample level 2 by 4 as the final flow prediction\n",
    "- Train the model on a mix of the `FlyingChairs` and `FlyingThings3DHalfRes` dataset using a Cyclic<sub>short</sub> schedule of our own\n",
    "- The Cyclic<sub>short</sub> schedule oscillates between `5e-04` and `1e-05` for 200,000 steps\n",
    "\n",
    "Below, look for `TODO` references and customize this notebook based on your own machine setup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\n",
    "[2018a]<a name=\"2018a\"></a> Sun et al. 2018. PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume. [[arXiv]](https://arxiv.org/abs/1709.02371) [[web]](http://research.nvidia.com/publication/2018-02_PWC-Net%3A-CNNs-for) [[PyTorch (Official)]](https://github.com/NVlabs/PWC-Net/tree/master/PyTorch) [[Caffe (Official)]](https://github.com/NVlabs/PWC-Net/tree/master/Caffe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "pwcnet_train.ipynb\n",
    "\n",
    "PWC-Net model training.\n",
    "\n",
    "Written by Phil Ferriere\n",
    "\n",
    "Licensed under the MIT License (see LICENSE for details)\n",
    "\n",
    "Tensorboard:\n",
    "    [win] tensorboard --logdir=E:\\\\repos\\\\tf-optflow\\\\tfoptflow\\\\pwcnet-lg-6-2-cyclic-chairsthingsmix\n",
    "    [ubu] tensorboard --logdir=/media/EDrive/repos/tf-optflow/tfoptflow/pwcnet-lg-6-2-cyclic-chairsthingsmix\n",
    "\"\"\"\n",
    "from __future__ import absolute_import, division, print_function\n",
    "import sys\n",
    "from copy import deepcopy\n",
    "\n",
    "from dataset_base import _DEFAULT_DS_TRAIN_OPTIONS\n",
    "from dataset_flyingchairs import FlyingChairsDataset\n",
    "from dataset_flyingthings3d import FlyingThings3DHalfResDataset\n",
    "from dataset_mixer import MixedDataset\n",
    "from model_pwcnet import ModelPWCNet, _DEFAULT_PWCNET_TRAIN_OPTIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Set this first!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: You MUST set dataset_root to the correct path on your machine!\n",
    "if sys.platform.startswith(\"win\"):\n",
    "    _DATASET_ROOT = 'E:/datasets/'\n",
    "else:\n",
    "    _DATASET_ROOT = '/media/EDrive/datasets/'\n",
    "_FLYINGCHAIRS_ROOT = _DATASET_ROOT + 'FlyingChairs_release'\n",
    "_FLYINGTHINGS3DHALFRES_ROOT = _DATASET_ROOT + 'FlyingThings3D_HalfRes'\n",
    "    \n",
    "# TODO: You MUST adjust the settings below based on the number of GPU(s) used for training\n",
    "# Set controller device and devices\n",
    "# A one-gpu setup would be something like controller='/device:GPU:0' and gpu_devices=['/device:GPU:0']\n",
    "# Here, we use a dual-GPU setup, as shown below\n",
    "gpu_devices = ['/device:GPU:0', '/device:GPU:1']\n",
    "controller = '/device:CPU:0'\n",
    "\n",
    "# TODO: You MUST adjust this setting below based on the amount of memory on your GPU(s)\n",
    "# Batch size\n",
    "batch_size = 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-train on `FlyingChairs+FlyingThings3DHalfRes` mix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: You MUST set the batch size based on the capabilities of your GPU(s) \n",
    "#  Load train dataset\n",
    "ds_opts = deepcopy(_DEFAULT_DS_TRAIN_OPTIONS)\n",
    "ds_opts['in_memory'] = False                          # Too many samples to keep in memory at once, so don't preload them\n",
    "ds_opts['aug_type'] = 'heavy'                         # Apply all supported augmentations\n",
    "ds_opts['batch_size'] = batch_size * len(gpu_devices) # Use a multiple of 8; here, 16 for dual-GPU mode (Titan X & 1080 Ti)\n",
    "ds_opts['crop_preproc'] = (256, 448)                  # Crop to a smaller input size\n",
    "ds1 = FlyingChairsDataset(mode='train_with_val', ds_root=_FLYINGCHAIRS_ROOT, options=ds_opts)\n",
    "ds_opts['type'] = 'into_future'\n",
    "ds2 = FlyingThings3DHalfResDataset(mode='train_with_val', ds_root=_FLYINGTHINGS3DHALFRES_ROOT, options=ds_opts)\n",
    "ds = MixedDataset(mode='train_with_val', datasets=[ds1, ds2], options=ds_opts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dataset Configuration:\n",
      "  verbose              False\n",
      "  in_memory            False\n",
      "  crop_preproc         (256, 448)\n",
      "  scale_preproc        None\n",
      "  input_channels       3\n",
      "  tb_test_imgs         False\n",
      "  random_seed          1969\n",
      "  val_split            0.03\n",
      "  aug_type             heavy\n",
      "  aug_labels           True\n",
      "  fliplr               0.5\n",
      "  flipud               0.5\n",
      "  translate            (0.5, 0.05)\n",
      "  scale                (0.5, 0.05)\n",
      "  batch_size           16\n",
      "  type                 into_future\n",
      "  mode                 train_with_val\n",
      "  train size           41282\n",
      "  val size             1230\n"
     ]
    }
   ],
   "source": [
    "# Display dataset configuration\n",
    "ds.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start from the default options\n",
    "nn_opts = deepcopy(_DEFAULT_PWCNET_TRAIN_OPTIONS)\n",
    "nn_opts['verbose'] = True\n",
    "nn_opts['ckpt_dir'] = './pwcnet-lg-6-2-cyclic-chairsthingsmix/'\n",
    "nn_opts['batch_size'] = ds_opts['batch_size']\n",
    "nn_opts['x_shape'] = [2, ds_opts['crop_preproc'][0], ds_opts['crop_preproc'][1], 3]\n",
    "nn_opts['y_shape'] = [ds_opts['crop_preproc'][0], ds_opts['crop_preproc'][1], 2]\n",
    "nn_opts['use_tf_data'] = True # Use tf.data reader\n",
    "nn_opts['gpu_devices'] = gpu_devices\n",
    "nn_opts['controller'] = controller\n",
    "\n",
    "# Use the PWC-Net-large model in quarter-resolution mode\n",
    "nn_opts['use_dense_cx'] = True\n",
    "nn_opts['use_res_cx'] = True\n",
    "nn_opts['pyr_lvls'] = 6\n",
    "nn_opts['flow_pred_lvl'] = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the learning rate schedule. This schedule is for a single GPU using a batch size of 8.\n",
    "# Below,we adjust the schedule to the size of the batch and the number of GPUs.\n",
    "nn_opts['lr_policy'] = 'cyclic'\n",
    "nn_opts['cyclic_lr_max'] = 5e-04 # Anything higher will generate NaNs\n",
    "nn_opts['cyclic_lr_base'] = 1e-05\n",
    "nn_opts['cyclic_lr_stepsize'] = 20000\n",
    "nn_opts['max_steps'] = 200000\n",
    "\n",
    "# Below,we adjust the schedule to the size of the batch and our number of GPUs (2).\n",
    "nn_opts['cyclic_lr_stepsize'] /= len(gpu_devices)\n",
    "nn_opts['max_steps'] /= len(gpu_devices)\n",
    "nn_opts['cyclic_lr_stepsize'] = int(nn_opts['cyclic_lr_stepsize'] / (float(ds_opts['batch_size']) / 8))\n",
    "nn_opts['max_steps'] = int(nn_opts['max_steps'] / (float(ds_opts['batch_size']) / 8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building model towers...\n",
      "  Building tower_0...\n",
      "WARNING:tensorflow:From e:\\toolkits.win\\anaconda3-5.2.0\\envs\\dlwin36\\lib\\site-packages\\tensorflow\\python\\keras\\initializers.py:104: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with distribution=normal is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "`normal` is a deprecated alias for `truncated_normal`\n",
      "  ...tower_0 built.\n",
      "  Building tower_1...\n",
      "  ...tower_1 built.\n",
      "... model towers built.\n",
      "Initializing model with random values for initial training...\n",
      "\n",
      "... model initialized\n",
      "\n",
      "Model Configuration:\n",
      "  verbose                True\n",
      "  ckpt_dir               ./pwcnet-lg-6-2-cyclic-chairsthingsmix/\n",
      "  max_to_keep            10\n",
      "  x_dtype                <dtype: 'float32'>\n",
      "  x_shape                [2, 256, 448, 3]\n",
      "  y_dtype                <dtype: 'float32'>\n",
      "  y_shape                [256, 448, 2]\n",
      "  train_mode             train\n",
      "  display_step           100\n",
      "  snapshot_step          1000\n",
      "  val_step               1000\n",
      "  val_batch_size         -1\n",
      "  tb_val_imgs            pyramid\n",
      "  tb_test_imgs           None\n",
      "  gpu_devices            ['/device:GPU:0', '/device:GPU:1']\n",
      "  controller             /device:CPU:0\n",
      "  use_tf_data            True\n",
      "  batch_size             16\n",
      "  lr_policy              cyclic\n",
      "  max_steps              50000\n",
      "  cyclic_lr_max          0.0005\n",
      "  cyclic_lr_base         1e-05\n",
      "  cyclic_lr_stepsize     5000\n",
      "  loss_fn                loss_multiscale\n",
      "  alphas                 [0.32, 0.08, 0.02, 0.01, 0.005, 0.0025]\n",
      "  gamma                  0.0004\n",
      "  q                      1.0\n",
      "  epsilon                0.0\n",
      "  pyr_lvls               6\n",
      "  flow_pred_lvl          2\n",
      "  search_range           4\n",
      "  use_dense_cx           True\n",
      "  use_res_cx             True\n",
      "  mode                   train_with_val\n",
      "  trainable params       14079050\n"
     ]
    }
   ],
   "source": [
    "# Instantiate the model and display the model configuration\n",
    "nn = ModelPWCNet(mode='train_with_val', options=nn_opts, dataset=ds)\n",
    "nn.print_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start training from scratch...\n",
      "2018-09-09 11:41:51 Iter 100 [Train]: loss=217.31, epe=17.64, lr=0.000020, samples/sec=17.9, sec/step=1.791, eta=1 day, 0:49:50\n",
      "2018-09-09 11:43:33 Iter 200 [Train]: loss=226.02, epe=18.37, lr=0.000030, samples/sec=34.5, sec/step=0.926, eta=12:48:48\n",
      "2018-09-09 11:45:15 Iter 300 [Train]: loss=222.64, epe=18.10, lr=0.000039, samples/sec=34.6, sec/step=0.926, eta=12:47:08\n",
      "2018-09-09 11:46:57 Iter 400 [Train]: loss=215.81, epe=17.55, lr=0.000049, samples/sec=34.5, sec/step=0.929, eta=12:47:36\n",
      "2018-09-09 11:48:40 Iter 500 [Train]: loss=221.65, epe=18.05, lr=0.000059, samples/sec=34.4, sec/step=0.930, eta=12:47:18\n",
      "2018-09-09 11:50:23 Iter 600 [Train]: loss=224.30, epe=18.27, lr=0.000069, samples/sec=34.4, sec/step=0.931, eta=12:46:16\n",
      "2018-09-09 11:52:06 Iter 700 [Train]: loss=221.96, epe=18.09, lr=0.000079, samples/sec=34.3, sec/step=0.933, eta=12:46:52\n",
      "2018-09-09 11:53:49 Iter 800 [Train]: loss=216.28, epe=17.54, lr=0.000088, samples/sec=34.0, sec/step=0.942, eta=12:52:22\n",
      "2018-09-09 11:55:32 Iter 900 [Train]: loss=195.18, epe=15.37, lr=0.000098, samples/sec=34.2, sec/step=0.937, eta=12:46:39\n",
      "2018-09-09 11:57:14 Iter 1000 [Train]: loss=192.24, epe=14.85, lr=0.000108, samples/sec=34.1, sec/step=0.937, eta=12:45:23\n",
      "2018-09-09 11:57:38 Iter 1000 [Val]: loss=174.56, epe=13.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-1000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-1000\n",
      "2018-09-09 11:59:41 Iter 1100 [Train]: loss=186.71, epe=14.28, lr=0.000118, samples/sec=34.1, sec/step=0.939, eta=12:45:22\n",
      "2018-09-09 12:01:26 Iter 1200 [Train]: loss=190.22, epe=14.50, lr=0.000128, samples/sec=33.6, sec/step=0.952, eta=12:54:07\n",
      "2018-09-09 12:03:10 Iter 1300 [Train]: loss=188.21, epe=14.31, lr=0.000137, samples/sec=33.8, sec/step=0.946, eta=12:47:45\n",
      "2018-09-09 12:04:54 Iter 1400 [Train]: loss=182.59, epe=13.70, lr=0.000147, samples/sec=33.8, sec/step=0.946, eta=12:46:10\n",
      "2018-09-09 12:06:37 Iter 1500 [Train]: loss=178.50, epe=13.38, lr=0.000157, samples/sec=33.9, sec/step=0.943, eta=12:42:32\n",
      "2018-09-09 12:08:21 Iter 1600 [Train]: loss=175.48, epe=13.27, lr=0.000167, samples/sec=33.9, sec/step=0.945, eta=12:42:21\n",
      "2018-09-09 12:10:05 Iter 1700 [Train]: loss=166.71, epe=12.55, lr=0.000177, samples/sec=33.9, sec/step=0.943, eta=12:39:03\n",
      "2018-09-09 12:11:49 Iter 1800 [Train]: loss=160.04, epe=12.01, lr=0.000186, samples/sec=33.9, sec/step=0.945, eta=12:39:06\n",
      "2018-09-09 12:13:33 Iter 1900 [Train]: loss=157.06, epe=11.84, lr=0.000196, samples/sec=33.8, sec/step=0.947, eta=12:39:11\n",
      "2018-09-09 12:15:17 Iter 2000 [Train]: loss=158.88, epe=12.10, lr=0.000206, samples/sec=33.8, sec/step=0.947, eta=12:37:32\n",
      "2018-09-09 12:15:33 Iter 2000 [Val]: loss=148.82, epe=11.54\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-2000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-2000\n",
      "2018-09-09 12:17:34 Iter 2100 [Train]: loss=150.35, epe=11.40, lr=0.000216, samples/sec=33.8, sec/step=0.946, eta=12:35:18\n",
      "2018-09-09 12:19:18 Iter 2200 [Train]: loss=151.75, epe=11.58, lr=0.000226, samples/sec=33.7, sec/step=0.949, eta=12:35:50\n",
      "2018-09-09 12:21:02 Iter 2300 [Train]: loss=141.94, epe=10.70, lr=0.000235, samples/sec=33.6, sec/step=0.952, eta=12:36:59\n",
      "2018-09-09 12:22:47 Iter 2400 [Train]: loss=139.20, epe=10.50, lr=0.000245, samples/sec=33.5, sec/step=0.954, eta=12:36:41\n",
      "2018-09-09 12:24:31 Iter 2500 [Train]: loss=141.65, epe=10.70, lr=0.000255, samples/sec=33.7, sec/step=0.951, eta=12:32:30\n",
      "2018-09-09 12:26:15 Iter 2600 [Train]: loss=144.34, epe=10.89, lr=0.000265, samples/sec=33.8, sec/step=0.948, eta=12:28:53\n",
      "2018-09-09 12:27:59 Iter 2700 [Train]: loss=137.21, epe=10.31, lr=0.000275, samples/sec=33.8, sec/step=0.946, eta=12:26:09\n",
      "2018-09-09 12:29:43 Iter 2800 [Train]: loss=143.58, epe=10.81, lr=0.000284, samples/sec=33.8, sec/step=0.946, eta=12:23:58\n",
      "2018-09-09 12:31:27 Iter 2900 [Train]: loss=138.50, epe=10.37, lr=0.000294, samples/sec=33.9, sec/step=0.944, eta=12:21:03\n",
      "2018-09-09 12:33:10 Iter 3000 [Train]: loss=138.61, epe=10.41, lr=0.000304, samples/sec=33.8, sec/step=0.946, eta=12:21:01\n",
      "2018-09-09 12:33:26 Iter 3000 [Val]: loss=132.08, epe=10.05\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-3000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-3000\n",
      "2018-09-09 12:35:28 Iter 3100 [Train]: loss=136.65, epe=10.22, lr=0.000314, samples/sec=33.9, sec/step=0.944, eta=12:17:56\n",
      "2018-09-09 12:37:12 Iter 3200 [Train]: loss=133.00, epe=9.91, lr=0.000324, samples/sec=33.7, sec/step=0.949, eta=12:20:01\n",
      "2018-09-09 12:38:56 Iter 3300 [Train]: loss=131.42, epe=9.77, lr=0.000333, samples/sec=33.8, sec/step=0.947, eta=12:16:55\n",
      "2018-09-09 12:40:40 Iter 3400 [Train]: loss=131.46, epe=9.79, lr=0.000343, samples/sec=33.8, sec/step=0.947, eta=12:15:36\n",
      "2018-09-09 12:42:24 Iter 3500 [Train]: loss=135.52, epe=10.09, lr=0.000353, samples/sec=33.8, sec/step=0.947, eta=12:14:17\n",
      "2018-09-09 12:44:08 Iter 3600 [Train]: loss=126.06, epe=9.32, lr=0.000363, samples/sec=33.7, sec/step=0.948, eta=12:13:16\n",
      "2018-09-09 12:45:52 Iter 3700 [Train]: loss=131.88, epe=9.76, lr=0.000373, samples/sec=33.7, sec/step=0.948, eta=12:11:44\n",
      "2018-09-09 12:47:37 Iter 3800 [Train]: loss=123.93, epe=9.12, lr=0.000382, samples/sec=33.7, sec/step=0.949, eta=12:10:29\n",
      "2018-09-09 12:49:21 Iter 3900 [Train]: loss=125.46, epe=9.25, lr=0.000392, samples/sec=33.8, sec/step=0.946, eta=12:07:10\n",
      "2018-09-09 12:51:04 Iter 4000 [Train]: loss=126.72, epe=9.35, lr=0.000402, samples/sec=33.9, sec/step=0.945, eta=12:04:33\n",
      "2018-09-09 12:51:20 Iter 4000 [Val]: loss=125.64, epe=9.42\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-4000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-4000\n",
      "2018-09-09 12:53:21 Iter 4100 [Train]: loss=128.11, epe=9.43, lr=0.000412, samples/sec=34.0, sec/step=0.940, eta=11:59:18\n",
      "2018-09-09 12:55:05 Iter 4200 [Train]: loss=137.04, epe=10.15, lr=0.000422, samples/sec=33.8, sec/step=0.947, eta=12:02:55\n",
      "2018-09-09 12:56:49 Iter 4300 [Train]: loss=128.71, epe=9.51, lr=0.000431, samples/sec=33.8, sec/step=0.948, eta=12:02:08\n",
      "2018-09-09 12:58:33 Iter 4400 [Train]: loss=120.50, epe=8.83, lr=0.000441, samples/sec=33.8, sec/step=0.948, eta=12:00:10\n",
      "2018-09-09 13:00:18 Iter 4500 [Train]: loss=128.16, epe=9.45, lr=0.000451, samples/sec=33.7, sec/step=0.948, eta=11:59:02\n",
      "2018-09-09 13:02:02 Iter 4600 [Train]: loss=124.97, epe=9.16, lr=0.000461, samples/sec=33.7, sec/step=0.949, eta=11:58:00\n",
      "2018-09-09 13:03:46 Iter 4700 [Train]: loss=125.30, epe=9.21, lr=0.000471, samples/sec=33.7, sec/step=0.949, eta=11:56:50\n",
      "2018-09-09 13:05:31 Iter 4800 [Train]: loss=121.44, epe=8.85, lr=0.000480, samples/sec=33.8, sec/step=0.947, eta=11:53:08\n",
      "2018-09-09 13:07:15 Iter 4900 [Train]: loss=126.89, epe=9.29, lr=0.000490, samples/sec=33.7, sec/step=0.951, eta=11:54:44\n",
      "2018-09-09 13:09:00 Iter 5000 [Train]: loss=124.55, epe=9.09, lr=0.000500, samples/sec=33.5, sec/step=0.954, eta=11:55:27\n",
      "2018-09-09 13:09:16 Iter 5000 [Val]: loss=125.92, epe=9.42\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-5000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-5000\n",
      "2018-09-09 13:11:16 Iter 5100 [Train]: loss=124.35, epe=9.12, lr=0.000490, samples/sec=33.8, sec/step=0.946, eta=11:47:41\n",
      "2018-09-09 13:13:01 Iter 5200 [Train]: loss=121.77, epe=8.85, lr=0.000480, samples/sec=33.6, sec/step=0.952, eta=11:50:40\n",
      "2018-09-09 13:14:45 Iter 5300 [Train]: loss=118.93, epe=8.62, lr=0.000471, samples/sec=33.6, sec/step=0.952, eta=11:49:18\n",
      "2018-09-09 13:16:29 Iter 5400 [Train]: loss=122.16, epe=8.89, lr=0.000461, samples/sec=33.7, sec/step=0.948, eta=11:44:58\n",
      "2018-09-09 13:18:13 Iter 5500 [Train]: loss=117.15, epe=8.48, lr=0.000451, samples/sec=33.7, sec/step=0.950, eta=11:44:40\n",
      "2018-09-09 13:19:58 Iter 5600 [Train]: loss=119.41, epe=8.62, lr=0.000441, samples/sec=33.7, sec/step=0.950, eta=11:43:05\n",
      "2018-09-09 13:21:42 Iter 5700 [Train]: loss=120.36, epe=8.70, lr=0.000431, samples/sec=33.7, sec/step=0.950, eta=11:41:22\n",
      "2018-09-09 13:23:27 Iter 5800 [Train]: loss=113.58, epe=8.17, lr=0.000422, samples/sec=33.7, sec/step=0.950, eta=11:39:57\n",
      "2018-09-09 13:25:11 Iter 5900 [Train]: loss=115.20, epe=8.31, lr=0.000412, samples/sec=33.6, sec/step=0.953, eta=11:40:44\n",
      "2018-09-09 13:26:56 Iter 6000 [Train]: loss=111.48, epe=7.99, lr=0.000402, samples/sec=33.6, sec/step=0.953, eta=11:38:42\n",
      "2018-09-09 13:27:12 Iter 6000 [Val]: loss=113.44, epe=8.35\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-6000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-6000\n",
      "2018-09-09 13:29:14 Iter 6100 [Train]: loss=117.18, epe=8.45, lr=0.000392, samples/sec=33.6, sec/step=0.951, eta=11:35:49\n",
      "2018-09-09 13:31:00 Iter 6200 [Train]: loss=113.58, epe=8.13, lr=0.000382, samples/sec=33.4, sec/step=0.958, eta=11:39:18\n",
      "2018-09-09 13:32:45 Iter 6300 [Train]: loss=111.54, epe=7.96, lr=0.000373, samples/sec=33.4, sec/step=0.958, eta=11:38:06\n",
      "2018-09-09 13:34:30 Iter 6400 [Train]: loss=111.70, epe=7.96, lr=0.000363, samples/sec=33.5, sec/step=0.956, eta=11:34:25\n",
      "2018-09-09 13:36:15 Iter 6500 [Train]: loss=117.75, epe=8.44, lr=0.000353, samples/sec=33.5, sec/step=0.956, eta=11:33:06\n",
      "2018-09-09 13:37:59 Iter 6600 [Train]: loss=110.17, epe=7.81, lr=0.000343, samples/sec=33.5, sec/step=0.956, eta=11:31:45\n",
      "2018-09-09 13:39:44 Iter 6700 [Train]: loss=106.66, epe=7.53, lr=0.000333, samples/sec=33.6, sec/step=0.952, eta=11:27:07\n",
      "2018-09-09 13:41:28 Iter 6800 [Train]: loss=107.93, epe=7.62, lr=0.000324, samples/sec=33.6, sec/step=0.952, eta=11:25:39\n",
      "2018-09-09 13:43:13 Iter 6900 [Train]: loss=108.47, epe=7.65, lr=0.000314, samples/sec=33.5, sec/step=0.955, eta=11:26:18\n",
      "2018-09-09 13:44:58 Iter 7000 [Train]: loss=110.03, epe=7.80, lr=0.000304, samples/sec=33.5, sec/step=0.955, eta=11:24:42\n",
      "2018-09-09 13:45:14 Iter 7000 [Val]: loss=107.60, epe=7.80\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-7000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-7000\n",
      "2018-09-09 13:47:14 Iter 7100 [Train]: loss=108.27, epe=7.66, lr=0.000294, samples/sec=33.7, sec/step=0.950, eta=11:19:12\n",
      "2018-09-09 13:49:00 Iter 7200 [Train]: loss=109.52, epe=7.71, lr=0.000284, samples/sec=33.4, sec/step=0.958, eta=11:23:16\n",
      "2018-09-09 13:50:45 Iter 7300 [Train]: loss=110.25, epe=7.77, lr=0.000275, samples/sec=33.5, sec/step=0.957, eta=11:20:43\n",
      "2018-09-09 13:52:30 Iter 7400 [Train]: loss=111.57, epe=7.93, lr=0.000265, samples/sec=33.4, sec/step=0.959, eta=11:20:38\n",
      "2018-09-09 13:54:15 Iter 7500 [Train]: loss=108.43, epe=7.59, lr=0.000255, samples/sec=33.5, sec/step=0.956, eta=11:17:00\n",
      "2018-09-09 13:56:00 Iter 7600 [Train]: loss=102.03, epe=7.10, lr=0.000245, samples/sec=33.4, sec/step=0.958, eta=11:16:42\n",
      "2018-09-09 13:57:44 Iter 7700 [Train]: loss=101.29, epe=7.05, lr=0.000235, samples/sec=33.5, sec/step=0.954, eta=11:12:55\n",
      "2018-09-09 13:59:29 Iter 7800 [Train]: loss=104.02, epe=7.28, lr=0.000226, samples/sec=33.6, sec/step=0.953, eta=11:10:22\n",
      "2018-09-09 14:01:14 Iter 7900 [Train]: loss=100.87, epe=6.97, lr=0.000216, samples/sec=33.4, sec/step=0.959, eta=11:12:37\n",
      "2018-09-09 14:02:58 Iter 8000 [Train]: loss=104.67, epe=7.29, lr=0.000206, samples/sec=33.5, sec/step=0.954, eta=11:07:47\n",
      "2018-09-09 14:03:14 Iter 8000 [Val]: loss=101.71, epe=7.25\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-8000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-8000\n",
      "2018-09-09 14:05:15 Iter 8100 [Train]: loss=95.40, epe=6.52, lr=0.000196, samples/sec=33.6, sec/step=0.954, eta=11:06:03\n",
      "2018-09-09 14:07:01 Iter 8200 [Train]: loss=100.64, epe=6.95, lr=0.000186, samples/sec=33.3, sec/step=0.961, eta=11:09:39\n",
      "2018-09-09 14:08:46 Iter 8300 [Train]: loss=102.35, epe=7.09, lr=0.000177, samples/sec=33.3, sec/step=0.962, eta=11:08:19\n",
      "2018-09-09 14:10:32 Iter 8400 [Train]: loss=103.01, epe=7.11, lr=0.000167, samples/sec=33.2, sec/step=0.963, eta=11:07:41\n",
      "2018-09-09 14:12:17 Iter 8500 [Train]: loss=101.24, epe=7.01, lr=0.000157, samples/sec=33.2, sec/step=0.964, eta=11:06:55\n",
      "2018-09-09 14:14:03 Iter 8600 [Train]: loss=98.54, epe=6.78, lr=0.000147, samples/sec=33.2, sec/step=0.963, eta=11:04:14\n",
      "2018-09-09 14:15:49 Iter 8700 [Train]: loss=102.52, epe=7.11, lr=0.000137, samples/sec=33.3, sec/step=0.960, eta=11:00:46\n",
      "2018-09-09 14:17:34 Iter 8800 [Train]: loss=100.07, epe=6.89, lr=0.000128, samples/sec=33.3, sec/step=0.961, eta=10:59:40\n",
      "2018-09-09 14:19:20 Iter 8900 [Train]: loss=102.38, epe=7.10, lr=0.000118, samples/sec=33.2, sec/step=0.965, eta=11:01:13\n",
      "2018-09-09 14:21:06 Iter 9000 [Train]: loss=99.01, epe=6.81, lr=0.000108, samples/sec=33.2, sec/step=0.964, eta=10:58:41\n",
      "2018-09-09 14:21:22 Iter 9000 [Val]: loss=100.44, epe=7.13\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-9000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-9000\n",
      "2018-09-09 14:23:24 Iter 9100 [Train]: loss=99.45, epe=6.80, lr=0.000098, samples/sec=33.4, sec/step=0.958, eta=10:52:49\n",
      "2018-09-09 14:25:10 Iter 9200 [Train]: loss=95.54, epe=6.51, lr=0.000088, samples/sec=33.3, sec/step=0.962, eta=10:54:10\n",
      "2018-09-09 14:26:55 Iter 9300 [Train]: loss=96.29, epe=6.58, lr=0.000079, samples/sec=33.2, sec/step=0.962, eta=10:52:52\n",
      "2018-09-09 14:28:41 Iter 9400 [Train]: loss=93.97, epe=6.41, lr=0.000069, samples/sec=33.2, sec/step=0.965, eta=10:52:53\n",
      "2018-09-09 14:30:27 Iter 9500 [Train]: loss=98.78, epe=6.77, lr=0.000059, samples/sec=33.2, sec/step=0.964, eta=10:50:48\n",
      "2018-09-09 14:32:12 Iter 9600 [Train]: loss=92.89, epe=6.31, lr=0.000049, samples/sec=33.2, sec/step=0.962, eta=10:48:03\n",
      "2018-09-09 14:33:58 Iter 9700 [Train]: loss=94.35, epe=6.41, lr=0.000039, samples/sec=33.3, sec/step=0.961, eta=10:45:44\n",
      "2018-09-09 14:35:43 Iter 9800 [Train]: loss=90.01, epe=6.07, lr=0.000030, samples/sec=33.4, sec/step=0.959, eta=10:42:27\n",
      "2018-09-09 14:37:29 Iter 9900 [Train]: loss=94.80, epe=6.41, lr=0.000020, samples/sec=33.2, sec/step=0.965, eta=10:44:45\n",
      "2018-09-09 14:39:14 Iter 10000 [Train]: loss=94.99, epe=6.45, lr=0.000010, samples/sec=33.3, sec/step=0.961, eta=10:40:54\n",
      "2018-09-09 14:39:30 Iter 10000 [Val]: loss=95.81, epe=6.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-10000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-10000\n",
      "2018-09-09 14:41:35 Iter 10100 [Train]: loss=95.65, epe=6.52, lr=0.000015, samples/sec=33.2, sec/step=0.965, eta=10:41:51\n",
      "2018-09-09 14:43:21 Iter 10200 [Train]: loss=92.88, epe=6.29, lr=0.000020, samples/sec=33.0, sec/step=0.971, eta=10:43:54\n",
      "2018-09-09 14:45:08 Iter 10300 [Train]: loss=95.51, epe=6.50, lr=0.000025, samples/sec=33.0, sec/step=0.970, eta=10:42:02\n",
      "2018-09-09 14:46:54 Iter 10400 [Train]: loss=89.01, epe=5.98, lr=0.000030, samples/sec=32.9, sec/step=0.974, eta=10:42:46\n",
      "2018-09-09 14:48:40 Iter 10500 [Train]: loss=95.92, epe=6.54, lr=0.000034, samples/sec=33.0, sec/step=0.968, eta=10:37:27\n",
      "2018-09-09 14:50:27 Iter 10600 [Train]: loss=97.56, epe=6.67, lr=0.000039, samples/sec=33.0, sec/step=0.971, eta=10:37:38\n",
      "2018-09-09 14:52:13 Iter 10700 [Train]: loss=93.39, epe=6.34, lr=0.000044, samples/sec=32.9, sec/step=0.972, eta=10:36:54\n",
      "2018-09-09 14:54:00 Iter 10800 [Train]: loss=94.70, epe=6.45, lr=0.000049, samples/sec=32.9, sec/step=0.973, eta=10:36:00\n",
      "2018-09-09 14:55:47 Iter 10900 [Train]: loss=92.23, epe=6.23, lr=0.000054, samples/sec=32.9, sec/step=0.973, eta=10:33:55\n",
      "2018-09-09 14:57:33 Iter 11000 [Train]: loss=94.77, epe=6.41, lr=0.000059, samples/sec=33.0, sec/step=0.969, eta=10:30:01\n",
      "2018-09-09 14:57:49 Iter 11000 [Val]: loss=91.67, epe=6.39\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-11000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-11000\n",
      "2018-09-09 14:59:52 Iter 11100 [Train]: loss=92.57, epe=6.27, lr=0.000064, samples/sec=33.1, sec/step=0.967, eta=10:26:56\n",
      "2018-09-09 15:01:39 Iter 11200 [Train]: loss=96.58, epe=6.58, lr=0.000069, samples/sec=32.9, sec/step=0.973, eta=10:29:18\n",
      "2018-09-09 15:03:26 Iter 11300 [Train]: loss=92.92, epe=6.30, lr=0.000074, samples/sec=32.7, sec/step=0.978, eta=10:30:29\n",
      "2018-09-09 15:05:13 Iter 11400 [Train]: loss=92.08, epe=6.25, lr=0.000079, samples/sec=32.9, sec/step=0.974, eta=10:26:32\n",
      "2018-09-09 15:07:00 Iter 11500 [Train]: loss=96.03, epe=6.59, lr=0.000083, samples/sec=32.9, sec/step=0.974, eta=10:24:42\n",
      "2018-09-09 15:08:46 Iter 11600 [Train]: loss=92.28, epe=6.28, lr=0.000088, samples/sec=32.9, sec/step=0.974, eta=10:23:23\n",
      "2018-09-09 15:10:33 Iter 11700 [Train]: loss=95.56, epe=6.53, lr=0.000093, samples/sec=32.9, sec/step=0.973, eta=10:20:56\n",
      "2018-09-09 15:12:20 Iter 11800 [Train]: loss=94.79, epe=6.50, lr=0.000098, samples/sec=32.8, sec/step=0.977, eta=10:21:46\n",
      "2018-09-09 15:14:06 Iter 11900 [Train]: loss=92.70, epe=6.37, lr=0.000103, samples/sec=32.8, sec/step=0.975, eta=10:19:05\n",
      "2018-09-09 15:15:52 Iter 12000 [Train]: loss=92.93, epe=6.36, lr=0.000108, samples/sec=33.0, sec/step=0.970, eta=10:14:28\n",
      "2018-09-09 15:16:09 Iter 12000 [Val]: loss=96.79, epe=6.82\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-12000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-12000\n",
      "2018-09-09 15:18:11 Iter 12100 [Train]: loss=92.80, epe=6.37, lr=0.000113, samples/sec=33.0, sec/step=0.969, eta=10:12:14\n",
      "2018-09-09 15:19:58 Iter 12200 [Train]: loss=96.73, epe=6.67, lr=0.000118, samples/sec=32.8, sec/step=0.976, eta=10:15:11\n",
      "2018-09-09 15:21:45 Iter 12300 [Train]: loss=91.44, epe=6.29, lr=0.000123, samples/sec=32.8, sec/step=0.975, eta=10:12:43\n",
      "2018-09-09 15:23:32 Iter 12400 [Train]: loss=92.84, epe=6.39, lr=0.000128, samples/sec=32.7, sec/step=0.979, eta=10:13:18\n",
      "2018-09-09 15:25:19 Iter 12500 [Train]: loss=91.49, epe=6.30, lr=0.000133, samples/sec=32.8, sec/step=0.976, eta=10:10:17\n",
      "2018-09-09 15:27:06 Iter 12600 [Train]: loss=95.37, epe=6.62, lr=0.000137, samples/sec=32.8, sec/step=0.975, eta=10:07:48\n",
      "2018-09-09 15:28:53 Iter 12700 [Train]: loss=95.86, epe=6.67, lr=0.000142, samples/sec=32.7, sec/step=0.979, eta=10:08:49\n",
      "2018-09-09 15:30:40 Iter 12800 [Train]: loss=90.61, epe=6.25, lr=0.000147, samples/sec=32.8, sec/step=0.974, eta=10:04:11\n",
      "2018-09-09 15:32:27 Iter 12900 [Train]: loss=94.21, epe=6.51, lr=0.000152, samples/sec=32.8, sec/step=0.977, eta=10:03:55\n",
      "2018-09-09 15:34:14 Iter 13000 [Train]: loss=92.21, epe=6.40, lr=0.000157, samples/sec=32.6, sec/step=0.981, eta=10:04:40\n",
      "2018-09-09 15:34:30 Iter 13000 [Val]: loss=91.34, epe=6.47\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-13000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-13000\n",
      "2018-09-09 15:36:34 Iter 13100 [Train]: loss=99.26, epe=6.92, lr=0.000162, samples/sec=32.8, sec/step=0.976, eta=10:00:28\n",
      "2018-09-09 15:38:21 Iter 13200 [Train]: loss=93.50, epe=6.47, lr=0.000167, samples/sec=32.6, sec/step=0.982, eta=10:02:19\n",
      "2018-09-09 15:40:09 Iter 13300 [Train]: loss=95.39, epe=6.63, lr=0.000172, samples/sec=32.6, sec/step=0.983, eta=10:01:03\n",
      "2018-09-09 15:41:56 Iter 13400 [Train]: loss=94.29, epe=6.55, lr=0.000177, samples/sec=32.6, sec/step=0.982, eta=9:59:13\n",
      "2018-09-09 15:43:43 Iter 13500 [Train]: loss=94.32, epe=6.54, lr=0.000182, samples/sec=32.7, sec/step=0.979, eta=9:55:24\n",
      "2018-09-09 15:45:31 Iter 13600 [Train]: loss=86.50, epe=5.92, lr=0.000186, samples/sec=32.6, sec/step=0.982, eta=9:55:29\n",
      "2018-09-09 15:47:18 Iter 13700 [Train]: loss=92.65, epe=6.42, lr=0.000191, samples/sec=32.7, sec/step=0.978, eta=9:51:33\n",
      "2018-09-09 15:49:06 Iter 13800 [Train]: loss=89.57, epe=6.16, lr=0.000196, samples/sec=32.6, sec/step=0.981, eta=9:51:36\n",
      "2018-09-09 15:50:53 Iter 13900 [Train]: loss=91.31, epe=6.31, lr=0.000201, samples/sec=32.6, sec/step=0.981, eta=9:50:09\n",
      "2018-09-09 15:52:40 Iter 14000 [Train]: loss=96.47, epe=6.73, lr=0.000206, samples/sec=32.6, sec/step=0.982, eta=9:49:02\n",
      "2018-09-09 15:52:57 Iter 14000 [Val]: loss=95.17, epe=6.74\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-14000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-14000\n",
      "2018-09-09 15:55:01 Iter 14100 [Train]: loss=90.16, epe=6.23, lr=0.000211, samples/sec=32.7, sec/step=0.977, eta=9:44:44\n",
      "2018-09-09 15:56:49 Iter 14200 [Train]: loss=91.51, epe=6.33, lr=0.000216, samples/sec=32.4, sec/step=0.987, eta=9:49:10\n",
      "2018-09-09 15:58:37 Iter 14300 [Train]: loss=91.34, epe=6.36, lr=0.000221, samples/sec=32.4, sec/step=0.987, eta=9:47:05\n",
      "2018-09-09 16:00:25 Iter 14400 [Train]: loss=90.89, epe=6.28, lr=0.000226, samples/sec=32.4, sec/step=0.989, eta=9:46:35\n",
      "2018-09-09 16:02:13 Iter 14500 [Train]: loss=91.82, epe=6.33, lr=0.000230, samples/sec=32.3, sec/step=0.990, eta=9:45:53\n",
      "2018-09-09 16:04:01 Iter 14600 [Train]: loss=91.26, epe=6.32, lr=0.000235, samples/sec=32.5, sec/step=0.985, eta=9:41:13\n",
      "2018-09-09 16:05:49 Iter 14700 [Train]: loss=97.77, epe=6.84, lr=0.000240, samples/sec=32.4, sec/step=0.988, eta=9:41:22\n",
      "2018-09-09 16:07:37 Iter 14800 [Train]: loss=90.73, epe=6.27, lr=0.000245, samples/sec=32.5, sec/step=0.986, eta=9:38:31\n",
      "2018-09-09 16:09:25 Iter 14900 [Train]: loss=90.87, epe=6.27, lr=0.000250, samples/sec=32.4, sec/step=0.988, eta=9:37:59\n",
      "2018-09-09 16:11:14 Iter 15000 [Train]: loss=95.68, epe=6.69, lr=0.000255, samples/sec=32.4, sec/step=0.988, eta=9:36:25\n",
      "2018-09-09 16:11:30 Iter 15000 [Val]: loss=94.03, epe=6.67\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-15000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-15000\n",
      "2018-09-09 16:13:37 Iter 15100 [Train]: loss=90.28, epe=6.24, lr=0.000250, samples/sec=32.5, sec/step=0.984, eta=9:32:15\n",
      "2018-09-09 16:15:25 Iter 15200 [Train]: loss=86.85, epe=5.99, lr=0.000245, samples/sec=32.4, sec/step=0.988, eta=9:32:52\n",
      "2018-09-09 16:17:14 Iter 15300 [Train]: loss=89.17, epe=6.15, lr=0.000240, samples/sec=32.4, sec/step=0.989, eta=9:31:52\n",
      "2018-09-09 16:19:02 Iter 15400 [Train]: loss=88.42, epe=6.10, lr=0.000235, samples/sec=32.4, sec/step=0.988, eta=9:29:55\n",
      "2018-09-09 16:20:51 Iter 15500 [Train]: loss=88.90, epe=6.10, lr=0.000231, samples/sec=32.3, sec/step=0.991, eta=9:29:40\n",
      "2018-09-09 16:22:39 Iter 15600 [Train]: loss=86.49, epe=5.96, lr=0.000226, samples/sec=32.3, sec/step=0.991, eta=9:28:02\n",
      "2018-09-09 16:24:28 Iter 15700 [Train]: loss=87.03, epe=5.96, lr=0.000221, samples/sec=32.1, sec/step=0.996, eta=9:29:18\n",
      "2018-09-09 16:26:17 Iter 15800 [Train]: loss=88.52, epe=6.07, lr=0.000216, samples/sec=32.2, sec/step=0.994, eta=9:26:19\n",
      "2018-09-09 16:28:06 Iter 15900 [Train]: loss=90.69, epe=6.25, lr=0.000211, samples/sec=32.2, sec/step=0.995, eta=9:25:23\n",
      "2018-09-09 16:29:55 Iter 16000 [Train]: loss=91.10, epe=6.24, lr=0.000206, samples/sec=32.1, sec/step=0.996, eta=9:24:38\n",
      "2018-09-09 16:30:11 Iter 16000 [Val]: loss=88.59, epe=6.21\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-16000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-16000\n",
      "2018-09-09 16:32:17 Iter 16100 [Train]: loss=84.80, epe=5.81, lr=0.000201, samples/sec=32.2, sec/step=0.994, eta=9:21:48\n",
      "2018-09-09 16:34:06 Iter 16200 [Train]: loss=90.99, epe=6.30, lr=0.000196, samples/sec=32.2, sec/step=0.994, eta=9:19:59\n",
      "2018-09-09 16:35:55 Iter 16300 [Train]: loss=86.68, epe=5.98, lr=0.000191, samples/sec=32.2, sec/step=0.993, eta=9:17:52\n",
      "2018-09-09 16:37:44 Iter 16400 [Train]: loss=85.50, epe=5.88, lr=0.000186, samples/sec=32.0, sec/step=0.999, eta=9:19:25\n",
      "2018-09-09 16:39:33 Iter 16500 [Train]: loss=89.27, epe=6.17, lr=0.000182, samples/sec=32.0, sec/step=0.999, eta=9:17:49\n",
      "2018-09-09 16:41:23 Iter 16600 [Train]: loss=84.36, epe=5.76, lr=0.000177, samples/sec=32.0, sec/step=0.999, eta=9:15:59\n",
      "2018-09-09 16:43:12 Iter 16700 [Train]: loss=84.71, epe=5.79, lr=0.000172, samples/sec=32.0, sec/step=0.999, eta=9:14:33\n",
      "2018-09-09 16:45:02 Iter 16800 [Train]: loss=82.91, epe=5.68, lr=0.000167, samples/sec=32.0, sec/step=1.000, eta=9:13:20\n",
      "2018-09-09 16:46:51 Iter 16900 [Train]: loss=82.24, epe=5.61, lr=0.000162, samples/sec=32.0, sec/step=0.999, eta=9:11:09\n",
      "2018-09-09 16:48:40 Iter 17000 [Train]: loss=84.43, epe=5.77, lr=0.000157, samples/sec=32.1, sec/step=0.998, eta=9:09:07\n",
      "2018-09-09 16:48:57 Iter 17000 [Val]: loss=82.20, epe=5.72\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-17000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-17000\n",
      "2018-09-09 16:51:03 Iter 17100 [Train]: loss=87.44, epe=6.04, lr=0.000152, samples/sec=32.1, sec/step=0.997, eta=9:06:30\n",
      "2018-09-09 16:52:53 Iter 17200 [Train]: loss=84.08, epe=5.76, lr=0.000147, samples/sec=31.9, sec/step=1.002, eta=9:07:48\n",
      "2018-09-09 16:54:43 Iter 17300 [Train]: loss=83.72, epe=5.70, lr=0.000142, samples/sec=31.8, sec/step=1.005, eta=9:08:00\n",
      "2018-09-09 16:56:32 Iter 17400 [Train]: loss=83.03, epe=5.65, lr=0.000137, samples/sec=32.0, sec/step=1.001, eta=9:04:05\n",
      "2018-09-09 16:58:22 Iter 17500 [Train]: loss=81.49, epe=5.58, lr=0.000133, samples/sec=31.9, sec/step=1.002, eta=9:02:36\n",
      "2018-09-09 17:00:12 Iter 17600 [Train]: loss=79.03, epe=5.36, lr=0.000128, samples/sec=31.9, sec/step=1.003, eta=9:01:42\n",
      "2018-09-09 17:02:01 Iter 17700 [Train]: loss=79.68, epe=5.40, lr=0.000123, samples/sec=32.1, sec/step=0.998, eta=8:57:27\n",
      "2018-09-09 17:03:51 Iter 17800 [Train]: loss=82.35, epe=5.63, lr=0.000118, samples/sec=32.0, sec/step=1.001, eta=8:57:28\n",
      "2018-09-09 17:05:40 Iter 17900 [Train]: loss=76.94, epe=5.21, lr=0.000113, samples/sec=31.9, sec/step=1.002, eta=8:56:20\n",
      "2018-09-09 17:07:30 Iter 18000 [Train]: loss=77.27, epe=5.21, lr=0.000108, samples/sec=32.1, sec/step=0.998, eta=8:52:02\n",
      "2018-09-09 17:07:47 Iter 18000 [Val]: loss=84.09, epe=5.86\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-18000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-18000\n",
      "2018-09-09 17:09:54 Iter 18100 [Train]: loss=83.52, epe=5.71, lr=0.000103, samples/sec=31.7, sec/step=1.010, eta=8:56:52\n",
      "2018-09-09 17:11:44 Iter 18200 [Train]: loss=78.64, epe=5.34, lr=0.000098, samples/sec=31.7, sec/step=1.008, eta=8:54:13\n",
      "2018-09-09 17:13:35 Iter 18300 [Train]: loss=76.30, epe=5.17, lr=0.000093, samples/sec=31.7, sec/step=1.011, eta=8:54:08\n",
      "2018-09-09 17:15:26 Iter 18400 [Train]: loss=79.02, epe=5.39, lr=0.000088, samples/sec=31.5, sec/step=1.015, eta=8:54:43\n",
      "2018-09-09 17:17:16 Iter 18500 [Train]: loss=78.72, epe=5.33, lr=0.000084, samples/sec=31.6, sec/step=1.013, eta=8:51:40\n",
      "2018-09-09 17:19:06 Iter 18600 [Train]: loss=80.61, epe=5.49, lr=0.000079, samples/sec=31.8, sec/step=1.007, eta=8:46:59\n",
      "2018-09-09 17:20:56 Iter 18700 [Train]: loss=79.48, epe=5.40, lr=0.000074, samples/sec=31.9, sec/step=1.003, eta=8:43:29\n",
      "2018-09-09 17:22:46 Iter 18800 [Train]: loss=78.64, epe=5.34, lr=0.000069, samples/sec=31.8, sec/step=1.005, eta=8:42:40\n",
      "2018-09-09 17:24:36 Iter 18900 [Train]: loss=75.42, epe=5.10, lr=0.000064, samples/sec=31.8, sec/step=1.007, eta=8:41:55\n",
      "2018-09-09 17:26:26 Iter 19000 [Train]: loss=75.31, epe=5.05, lr=0.000059, samples/sec=31.8, sec/step=1.005, eta=8:39:08\n",
      "2018-09-09 17:26:44 Iter 19000 [Val]: loss=77.47, epe=5.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-19000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-19000\n",
      "2018-09-09 17:28:50 Iter 19100 [Train]: loss=77.36, epe=5.23, lr=0.000054, samples/sec=31.8, sec/step=1.005, eta=8:37:32\n",
      "2018-09-09 17:30:40 Iter 19200 [Train]: loss=75.92, epe=5.12, lr=0.000049, samples/sec=31.8, sec/step=1.007, eta=8:36:44\n",
      "2018-09-09 17:32:30 Iter 19300 [Train]: loss=73.54, epe=4.94, lr=0.000044, samples/sec=31.7, sec/step=1.010, eta=8:36:47\n",
      "2018-09-09 17:34:21 Iter 19400 [Train]: loss=76.11, epe=5.16, lr=0.000039, samples/sec=31.8, sec/step=1.008, eta=8:33:59\n",
      "2018-09-09 17:36:11 Iter 19500 [Train]: loss=74.27, epe=5.01, lr=0.000035, samples/sec=31.5, sec/step=1.014, eta=8:35:40\n",
      "2018-09-09 17:38:02 Iter 19600 [Train]: loss=75.35, epe=5.06, lr=0.000030, samples/sec=31.7, sec/step=1.010, eta=8:31:57\n",
      "2018-09-09 17:39:52 Iter 19700 [Train]: loss=76.42, epe=5.16, lr=0.000025, samples/sec=31.6, sec/step=1.012, eta=8:31:15\n",
      "2018-09-09 17:41:43 Iter 19800 [Train]: loss=75.59, epe=5.12, lr=0.000020, samples/sec=31.7, sec/step=1.010, eta=8:28:26\n",
      "2018-09-09 17:43:33 Iter 19900 [Train]: loss=73.72, epe=nan, lr=0.000015, samples/sec=31.7, sec/step=1.010, eta=8:26:32\n",
      "2018-09-09 17:45:24 Iter 20000 [Train]: loss=74.15, epe=4.98, lr=0.000010, samples/sec=31.6, sec/step=1.014, eta=8:27:02\n",
      "2018-09-09 17:45:41 Iter 20000 [Val]: loss=77.75, epe=5.37\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-20000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-20000\n",
      "2018-09-09 17:47:51 Iter 20100 [Train]: loss=73.23, epe=4.93, lr=0.000012, samples/sec=31.7, sec/step=1.010, eta=8:23:05\n",
      "2018-09-09 17:49:42 Iter 20200 [Train]: loss=74.62, epe=5.02, lr=0.000015, samples/sec=31.5, sec/step=1.017, eta=8:25:17\n",
      "2018-09-09 17:51:33 Iter 20300 [Train]: loss=72.96, epe=4.91, lr=0.000017, samples/sec=31.4, sec/step=1.018, eta=8:23:41\n",
      "2018-09-09 17:53:25 Iter 20400 [Train]: loss=73.43, epe=4.96, lr=0.000020, samples/sec=31.4, sec/step=1.019, eta=8:22:33\n",
      "2018-09-09 17:55:16 Iter 20500 [Train]: loss=75.45, epe=5.10, lr=0.000022, samples/sec=31.4, sec/step=1.021, eta=8:21:49\n",
      "2018-09-09 17:57:08 Iter 20600 [Train]: loss=75.48, epe=5.09, lr=0.000025, samples/sec=31.4, sec/step=1.018, eta=8:18:35\n",
      "2018-09-09 17:58:59 Iter 20700 [Train]: loss=74.79, epe=5.04, lr=0.000027, samples/sec=31.3, sec/step=1.022, eta=8:19:04\n",
      "2018-09-09 18:00:51 Iter 20800 [Train]: loss=73.21, epe=4.92, lr=0.000030, samples/sec=31.3, sec/step=1.022, eta=8:17:10\n",
      "2018-09-09 18:02:42 Iter 20900 [Train]: loss=75.27, epe=5.09, lr=0.000032, samples/sec=31.5, sec/step=1.016, eta=8:12:54\n",
      "2018-09-09 18:04:33 Iter 21000 [Train]: loss=73.22, epe=4.91, lr=0.000034, samples/sec=31.4, sec/step=1.019, eta=8:12:28\n",
      "2018-09-09 18:04:51 Iter 21000 [Val]: loss=79.62, epe=5.53\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-21000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-21000\n",
      "2018-09-09 18:06:59 Iter 21100 [Train]: loss=74.64, epe=5.05, lr=0.000037, samples/sec=31.3, sec/step=1.021, eta=8:11:51\n",
      "2018-09-09 18:08:51 Iter 21200 [Train]: loss=74.01, epe=4.98, lr=0.000039, samples/sec=31.1, sec/step=1.028, eta=8:13:31\n",
      "2018-09-09 18:10:43 Iter 21300 [Train]: loss=74.92, epe=5.05, lr=0.000042, samples/sec=31.3, sec/step=1.024, eta=8:09:45\n",
      "2018-09-09 18:12:34 Iter 21400 [Train]: loss=76.79, epe=5.23, lr=0.000044, samples/sec=31.3, sec/step=1.022, eta=8:07:01\n",
      "2018-09-09 18:14:27 Iter 21500 [Train]: loss=78.65, epe=5.34, lr=0.000047, samples/sec=31.2, sec/step=1.026, eta=8:07:18\n",
      "2018-09-09 18:16:18 Iter 21600 [Train]: loss=75.23, epe=5.07, lr=0.000049, samples/sec=31.3, sec/step=1.023, eta=8:04:12\n",
      "2018-09-09 18:18:11 Iter 21700 [Train]: loss=72.27, epe=4.84, lr=0.000052, samples/sec=31.2, sec/step=1.024, eta=8:03:10\n",
      "2018-09-09 18:20:02 Iter 21800 [Train]: loss=73.57, epe=4.95, lr=0.000054, samples/sec=31.4, sec/step=1.018, eta=7:58:15\n",
      "2018-09-09 18:21:54 Iter 21900 [Train]: loss=74.72, epe=5.03, lr=0.000057, samples/sec=31.2, sec/step=1.025, eta=7:59:51\n",
      "2018-09-09 18:23:46 Iter 22000 [Train]: loss=76.11, epe=5.14, lr=0.000059, samples/sec=31.4, sec/step=1.020, eta=7:56:10\n",
      "2018-09-09 18:24:03 Iter 22000 [Val]: loss=75.97, epe=5.22\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-22000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-22000\n",
      "2018-09-09 18:26:12 Iter 22100 [Train]: loss=72.20, epe=4.83, lr=0.000061, samples/sec=31.2, sec/step=1.024, eta=7:56:13\n",
      "2018-09-09 18:28:04 Iter 22200 [Train]: loss=78.13, epe=5.29, lr=0.000064, samples/sec=31.3, sec/step=1.023, eta=7:53:47\n",
      "2018-09-09 18:29:57 Iter 22300 [Train]: loss=72.79, epe=4.88, lr=0.000066, samples/sec=30.9, sec/step=1.035, eta=7:57:47\n",
      "2018-09-09 18:31:50 Iter 22400 [Train]: loss=72.16, epe=4.83, lr=0.000069, samples/sec=31.0, sec/step=1.031, eta=7:54:24\n",
      "2018-09-09 18:33:42 Iter 22500 [Train]: loss=75.61, epe=5.11, lr=0.000071, samples/sec=31.0, sec/step=1.032, eta=7:53:01\n",
      "2018-09-09 18:35:35 Iter 22600 [Train]: loss=75.58, epe=5.09, lr=0.000074, samples/sec=31.1, sec/step=1.029, eta=7:49:44\n",
      "2018-09-09 18:37:27 Iter 22700 [Train]: loss=77.16, epe=5.20, lr=0.000076, samples/sec=31.0, sec/step=1.033, eta=7:50:04\n",
      "2018-09-09 18:39:20 Iter 22800 [Train]: loss=77.69, epe=5.26, lr=0.000079, samples/sec=31.2, sec/step=1.024, eta=7:44:16\n",
      "2018-09-09 18:41:12 Iter 22900 [Train]: loss=74.68, epe=5.05, lr=0.000081, samples/sec=31.1, sec/step=1.029, eta=7:44:54\n",
      "2018-09-09 18:43:05 Iter 23000 [Train]: loss=79.67, epe=5.43, lr=0.000083, samples/sec=31.1, sec/step=1.028, eta=7:42:46\n",
      "2018-09-09 18:43:22 Iter 23000 [Val]: loss=77.89, epe=5.34\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-23000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-23000\n",
      "2018-09-09 18:45:32 Iter 23100 [Train]: loss=76.82, epe=5.19, lr=0.000086, samples/sec=31.2, sec/step=1.027, eta=7:40:15\n",
      "2018-09-09 18:47:26 Iter 23200 [Train]: loss=74.25, epe=5.01, lr=0.000088, samples/sec=30.8, sec/step=1.039, eta=7:44:17\n",
      "2018-09-09 18:49:19 Iter 23300 [Train]: loss=75.80, epe=5.13, lr=0.000091, samples/sec=30.9, sec/step=1.036, eta=7:41:05\n",
      "2018-09-09 18:51:13 Iter 23400 [Train]: loss=77.35, epe=5.23, lr=0.000093, samples/sec=30.7, sec/step=1.041, eta=7:41:26\n",
      "2018-09-09 18:53:06 Iter 23500 [Train]: loss=78.57, epe=5.33, lr=0.000096, samples/sec=30.7, sec/step=1.043, eta=7:40:36\n",
      "2018-09-09 18:55:00 Iter 23600 [Train]: loss=80.94, epe=5.50, lr=0.000098, samples/sec=30.8, sec/step=1.038, eta=7:36:36\n",
      "2018-09-09 18:56:53 Iter 23700 [Train]: loss=73.26, epe=4.91, lr=0.000101, samples/sec=30.8, sec/step=1.040, eta=7:36:04\n",
      "2018-09-09 18:58:47 Iter 23800 [Train]: loss=76.97, epe=5.22, lr=0.000103, samples/sec=30.7, sec/step=1.042, eta=7:34:51\n",
      "2018-09-09 19:00:40 Iter 23900 [Train]: loss=76.14, epe=5.12, lr=0.000106, samples/sec=30.9, sec/step=1.037, eta=7:31:05\n",
      "2018-09-09 19:02:34 Iter 24000 [Train]: loss=78.30, epe=5.31, lr=0.000108, samples/sec=30.9, sec/step=1.037, eta=7:29:26\n",
      "2018-09-09 19:02:52 Iter 24000 [Val]: loss=77.19, epe=5.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-24000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-24000\n",
      "2018-09-09 19:05:02 Iter 24100 [Train]: loss=75.96, epe=5.10, lr=0.000110, samples/sec=30.8, sec/step=1.038, eta=7:28:03\n",
      "2018-09-09 19:06:57 Iter 24200 [Train]: loss=74.99, epe=5.04, lr=0.000113, samples/sec=30.5, sec/step=1.048, eta=7:30:26\n",
      "2018-09-09 19:08:50 Iter 24300 [Train]: loss=76.63, epe=5.18, lr=0.000115, samples/sec=30.8, sec/step=1.040, eta=7:25:33\n",
      "2018-09-09 19:10:45 Iter 24400 [Train]: loss=75.41, epe=5.06, lr=0.000118, samples/sec=30.6, sec/step=1.047, eta=7:26:32\n",
      "2018-09-09 19:12:39 Iter 24500 [Train]: loss=73.39, epe=4.92, lr=0.000120, samples/sec=30.7, sec/step=1.041, eta=7:22:28\n",
      "2018-09-09 19:14:32 Iter 24600 [Train]: loss=77.17, epe=5.21, lr=0.000123, samples/sec=30.6, sec/step=1.046, eta=7:22:49\n",
      "2018-09-09 19:16:26 Iter 24700 [Train]: loss=77.16, epe=5.22, lr=0.000125, samples/sec=30.7, sec/step=1.042, eta=7:19:34\n",
      "2018-09-09 19:18:20 Iter 24800 [Train]: loss=73.75, epe=4.94, lr=0.000128, samples/sec=30.7, sec/step=1.043, eta=7:18:02\n",
      "2018-09-09 19:20:14 Iter 24900 [Train]: loss=75.37, epe=5.08, lr=0.000130, samples/sec=30.6, sec/step=1.044, eta=7:16:54\n",
      "2018-09-09 19:22:08 Iter 25000 [Train]: loss=73.33, epe=4.92, lr=0.000133, samples/sec=30.6, sec/step=1.046, eta=7:15:52\n",
      "2018-09-09 19:22:26 Iter 25000 [Val]: loss=80.70, epe=5.61\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-25000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-25000\n",
      "2018-09-09 19:24:37 Iter 25100 [Train]: loss=76.28, epe=5.13, lr=0.000130, samples/sec=30.7, sec/step=1.041, eta=7:11:52\n",
      "2018-09-09 19:26:31 Iter 25200 [Train]: loss=79.53, epe=5.41, lr=0.000128, samples/sec=30.5, sec/step=1.049, eta=7:13:31\n",
      "2018-09-09 19:28:26 Iter 25300 [Train]: loss=81.17, epe=5.49, lr=0.000125, samples/sec=30.5, sec/step=1.050, eta=7:12:15\n",
      "2018-09-09 19:30:20 Iter 25400 [Train]: loss=72.08, epe=4.81, lr=0.000123, samples/sec=30.5, sec/step=1.049, eta=7:10:11\n",
      "2018-09-09 19:32:15 Iter 25500 [Train]: loss=73.26, epe=4.92, lr=0.000120, samples/sec=30.6, sec/step=1.046, eta=7:06:59\n",
      "2018-09-09 19:34:09 Iter 25600 [Train]: loss=74.70, epe=5.03, lr=0.000118, samples/sec=30.5, sec/step=1.049, eta=7:06:35\n",
      "2018-09-09 19:36:04 Iter 25700 [Train]: loss=74.56, epe=4.99, lr=0.000115, samples/sec=30.5, sec/step=1.050, eta=7:05:16\n",
      "2018-09-09 19:37:58 Iter 25800 [Train]: loss=76.54, epe=5.15, lr=0.000113, samples/sec=30.5, sec/step=1.050, eta=7:03:34\n",
      "2018-09-09 19:39:53 Iter 25900 [Train]: loss=72.47, epe=4.84, lr=0.000110, samples/sec=30.4, sec/step=1.053, eta=7:03:06\n",
      "2018-09-09 19:41:48 Iter 26000 [Train]: loss=71.30, epe=4.75, lr=0.000108, samples/sec=30.4, sec/step=1.054, eta=7:01:26\n",
      "2018-09-09 19:42:06 Iter 26000 [Val]: loss=76.33, epe=5.24\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-26000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-26000\n",
      "2018-09-09 19:44:17 Iter 26100 [Train]: loss=72.24, epe=4.83, lr=0.000106, samples/sec=30.4, sec/step=1.051, eta=6:58:39\n",
      "2018-09-09 19:46:13 Iter 26200 [Train]: loss=73.44, epe=4.92, lr=0.000103, samples/sec=30.1, sec/step=1.064, eta=7:02:00\n",
      "2018-09-09 19:48:09 Iter 26300 [Train]: loss=72.81, epe=4.87, lr=0.000101, samples/sec=30.2, sec/step=1.061, eta=6:59:11\n",
      "2018-09-09 19:50:05 Iter 26400 [Train]: loss=73.44, epe=4.93, lr=0.000098, samples/sec=30.1, sec/step=1.062, eta=6:57:38\n",
      "2018-09-09 19:52:00 Iter 26500 [Train]: loss=74.31, epe=4.99, lr=0.000096, samples/sec=30.3, sec/step=1.056, eta=6:53:45\n",
      "2018-09-09 19:53:55 Iter 26600 [Train]: loss=77.75, epe=5.26, lr=0.000093, samples/sec=30.3, sec/step=1.056, eta=6:51:54\n",
      "2018-09-09 19:55:51 Iter 26700 [Train]: loss=71.87, epe=4.81, lr=0.000091, samples/sec=30.3, sec/step=1.057, eta=6:50:40\n",
      "2018-09-09 19:57:46 Iter 26800 [Train]: loss=76.96, epe=5.20, lr=0.000088, samples/sec=30.3, sec/step=1.056, eta=6:48:15\n",
      "2018-09-09 19:59:41 Iter 26900 [Train]: loss=72.31, epe=4.80, lr=0.000086, samples/sec=30.3, sec/step=1.055, eta=6:46:03\n",
      "2018-09-09 20:01:36 Iter 27000 [Train]: loss=75.20, epe=5.05, lr=0.000083, samples/sec=30.5, sec/step=1.049, eta=6:41:59\n",
      "2018-09-09 20:01:55 Iter 27000 [Val]: loss=75.77, epe=5.18\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-27000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-27000\n",
      "2018-09-09 20:04:10 Iter 27100 [Train]: loss=72.38, epe=4.84, lr=0.000081, samples/sec=30.3, sec/step=1.057, eta=6:43:33\n",
      "2018-09-09 20:06:07 Iter 27200 [Train]: loss=70.34, epe=4.69, lr=0.000079, samples/sec=29.8, sec/step=1.073, eta=6:47:47\n",
      "2018-09-09 20:08:04 Iter 27300 [Train]: loss=70.05, epe=4.64, lr=0.000076, samples/sec=29.9, sec/step=1.072, eta=6:45:31\n",
      "2018-09-09 20:10:01 Iter 27400 [Train]: loss=73.51, epe=4.94, lr=0.000074, samples/sec=30.0, sec/step=1.067, eta=6:41:56\n",
      "2018-09-09 20:11:59 Iter 27500 [Train]: loss=73.68, epe=4.95, lr=0.000071, samples/sec=29.5, sec/step=1.083, eta=6:46:06\n",
      "2018-09-09 20:13:56 Iter 27600 [Train]: loss=71.50, epe=4.77, lr=0.000069, samples/sec=29.7, sec/step=1.078, eta=6:42:16\n",
      "2018-09-09 20:15:53 Iter 27700 [Train]: loss=73.54, epe=4.91, lr=0.000066, samples/sec=30.0, sec/step=1.068, eta=6:36:52\n",
      "2018-09-09 20:17:49 Iter 27800 [Train]: loss=71.58, epe=4.78, lr=0.000064, samples/sec=30.1, sec/step=1.063, eta=6:33:09\n",
      "2018-09-09 20:19:46 Iter 27900 [Train]: loss=71.17, epe=4.75, lr=0.000061, samples/sec=29.9, sec/step=1.069, eta=6:33:45\n",
      "2018-09-09 20:21:43 Iter 28000 [Train]: loss=70.64, epe=4.71, lr=0.000059, samples/sec=29.9, sec/step=1.069, eta=6:32:05\n",
      "2018-09-09 20:22:02 Iter 28000 [Val]: loss=76.53, epe=5.27\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-28000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-28000\n",
      "2018-09-09 20:24:18 Iter 28100 [Train]: loss=71.39, epe=4.74, lr=0.000057, samples/sec=29.9, sec/step=1.071, eta=6:30:45\n",
      "2018-09-09 20:26:15 Iter 28200 [Train]: loss=67.98, epe=4.52, lr=0.000054, samples/sec=29.9, sec/step=1.071, eta=6:29:02\n",
      "2018-09-09 20:28:13 Iter 28300 [Train]: loss=71.35, epe=4.78, lr=0.000052, samples/sec=29.7, sec/step=1.078, eta=6:30:01\n",
      "2018-09-09 20:30:11 Iter 28400 [Train]: loss=68.15, epe=4.52, lr=0.000049, samples/sec=29.6, sec/step=1.080, eta=6:28:59\n",
      "2018-09-09 20:32:09 Iter 28500 [Train]: loss=69.64, epe=4.64, lr=0.000047, samples/sec=29.5, sec/step=1.084, eta=6:28:19\n",
      "2018-09-09 20:34:07 Iter 28600 [Train]: loss=72.98, epe=4.89, lr=0.000044, samples/sec=29.6, sec/step=1.079, eta=6:24:59\n",
      "2018-09-09 20:36:04 Iter 28700 [Train]: loss=68.92, epe=4.58, lr=0.000042, samples/sec=29.7, sec/step=1.077, eta=6:22:17\n",
      "2018-09-09 20:38:02 Iter 28800 [Train]: loss=68.51, epe=4.52, lr=0.000039, samples/sec=29.6, sec/step=1.081, eta=6:22:07\n",
      "2018-09-09 20:40:00 Iter 28900 [Train]: loss=70.44, epe=4.72, lr=0.000037, samples/sec=29.6, sec/step=1.082, eta=6:20:20\n",
      "2018-09-09 20:41:58 Iter 29000 [Train]: loss=69.41, epe=4.60, lr=0.000035, samples/sec=29.6, sec/step=1.082, eta=6:18:37\n",
      "2018-09-09 20:42:18 Iter 29000 [Val]: loss=73.55, epe=5.04\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-29000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-29000\n",
      "2018-09-09 20:44:32 Iter 29100 [Train]: loss=67.92, epe=4.51, lr=0.000032, samples/sec=29.7, sec/step=1.077, eta=6:15:13\n",
      "2018-09-09 20:46:30 Iter 29200 [Train]: loss=69.57, epe=4.63, lr=0.000030, samples/sec=29.4, sec/step=1.088, eta=6:17:07\n",
      "2018-09-09 20:48:28 Iter 29300 [Train]: loss=69.02, epe=4.57, lr=0.000027, samples/sec=29.6, sec/step=1.080, eta=6:12:31\n",
      "2018-09-09 20:50:28 Iter 29400 [Train]: loss=70.55, epe=4.72, lr=0.000025, samples/sec=29.3, sec/step=1.091, eta=6:14:37\n",
      "2018-09-09 20:52:28 Iter 29500 [Train]: loss=69.03, epe=4.58, lr=0.000022, samples/sec=29.1, sec/step=1.098, eta=6:15:13\n",
      "2018-09-09 20:54:27 Iter 29600 [Train]: loss=68.20, epe=4.50, lr=0.000020, samples/sec=29.3, sec/step=1.091, eta=6:10:47\n",
      "2018-09-09 20:56:24 Iter 29700 [Train]: loss=68.23, epe=4.51, lr=0.000017, samples/sec=29.7, sec/step=1.078, eta=6:04:43\n",
      "2018-09-09 20:58:22 Iter 29800 [Train]: loss=66.85, epe=4.42, lr=0.000015, samples/sec=29.5, sec/step=1.085, eta=6:05:07\n",
      "2018-09-09 21:00:21 Iter 29900 [Train]: loss=69.47, epe=4.62, lr=0.000012, samples/sec=29.4, sec/step=1.087, eta=6:04:03\n",
      "2018-09-09 21:02:19 Iter 30000 [Train]: loss=69.31, epe=4.59, lr=0.000010, samples/sec=29.5, sec/step=1.086, eta=6:01:52\n",
      "2018-09-09 21:02:38 Iter 30000 [Val]: loss=70.52, epe=4.82\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-30000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-30000\n",
      "2018-09-09 21:04:54 Iter 30100 [Train]: loss=68.66, epe=4.56, lr=0.000011, samples/sec=29.3, sec/step=1.092, eta=6:02:06\n",
      "2018-09-09 21:06:53 Iter 30200 [Train]: loss=65.25, epe=4.26, lr=0.000012, samples/sec=29.3, sec/step=1.091, eta=6:00:05\n",
      "2018-09-09 21:08:53 Iter 30300 [Train]: loss=71.46, epe=4.78, lr=0.000014, samples/sec=29.0, sec/step=1.102, eta=6:01:58\n",
      "2018-09-09 21:10:51 Iter 30400 [Train]: loss=68.50, epe=4.55, lr=0.000015, samples/sec=29.5, sec/step=1.086, eta=5:54:50\n",
      "2018-09-09 21:12:51 Iter 30500 [Train]: loss=69.51, epe=4.62, lr=0.000016, samples/sec=29.3, sec/step=1.092, eta=5:54:45\n",
      "2018-09-09 21:14:49 Iter 30600 [Train]: loss=67.88, epe=4.48, lr=0.000017, samples/sec=29.4, sec/step=1.089, eta=5:51:59\n",
      "2018-09-09 21:16:48 Iter 30700 [Train]: loss=64.89, epe=4.26, lr=0.000019, samples/sec=29.3, sec/step=1.093, eta=5:51:30\n",
      "2018-09-09 21:18:47 Iter 30800 [Train]: loss=68.54, epe=4.54, lr=0.000020, samples/sec=29.3, sec/step=1.093, eta=5:49:37\n",
      "2018-09-09 21:20:46 Iter 30900 [Train]: loss=68.35, epe=4.52, lr=0.000021, samples/sec=29.4, sec/step=1.087, eta=5:46:02\n",
      "2018-09-09 21:22:46 Iter 31000 [Train]: loss=68.82, epe=4.56, lr=0.000022, samples/sec=29.2, sec/step=1.095, eta=5:46:54\n",
      "2018-09-09 21:23:05 Iter 31000 [Val]: loss=73.18, epe=5.02\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-31000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-31000\n",
      "2018-09-09 21:25:21 Iter 31100 [Train]: loss=71.74, epe=4.78, lr=0.000023, samples/sec=29.3, sec/step=1.094, eta=5:44:32\n",
      "2018-09-09 21:27:22 Iter 31200 [Train]: loss=66.87, epe=4.43, lr=0.000025, samples/sec=28.9, sec/step=1.106, eta=5:46:38\n",
      "2018-09-09 21:29:21 Iter 31300 [Train]: loss=67.37, epe=4.46, lr=0.000026, samples/sec=29.2, sec/step=1.096, eta=5:41:32\n",
      "2018-09-09 21:31:21 Iter 31400 [Train]: loss=67.11, epe=4.43, lr=0.000027, samples/sec=29.2, sec/step=1.096, eta=5:39:49\n",
      "2018-09-09 21:33:20 Iter 31500 [Train]: loss=66.61, epe=4.40, lr=0.000028, samples/sec=29.2, sec/step=1.096, eta=5:38:05\n",
      "2018-09-09 21:35:21 Iter 31600 [Train]: loss=68.00, epe=4.51, lr=0.000030, samples/sec=28.9, sec/step=1.109, eta=5:40:04\n",
      "2018-09-09 21:37:20 Iter 31700 [Train]: loss=69.47, epe=4.61, lr=0.000031, samples/sec=29.2, sec/step=1.095, eta=5:34:01\n",
      "2018-09-09 21:39:19 Iter 31800 [Train]: loss=70.04, epe=4.64, lr=0.000032, samples/sec=29.3, sec/step=1.093, eta=5:31:32\n",
      "2018-09-09 21:41:19 Iter 31900 [Train]: loss=69.02, epe=4.60, lr=0.000033, samples/sec=29.2, sec/step=1.095, eta=5:30:14\n",
      "2018-09-09 21:43:17 Iter 32000 [Train]: loss=67.34, epe=4.46, lr=0.000034, samples/sec=29.4, sec/step=1.088, eta=5:26:28\n",
      "2018-09-09 21:43:37 Iter 32000 [Val]: loss=69.67, epe=4.69\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-32000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-32000\n",
      "2018-09-09 21:45:54 Iter 32100 [Train]: loss=69.98, epe=4.67, lr=0.000036, samples/sec=29.2, sec/step=1.095, eta=5:26:48\n",
      "2018-09-09 21:47:54 Iter 32200 [Train]: loss=69.50, epe=4.62, lr=0.000037, samples/sec=29.0, sec/step=1.104, eta=5:27:28\n",
      "2018-09-09 21:49:55 Iter 32300 [Train]: loss=67.78, epe=4.50, lr=0.000038, samples/sec=28.9, sec/step=1.108, eta=5:26:47\n",
      "2018-09-09 21:51:55 Iter 32400 [Train]: loss=66.65, epe=4.40, lr=0.000039, samples/sec=28.9, sec/step=1.106, eta=5:24:34\n",
      "2018-09-09 21:53:55 Iter 32500 [Train]: loss=66.49, epe=4.40, lr=0.000041, samples/sec=29.1, sec/step=1.098, eta=5:20:16\n",
      "2018-09-09 21:55:55 Iter 32600 [Train]: loss=68.21, epe=4.52, lr=0.000042, samples/sec=29.1, sec/step=1.099, eta=5:18:42\n",
      "2018-09-09 21:57:55 Iter 32700 [Train]: loss=69.07, epe=4.58, lr=0.000043, samples/sec=29.1, sec/step=1.098, eta=5:16:37\n",
      "2018-09-09 21:59:54 Iter 32800 [Train]: loss=70.01, epe=4.66, lr=0.000044, samples/sec=29.1, sec/step=1.099, eta=5:14:56\n",
      "2018-09-09 22:01:54 Iter 32900 [Train]: loss=69.72, epe=4.63, lr=0.000046, samples/sec=29.1, sec/step=1.098, eta=5:13:00\n",
      "2018-09-09 22:03:54 Iter 33000 [Train]: loss=69.16, epe=4.59, lr=0.000047, samples/sec=29.2, sec/step=1.097, eta=5:10:50\n",
      "2018-09-09 22:04:14 Iter 33000 [Val]: loss=74.23, epe=5.12\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-33000 is not in all_model_checkpoint_paths. Manually adding it.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-33000\n",
      "2018-09-09 22:06:31 Iter 33100 [Train]: loss=69.27, epe=4.59, lr=0.000048, samples/sec=29.0, sec/step=1.104, eta=5:10:50\n",
      "2018-09-09 22:08:32 Iter 33200 [Train]: loss=70.25, epe=4.69, lr=0.000049, samples/sec=28.8, sec/step=1.113, eta=5:11:33\n",
      "2018-09-09 22:10:34 Iter 33300 [Train]: loss=69.89, epe=4.65, lr=0.000050, samples/sec=28.8, sec/step=1.113, eta=5:09:45\n",
      "2018-09-09 22:12:35 Iter 33400 [Train]: loss=67.74, epe=4.49, lr=0.000052, samples/sec=28.7, sec/step=1.114, eta=5:08:14\n",
      "2018-09-09 22:14:36 Iter 33500 [Train]: loss=68.59, epe=4.56, lr=0.000053, samples/sec=28.8, sec/step=1.111, eta=5:05:27\n",
      "2018-09-09 22:16:37 Iter 33600 [Train]: loss=70.25, epe=4.68, lr=0.000054, samples/sec=28.8, sec/step=1.110, eta=5:03:21\n",
      "2018-09-09 22:18:38 Iter 33700 [Train]: loss=67.69, epe=4.48, lr=0.000055, samples/sec=28.8, sec/step=1.111, eta=5:01:46\n",
      "2018-09-09 22:20:38 Iter 33800 [Train]: loss=69.50, epe=4.62, lr=0.000057, samples/sec=29.0, sec/step=1.104, eta=4:57:58\n",
      "2018-09-09 22:22:39 Iter 33900 [Train]: loss=71.53, epe=4.78, lr=0.000058, samples/sec=28.8, sec/step=1.111, eta=4:58:06\n",
      "2018-09-09 22:24:40 Iter 34000 [Train]: loss=68.99, epe=4.56, lr=0.000059, samples/sec=28.8, sec/step=1.111, eta=4:56:18\n",
      "2018-09-09 22:25:00 Iter 34000 [Val]: loss=75.14, epe=5.19\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-34000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-34000\n",
      "2018-09-09 22:27:17 Iter 34100 [Train]: loss=69.72, epe=4.65, lr=0.000060, samples/sec=29.0, sec/step=1.103, eta=4:52:13\n",
      "2018-09-09 22:29:18 Iter 34200 [Train]: loss=67.90, epe=4.51, lr=0.000061, samples/sec=28.9, sec/step=1.106, eta=4:51:21\n",
      "2018-09-09 22:31:19 Iter 34300 [Train]: loss=67.11, epe=4.41, lr=0.000063, samples/sec=28.9, sec/step=1.108, eta=4:49:50\n",
      "2018-09-09 22:33:19 Iter 34400 [Train]: loss=68.68, epe=4.55, lr=0.000064, samples/sec=29.0, sec/step=1.102, eta=4:46:32\n",
      "2018-09-09 22:35:19 Iter 34500 [Train]: loss=71.87, epe=4.83, lr=0.000065, samples/sec=28.9, sec/step=1.107, eta=4:45:54\n",
      "2018-09-09 22:37:20 Iter 34600 [Train]: loss=69.96, epe=4.68, lr=0.000066, samples/sec=28.9, sec/step=1.108, eta=4:44:26\n",
      "2018-09-09 22:39:21 Iter 34700 [Train]: loss=66.70, epe=4.41, lr=0.000068, samples/sec=29.0, sec/step=1.104, eta=4:41:38\n",
      "2018-09-09 22:41:21 Iter 34800 [Train]: loss=69.73, epe=4.61, lr=0.000069, samples/sec=28.9, sec/step=1.108, eta=4:40:41\n",
      "2018-09-09 22:43:22 Iter 34900 [Train]: loss=69.73, epe=4.64, lr=0.000070, samples/sec=28.9, sec/step=1.109, eta=4:39:06\n",
      "2018-09-09 22:45:22 Iter 35000 [Train]: loss=67.96, epe=4.49, lr=0.000071, samples/sec=29.0, sec/step=1.103, eta=4:35:52\n",
      "2018-09-09 22:45:42 Iter 35000 [Val]: loss=71.10, epe=4.84\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-35000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-35000\n",
      "2018-09-09 22:48:05 Iter 35100 [Train]: loss=67.84, epe=4.48, lr=0.000070, samples/sec=28.6, sec/step=1.120, eta=4:38:08\n",
      "2018-09-09 22:50:06 Iter 35200 [Train]: loss=66.49, epe=4.39, lr=0.000069, samples/sec=28.8, sec/step=1.110, eta=4:33:52\n",
      "2018-09-09 22:52:08 Iter 35300 [Train]: loss=67.98, epe=4.49, lr=0.000068, samples/sec=28.5, sec/step=1.122, eta=4:34:56\n",
      "2018-09-09 22:54:10 Iter 35400 [Train]: loss=69.04, epe=4.58, lr=0.000066, samples/sec=28.6, sec/step=1.118, eta=4:32:09\n",
      "2018-09-09 22:56:12 Iter 35500 [Train]: loss=72.82, epe=4.89, lr=0.000065, samples/sec=28.6, sec/step=1.120, eta=4:30:33\n",
      "2018-09-09 22:58:13 Iter 35600 [Train]: loss=68.32, epe=4.50, lr=0.000064, samples/sec=28.7, sec/step=1.114, eta=4:27:16\n",
      "2018-09-09 23:00:15 Iter 35700 [Train]: loss=67.02, epe=4.44, lr=0.000063, samples/sec=28.8, sec/step=1.112, eta=4:25:09\n",
      "2018-09-09 23:02:17 Iter 35800 [Train]: loss=68.43, epe=4.55, lr=0.000061, samples/sec=28.5, sec/step=1.122, eta=4:25:31\n",
      "2018-09-09 23:04:19 Iter 35900 [Train]: loss=68.24, epe=4.50, lr=0.000060, samples/sec=28.6, sec/step=1.120, eta=4:23:10\n",
      "2018-09-09 23:06:21 Iter 36000 [Train]: loss=67.42, epe=4.45, lr=0.000059, samples/sec=28.6, sec/step=1.120, eta=4:21:26\n",
      "2018-09-09 23:06:41 Iter 36000 [Val]: loss=72.09, epe=4.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-36000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-36000\n",
      "2018-09-09 23:09:00 Iter 36100 [Train]: loss=71.84, epe=4.79, lr=0.000058, samples/sec=28.6, sec/step=1.119, eta=4:19:11\n",
      "2018-09-09 23:11:03 Iter 36200 [Train]: loss=70.88, epe=4.71, lr=0.000057, samples/sec=28.2, sec/step=1.133, eta=4:20:41\n",
      "2018-09-09 23:13:06 Iter 36300 [Train]: loss=66.86, epe=4.42, lr=0.000055, samples/sec=28.4, sec/step=1.128, eta=4:17:33\n",
      "2018-09-09 23:15:09 Iter 36400 [Train]: loss=70.61, epe=4.71, lr=0.000054, samples/sec=28.3, sec/step=1.131, eta=4:16:18\n",
      "2018-09-09 23:17:12 Iter 36500 [Train]: loss=69.35, epe=4.60, lr=0.000053, samples/sec=28.4, sec/step=1.127, eta=4:13:33\n",
      "2018-09-09 23:19:15 Iter 36600 [Train]: loss=67.56, epe=4.45, lr=0.000052, samples/sec=28.3, sec/step=1.130, eta=4:12:28\n",
      "2018-09-09 23:21:18 Iter 36700 [Train]: loss=68.48, epe=4.53, lr=0.000050, samples/sec=28.4, sec/step=1.127, eta=4:09:50\n",
      "2018-09-09 23:23:20 Iter 36800 [Train]: loss=66.80, epe=4.40, lr=0.000049, samples/sec=28.4, sec/step=1.126, eta=4:07:42\n",
      "2018-09-09 23:25:22 Iter 36900 [Train]: loss=67.12, epe=4.44, lr=0.000048, samples/sec=28.6, sec/step=1.118, eta=4:04:01\n",
      "2018-09-09 23:27:25 Iter 37000 [Train]: loss=67.39, epe=4.46, lr=0.000047, samples/sec=28.4, sec/step=1.126, eta=4:04:03\n",
      "2018-09-09 23:27:45 Iter 37000 [Val]: loss=72.16, epe=4.92\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-37000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-37000\n",
      "2018-09-09 23:30:05 Iter 37100 [Train]: loss=64.48, epe=4.24, lr=0.000046, samples/sec=28.4, sec/step=1.125, eta=4:01:57\n",
      "2018-09-09 23:32:08 Iter 37200 [Train]: loss=68.01, epe=4.50, lr=0.000044, samples/sec=28.3, sec/step=1.131, eta=4:01:13\n",
      "2018-09-09 23:34:10 Iter 37300 [Train]: loss=67.46, epe=4.47, lr=0.000043, samples/sec=28.5, sec/step=1.123, eta=3:57:48\n",
      "2018-09-09 23:36:13 Iter 37400 [Train]: loss=66.43, epe=4.38, lr=0.000042, samples/sec=28.4, sec/step=1.125, eta=3:56:13\n",
      "2018-09-09 23:38:16 Iter 37500 [Train]: loss=65.97, epe=4.33, lr=0.000041, samples/sec=28.3, sec/step=1.131, eta=3:55:38\n",
      "2018-09-09 23:40:19 Iter 37600 [Train]: loss=66.81, epe=4.38, lr=0.000039, samples/sec=28.2, sec/step=1.135, eta=3:54:33\n",
      "2018-09-09 23:42:22 Iter 37700 [Train]: loss=66.83, epe=4.40, lr=0.000038, samples/sec=28.3, sec/step=1.131, eta=3:51:52\n",
      "2018-09-09 23:44:25 Iter 37800 [Train]: loss=66.57, epe=4.39, lr=0.000037, samples/sec=28.2, sec/step=1.133, eta=3:50:25\n",
      "2018-09-09 23:46:28 Iter 37900 [Train]: loss=69.90, epe=4.66, lr=0.000036, samples/sec=28.4, sec/step=1.127, eta=3:47:18\n",
      "2018-09-09 23:48:31 Iter 38000 [Train]: loss=68.31, epe=4.54, lr=0.000035, samples/sec=28.4, sec/step=1.127, eta=3:45:22\n",
      "2018-09-09 23:48:51 Iter 38000 [Val]: loss=69.84, epe=4.75\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-38000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-38000\n",
      "2018-09-09 23:51:11 Iter 38100 [Train]: loss=66.78, epe=4.41, lr=0.000033, samples/sec=28.3, sec/step=1.132, eta=3:44:35\n",
      "2018-09-09 23:53:15 Iter 38200 [Train]: loss=67.08, epe=4.43, lr=0.000032, samples/sec=28.2, sec/step=1.134, eta=3:42:58\n",
      "2018-09-09 23:55:18 Iter 38300 [Train]: loss=65.21, epe=4.28, lr=0.000031, samples/sec=28.1, sec/step=1.137, eta=3:41:47\n",
      "2018-09-09 23:57:22 Iter 38400 [Train]: loss=66.24, epe=4.37, lr=0.000030, samples/sec=28.2, sec/step=1.134, eta=3:39:09\n",
      "2018-09-09 23:59:25 Iter 38500 [Train]: loss=65.26, epe=4.30, lr=0.000028, samples/sec=28.2, sec/step=1.136, eta=3:37:50\n",
      "2018-09-10 00:01:29 Iter 38600 [Train]: loss=64.72, epe=4.24, lr=0.000027, samples/sec=28.2, sec/step=1.135, eta=3:35:44\n",
      "2018-09-10 00:03:33 Iter 38700 [Train]: loss=67.66, epe=4.47, lr=0.000026, samples/sec=28.1, sec/step=1.139, eta=3:34:28\n",
      "2018-09-10 00:05:36 Iter 38800 [Train]: loss=67.68, epe=4.47, lr=0.000025, samples/sec=28.2, sec/step=1.137, eta=3:32:10\n",
      "2018-09-10 00:07:40 Iter 38900 [Train]: loss=65.97, epe=4.35, lr=0.000023, samples/sec=28.2, sec/step=1.133, eta=3:29:35\n",
      "2018-09-10 00:09:43 Iter 39000 [Train]: loss=65.70, epe=4.31, lr=0.000022, samples/sec=28.3, sec/step=1.131, eta=3:27:25\n",
      "2018-09-10 00:10:04 Iter 39000 [Val]: loss=67.01, epe=4.53\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-39000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-39000\n",
      "2018-09-10 00:12:25 Iter 39100 [Train]: loss=65.98, epe=4.35, lr=0.000021, samples/sec=28.1, sec/step=1.140, eta=3:27:10\n",
      "2018-09-10 00:14:30 Iter 39200 [Train]: loss=66.88, epe=4.42, lr=0.000020, samples/sec=28.0, sec/step=1.142, eta=3:25:31\n",
      "2018-09-10 00:16:35 Iter 39300 [Train]: loss=62.07, epe=4.06, lr=0.000019, samples/sec=27.8, sec/step=1.150, eta=3:25:03\n",
      "2018-09-10 00:18:39 Iter 39400 [Train]: loss=65.20, epe=4.28, lr=0.000017, samples/sec=28.1, sec/step=1.140, eta=3:21:25\n",
      "2018-09-10 00:20:43 Iter 39500 [Train]: loss=63.63, epe=4.16, lr=0.000016, samples/sec=28.0, sec/step=1.141, eta=3:19:42\n",
      "2018-09-10 00:22:48 Iter 39600 [Train]: loss=64.40, epe=4.23, lr=0.000015, samples/sec=28.1, sec/step=1.140, eta=3:17:35\n",
      "2018-09-10 00:24:52 Iter 39700 [Train]: loss=68.14, epe=4.51, lr=0.000014, samples/sec=28.0, sec/step=1.145, eta=3:16:31\n",
      "2018-09-10 00:26:57 Iter 39800 [Train]: loss=65.28, epe=4.29, lr=0.000012, samples/sec=27.9, sec/step=1.146, eta=3:14:47\n",
      "2018-09-10 00:29:01 Iter 39900 [Train]: loss=65.26, epe=4.30, lr=0.000011, samples/sec=28.0, sec/step=1.142, eta=3:12:15\n",
      "2018-09-10 00:31:06 Iter 40000 [Train]: loss=64.96, epe=4.26, lr=0.000010, samples/sec=27.9, sec/step=1.146, eta=3:10:56\n",
      "2018-09-10 00:31:27 Iter 40000 [Val]: loss=72.14, epe=4.90\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-40000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-40000\n",
      "2018-09-10 00:33:48 Iter 40100 [Train]: loss=65.94, epe=4.33, lr=0.000011, samples/sec=28.2, sec/step=1.136, eta=3:07:23\n",
      "2018-09-10 00:35:53 Iter 40200 [Train]: loss=67.90, epe=4.48, lr=0.000011, samples/sec=27.9, sec/step=1.148, eta=3:07:29\n",
      "2018-09-10 00:37:58 Iter 40300 [Train]: loss=66.52, epe=4.37, lr=0.000012, samples/sec=27.8, sec/step=1.150, eta=3:05:58\n",
      "2018-09-10 00:40:04 Iter 40400 [Train]: loss=67.07, epe=4.45, lr=0.000012, samples/sec=27.8, sec/step=1.152, eta=3:04:22\n",
      "2018-09-10 00:42:09 Iter 40500 [Train]: loss=63.60, epe=4.16, lr=0.000013, samples/sec=27.8, sec/step=1.151, eta=3:02:13\n",
      "2018-09-10 00:44:14 Iter 40600 [Train]: loss=65.73, epe=4.32, lr=0.000014, samples/sec=27.8, sec/step=1.151, eta=3:00:23\n",
      "2018-09-10 00:46:20 Iter 40700 [Train]: loss=66.87, epe=4.43, lr=0.000014, samples/sec=27.8, sec/step=1.150, eta=2:58:20\n",
      "2018-09-10 00:48:25 Iter 40800 [Train]: loss=64.49, epe=4.23, lr=0.000015, samples/sec=27.9, sec/step=1.148, eta=2:55:59\n",
      "2018-09-10 00:50:30 Iter 40900 [Train]: loss=64.06, epe=4.20, lr=0.000016, samples/sec=27.8, sec/step=1.151, eta=2:54:33\n",
      "2018-09-10 00:52:35 Iter 41000 [Train]: loss=62.21, epe=4.06, lr=0.000016, samples/sec=27.9, sec/step=1.146, eta=2:51:51\n",
      "2018-09-10 00:52:56 Iter 41000 [Val]: loss=68.32, epe=4.65\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-41000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-41000\n",
      "2018-09-10 00:55:18 Iter 41100 [Train]: loss=65.29, epe=4.29, lr=0.000017, samples/sec=27.7, sec/step=1.154, eta=2:51:12\n",
      "2018-09-10 00:57:25 Iter 41200 [Train]: loss=62.55, epe=4.07, lr=0.000017, samples/sec=27.5, sec/step=1.165, eta=2:50:54\n",
      "2018-09-10 00:59:32 Iter 41300 [Train]: loss=61.96, epe=4.04, lr=0.000018, samples/sec=27.5, sec/step=1.165, eta=2:48:55\n",
      "2018-09-10 01:01:38 Iter 41400 [Train]: loss=66.05, epe=4.34, lr=0.000019, samples/sec=27.7, sec/step=1.155, eta=2:45:34\n",
      "2018-09-10 01:03:44 Iter 41500 [Train]: loss=63.10, epe=4.11, lr=0.000019, samples/sec=27.6, sec/step=1.160, eta=2:44:19\n",
      "2018-09-10 01:05:49 Iter 41600 [Train]: loss=64.68, epe=4.23, lr=0.000020, samples/sec=27.8, sec/step=1.152, eta=2:41:20\n",
      "2018-09-10 01:07:55 Iter 41700 [Train]: loss=65.80, epe=4.33, lr=0.000020, samples/sec=27.6, sec/step=1.158, eta=2:40:08\n",
      "2018-09-10 01:10:01 Iter 41800 [Train]: loss=65.25, epe=4.31, lr=0.000021, samples/sec=27.7, sec/step=1.156, eta=2:38:00\n",
      "2018-09-10 01:12:06 Iter 41900 [Train]: loss=65.87, epe=4.33, lr=0.000022, samples/sec=27.8, sec/step=1.150, eta=2:35:16\n",
      "2018-09-10 01:14:13 Iter 42000 [Train]: loss=64.19, epe=4.22, lr=0.000022, samples/sec=27.5, sec/step=1.162, eta=2:34:55\n",
      "2018-09-10 01:14:34 Iter 42000 [Val]: loss=64.70, epe=4.31\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-42000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-42000\n",
      "2018-09-10 01:16:57 Iter 42100 [Train]: loss=64.43, epe=4.23, lr=0.000023, samples/sec=27.5, sec/step=1.164, eta=2:33:19\n",
      "2018-09-10 01:19:04 Iter 42200 [Train]: loss=65.10, epe=4.28, lr=0.000023, samples/sec=27.4, sec/step=1.169, eta=2:31:56\n",
      "2018-09-10 01:21:11 Iter 42300 [Train]: loss=66.12, epe=4.35, lr=0.000024, samples/sec=27.5, sec/step=1.163, eta=2:29:11\n",
      "2018-09-10 01:23:17 Iter 42400 [Train]: loss=63.79, epe=4.19, lr=0.000025, samples/sec=27.5, sec/step=1.162, eta=2:27:15\n",
      "2018-09-10 01:25:24 Iter 42500 [Train]: loss=66.54, epe=4.38, lr=0.000025, samples/sec=27.4, sec/step=1.166, eta=2:25:44\n",
      "2018-09-10 01:27:30 Iter 42600 [Train]: loss=66.64, epe=4.38, lr=0.000026, samples/sec=27.5, sec/step=1.163, eta=2:23:25\n",
      "2018-09-10 01:29:37 Iter 42700 [Train]: loss=66.50, epe=4.37, lr=0.000027, samples/sec=27.4, sec/step=1.168, eta=2:22:07\n",
      "2018-09-10 01:31:43 Iter 42800 [Train]: loss=67.41, epe=4.44, lr=0.000027, samples/sec=27.5, sec/step=1.162, eta=2:19:30\n",
      "2018-09-10 01:33:50 Iter 42900 [Train]: loss=65.45, epe=4.29, lr=0.000028, samples/sec=27.4, sec/step=1.169, eta=2:18:17\n",
      "2018-09-10 01:35:56 Iter 43000 [Train]: loss=65.00, epe=4.26, lr=0.000028, samples/sec=27.5, sec/step=1.164, eta=2:15:50\n",
      "2018-09-10 01:36:18 Iter 43000 [Val]: loss=68.13, epe=4.58\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-43000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-43000\n",
      "2018-09-10 01:38:41 Iter 43100 [Train]: loss=65.17, epe=4.29, lr=0.000029, samples/sec=27.7, sec/step=1.157, eta=2:13:01\n",
      "2018-09-10 01:40:49 Iter 43200 [Train]: loss=66.70, epe=4.40, lr=0.000030, samples/sec=27.3, sec/step=1.172, eta=2:12:48\n",
      "2018-09-10 01:42:56 Iter 43300 [Train]: loss=63.45, epe=4.15, lr=0.000030, samples/sec=27.4, sec/step=1.170, eta=2:10:37\n",
      "2018-09-10 01:45:03 Iter 43400 [Train]: loss=65.00, epe=4.26, lr=0.000031, samples/sec=27.4, sec/step=1.167, eta=2:08:23\n",
      "2018-09-10 01:47:09 Iter 43500 [Train]: loss=64.54, epe=4.25, lr=0.000031, samples/sec=27.5, sec/step=1.165, eta=2:06:12\n",
      "2018-09-10 01:49:16 Iter 43600 [Train]: loss=67.10, epe=4.43, lr=0.000032, samples/sec=27.4, sec/step=1.168, eta=2:04:34\n",
      "2018-09-10 01:51:23 Iter 43700 [Train]: loss=61.93, epe=4.03, lr=0.000033, samples/sec=27.4, sec/step=1.170, eta=2:02:48\n",
      "2018-09-10 01:53:29 Iter 43800 [Train]: loss=66.08, epe=4.37, lr=0.000033, samples/sec=27.6, sec/step=1.160, eta=1:59:52\n",
      "2018-09-10 01:55:36 Iter 43900 [Train]: loss=63.63, epe=4.16, lr=0.000034, samples/sec=27.5, sec/step=1.162, eta=1:58:09\n",
      "2018-09-10 01:57:42 Iter 44000 [Train]: loss=67.89, epe=4.49, lr=0.000035, samples/sec=27.4, sec/step=1.169, eta=1:56:56\n",
      "2018-09-10 01:58:04 Iter 44000 [Val]: loss=72.70, epe=4.99\n",
      "Saving model...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... model wasn't saved -- its score (4.99) doesn't outperform other checkpoints\n",
      "2018-09-10 02:00:23 Iter 44100 [Train]: loss=63.83, epe=4.17, lr=0.000035, samples/sec=27.4, sec/step=1.169, eta=1:54:56\n",
      "2018-09-10 02:02:30 Iter 44200 [Train]: loss=63.63, epe=4.16, lr=0.000036, samples/sec=27.4, sec/step=1.169, eta=1:52:59\n",
      "2018-09-10 02:04:37 Iter 44300 [Train]: loss=65.67, epe=4.33, lr=0.000036, samples/sec=27.3, sec/step=1.170, eta=1:51:11\n",
      "2018-09-10 02:06:44 Iter 44400 [Train]: loss=65.32, epe=4.30, lr=0.000037, samples/sec=27.3, sec/step=1.171, eta=1:49:15\n",
      "2018-09-10 02:08:51 Iter 44500 [Train]: loss=66.40, epe=4.36, lr=0.000038, samples/sec=27.4, sec/step=1.168, eta=1:47:02\n",
      "2018-09-10 02:10:59 Iter 44600 [Train]: loss=65.51, epe=4.32, lr=0.000038, samples/sec=27.3, sec/step=1.171, eta=1:45:24\n",
      "2018-09-10 02:13:05 Iter 44700 [Train]: loss=66.74, epe=4.41, lr=0.000039, samples/sec=27.5, sec/step=1.163, eta=1:42:43\n",
      "2018-09-10 02:15:12 Iter 44800 [Train]: loss=66.40, epe=4.38, lr=0.000039, samples/sec=27.3, sec/step=1.170, eta=1:41:24\n",
      "2018-09-10 02:17:19 Iter 44900 [Train]: loss=63.69, epe=4.18, lr=0.000040, samples/sec=27.4, sec/step=1.168, eta=1:39:15\n",
      "2018-09-10 02:19:26 Iter 45000 [Train]: loss=65.48, epe=4.30, lr=0.000041, samples/sec=27.5, sec/step=1.165, eta=1:37:07\n",
      "2018-09-10 02:19:48 Iter 45000 [Val]: loss=66.52, epe=4.45\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-45000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-45000\n",
      "2018-09-10 02:22:17 Iter 45100 [Train]: loss=66.41, epe=4.36, lr=0.000040, samples/sec=27.5, sec/step=1.162, eta=1:34:52\n",
      "2018-09-10 02:24:25 Iter 45200 [Train]: loss=64.72, epe=4.25, lr=0.000039, samples/sec=27.4, sec/step=1.169, eta=1:33:32\n",
      "2018-09-10 02:26:32 Iter 45300 [Train]: loss=67.11, epe=4.42, lr=0.000039, samples/sec=27.2, sec/step=1.176, eta=1:32:06\n",
      "2018-09-10 02:28:40 Iter 45400 [Train]: loss=63.26, epe=4.14, lr=0.000038, samples/sec=27.3, sec/step=1.173, eta=1:29:56\n",
      "2018-09-10 02:30:47 Iter 45500 [Train]: loss=64.10, epe=4.19, lr=0.000038, samples/sec=27.3, sec/step=1.170, eta=1:27:45\n",
      "2018-09-10 02:32:54 Iter 45600 [Train]: loss=65.74, epe=4.33, lr=0.000037, samples/sec=27.3, sec/step=1.171, eta=1:25:54\n",
      "2018-09-10 02:35:01 Iter 45700 [Train]: loss=66.68, epe=4.41, lr=0.000036, samples/sec=27.4, sec/step=1.167, eta=1:23:40\n",
      "2018-09-10 02:37:09 Iter 45800 [Train]: loss=68.41, epe=4.52, lr=0.000036, samples/sec=27.3, sec/step=1.171, eta=1:21:57\n",
      "2018-09-10 02:39:15 Iter 45900 [Train]: loss=64.35, epe=4.22, lr=0.000035, samples/sec=27.6, sec/step=1.159, eta=1:19:13\n",
      "2018-09-10 02:41:22 Iter 46000 [Train]: loss=64.06, epe=4.18, lr=0.000035, samples/sec=27.4, sec/step=1.167, eta=1:17:47\n",
      "2018-09-10 02:41:44 Iter 46000 [Val]: loss=69.17, epe=4.68\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-46000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-46000\n",
      "2018-09-10 02:44:09 Iter 46100 [Train]: loss=65.13, epe=4.27, lr=0.000034, samples/sec=27.3, sec/step=1.173, eta=1:16:16\n",
      "2018-09-10 02:46:17 Iter 46200 [Train]: loss=68.34, epe=4.52, lr=0.000033, samples/sec=27.2, sec/step=1.176, eta=1:14:29\n",
      "2018-09-10 02:48:26 Iter 46300 [Train]: loss=63.60, epe=4.16, lr=0.000033, samples/sec=27.0, sec/step=1.183, eta=1:12:57\n",
      "2018-09-10 02:50:34 Iter 46400 [Train]: loss=64.25, epe=4.21, lr=0.000032, samples/sec=27.0, sec/step=1.183, eta=1:11:00\n",
      "2018-09-10 02:52:43 Iter 46500 [Train]: loss=63.00, epe=4.12, lr=0.000031, samples/sec=26.9, sec/step=1.189, eta=1:09:20\n",
      "2018-09-10 02:54:52 Iter 46600 [Train]: loss=65.35, epe=4.29, lr=0.000031, samples/sec=27.0, sec/step=1.185, eta=1:07:08\n",
      "2018-09-10 02:57:01 Iter 46700 [Train]: loss=61.83, epe=4.02, lr=0.000030, samples/sec=27.1, sec/step=1.182, eta=1:05:01\n",
      "2018-09-10 02:59:10 Iter 46800 [Train]: loss=64.14, epe=4.20, lr=0.000030, samples/sec=26.9, sec/step=1.189, eta=1:03:25\n",
      "2018-09-10 03:01:19 Iter 46900 [Train]: loss=65.30, epe=4.30, lr=0.000029, samples/sec=27.1, sec/step=1.182, eta=1:01:04\n",
      "2018-09-10 03:03:27 Iter 47000 [Train]: loss=64.43, epe=4.21, lr=0.000028, samples/sec=27.1, sec/step=1.180, eta=0:59:00\n",
      "2018-09-10 03:03:49 Iter 47000 [Val]: loss=67.79, epe=4.59\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-47000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-47000\n",
      "2018-09-10 03:06:14 Iter 47100 [Train]: loss=66.13, epe=4.36, lr=0.000028, samples/sec=27.4, sec/step=1.167, eta=0:56:25\n",
      "2018-09-10 03:08:22 Iter 47200 [Train]: loss=63.41, epe=4.15, lr=0.000027, samples/sec=27.3, sec/step=1.174, eta=0:54:48\n",
      "2018-09-10 03:10:30 Iter 47300 [Train]: loss=65.01, epe=4.26, lr=0.000027, samples/sec=27.1, sec/step=1.179, eta=0:53:02\n",
      "2018-09-10 03:12:38 Iter 47400 [Train]: loss=63.47, epe=4.15, lr=0.000026, samples/sec=27.1, sec/step=1.181, eta=0:51:11\n",
      "2018-09-10 03:14:46 Iter 47500 [Train]: loss=66.88, epe=4.42, lr=0.000025, samples/sec=27.0, sec/step=1.185, eta=0:49:23\n",
      "2018-09-10 03:16:55 Iter 47600 [Train]: loss=65.66, epe=4.34, lr=0.000025, samples/sec=27.1, sec/step=1.181, eta=0:47:15\n",
      "2018-09-10 03:19:02 Iter 47700 [Train]: loss=63.11, epe=4.11, lr=0.000024, samples/sec=27.3, sec/step=1.173, eta=0:44:57\n",
      "2018-09-10 03:21:10 Iter 47800 [Train]: loss=66.06, epe=4.36, lr=0.000023, samples/sec=27.1, sec/step=1.180, eta=0:43:16\n",
      "2018-09-10 03:23:18 Iter 47900 [Train]: loss=64.28, epe=4.19, lr=0.000023, samples/sec=27.1, sec/step=1.180, eta=0:41:18\n",
      "2018-09-10 03:25:26 Iter 48000 [Train]: loss=63.50, epe=nan, lr=0.000022, samples/sec=27.2, sec/step=1.174, eta=0:39:09\n",
      "2018-09-10 03:25:47 Iter 48000 [Val]: loss=69.32, epe=4.71\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-48000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-48000\n",
      "2018-09-10 03:28:13 Iter 48100 [Train]: loss=65.24, epe=4.29, lr=0.000022, samples/sec=27.0, sec/step=1.184, eta=0:37:29\n",
      "2018-09-10 03:30:22 Iter 48200 [Train]: loss=63.18, epe=4.14, lr=0.000021, samples/sec=26.9, sec/step=1.187, eta=0:35:37\n",
      "2018-09-10 03:32:32 Iter 48300 [Train]: loss=63.11, epe=4.12, lr=0.000020, samples/sec=26.8, sec/step=1.195, eta=0:33:51\n",
      "2018-09-10 03:34:41 Iter 48400 [Train]: loss=64.04, epe=4.18, lr=0.000020, samples/sec=26.9, sec/step=1.188, eta=0:31:40\n",
      "2018-09-10 03:36:50 Iter 48500 [Train]: loss=65.20, epe=4.28, lr=0.000019, samples/sec=26.9, sec/step=1.189, eta=0:29:43\n",
      "2018-09-10 03:38:59 Iter 48600 [Train]: loss=62.35, epe=4.06, lr=0.000019, samples/sec=26.9, sec/step=1.190, eta=0:27:46\n",
      "2018-09-10 03:41:08 Iter 48700 [Train]: loss=65.08, epe=4.29, lr=0.000018, samples/sec=26.9, sec/step=1.188, eta=0:25:44\n",
      "2018-09-10 03:43:17 Iter 48800 [Train]: loss=63.87, epe=4.18, lr=0.000017, samples/sec=27.0, sec/step=1.184, eta=0:23:41\n",
      "2018-09-10 03:45:26 Iter 48900 [Train]: loss=63.96, epe=4.16, lr=0.000017, samples/sec=27.0, sec/step=1.183, eta=0:21:42\n",
      "2018-09-10 03:47:34 Iter 49000 [Train]: loss=63.00, epe=4.10, lr=0.000016, samples/sec=27.0, sec/step=1.186, eta=0:19:46\n",
      "2018-09-10 03:47:57 Iter 49000 [Val]: loss=68.58, epe=4.64\n",
      "Saving model...\n",
      "INFO:tensorflow:./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-49000 is not in all_model_checkpoint_paths. Manually adding it.\n",
      "... model saved in ./pwcnet-lg-6-2-cyclic-chairsthingsmix/pwcnet.ckpt-49000\n",
      "2018-09-10 03:50:22 Iter 49100 [Train]: loss=63.04, epe=4.09, lr=0.000016, samples/sec=27.0, sec/step=1.185, eta=0:17:47\n",
      "2018-09-10 03:52:31 Iter 49200 [Train]: loss=62.13, epe=4.05, lr=0.000015, samples/sec=26.9, sec/step=1.191, eta=0:15:52\n",
      "2018-09-10 03:54:41 Iter 49300 [Train]: loss=64.75, epe=4.23, lr=0.000014, samples/sec=26.8, sec/step=1.193, eta=0:13:55\n",
      "2018-09-10 03:56:50 Iter 49400 [Train]: loss=64.47, epe=4.23, lr=0.000014, samples/sec=26.9, sec/step=1.191, eta=0:11:54\n",
      "2018-09-10 03:58:59 Iter 49500 [Train]: loss=63.63, epe=4.17, lr=0.000013, samples/sec=27.0, sec/step=1.187, eta=0:09:53\n",
      "2018-09-10 04:01:09 Iter 49600 [Train]: loss=65.49, epe=4.31, lr=0.000012, samples/sec=26.9, sec/step=1.190, eta=0:07:56\n",
      "2018-09-10 04:03:17 Iter 49700 [Train]: loss=61.36, epe=3.99, lr=0.000012, samples/sec=27.0, sec/step=1.186, eta=0:05:56\n",
      "2018-09-10 04:05:26 Iter 49800 [Train]: loss=62.10, epe=4.04, lr=0.000011, samples/sec=27.0, sec/step=1.187, eta=0:03:57\n",
      "2018-09-10 04:07:35 Iter 49900 [Train]: loss=62.72, epe=4.09, lr=0.000011, samples/sec=27.0, sec/step=1.187, eta=0:01:59\n",
      "2018-09-10 04:09:44 Iter 50000 [Train]: loss=64.02, epe=4.20, lr=0.000010, samples/sec=27.0, sec/step=1.184, eta=0:00:00\n",
      "2018-09-10 04:10:06 Iter 50000 [Val]: loss=69.72, epe=4.74\n",
      "Saving model...\n",
      "... model wasn't saved -- its score (4.74) doesn't outperform other checkpoints\n",
      "... done training.\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "nn.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the training curves for the run above:\n",
    "\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/loss.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/epe.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/lr.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the predictions issued by the model for a few validation samples:\n",
    "\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val1.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val2.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val3.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val4.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val5.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val6.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val7.png)\n",
    "![](img/pwcnet-lg-6-2-cyclic-chairsthingsmix/val8.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}