Loading Sagemaker NTM Artifacts

[salmanmashayekh]

I have trained a Neural Topic Model with Sagemaker and now I am trying to load/deploy the model locally. The artifacts include a symbol and an a parameters file.

I am using the following to load the model:

sym, arg_params, aux_params = mx.model.load_checkpoint(prefix, iteration)
module_model = mx.mod.Module(symbol=sym, label_names=None, context=mx.cpu())

But when I try to bind the model:

module_model.bind(
    for_training = False,
    data_shapes = [('data', (1, VOCAB_SIZE))]
)

It fails with the following error:

---------------------------------------------------------------------------
MXNetError                                Traceback (most recent call last)
~/anaconda3/envs/python3/lib/python3.6/site-packages/mxnet/symbol/symbol.py in simple_bind(self, ctx, grad_req, type_dict, stype_dict, group2ctx, shared_arg_names, shared_exec, shared_buffer, **kwargs)
   1622                                                    shared_exec_handle,
-> 1623                                                    ctypes.byref(exe_handle)))
   1624         except MXNetError as e:

~/anaconda3/envs/python3/lib/python3.6/site-packages/mxnet/base.py in check_call(ret)
    252     if ret != 0:
--> 253         raise MXNetError(py_str(_LIB.MXGetLastError()))
    254 

MXNetError: Error in operator sample_normal0: vector::_M_range_insert

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-349-822926d8cf09> in <module>()
      8     for_training = True,
      9     data_shapes = [('data', (1,VOCAB_SIZE))],
---> 10     force_rebind = True,
     11 )
     12 

~/anaconda3/envs/python3/lib/python3.6/site-packages/mxnet/module/module.py in bind(self, data_shapes, label_shapes, for_training, inputs_need_grad, force_rebind, shared_module, grad_req)
    427                                                      fixed_param_names=self._fixed_param_names,
    428                                                      grad_req=grad_req, group2ctxs=self._group2ctxs,
--> 429                                                      state_names=self._state_names)
    430         self._total_exec_bytes = self._exec_group._total_exec_bytes
    431         if shared_module is not None:

~/anaconda3/envs/python3/lib/python3.6/site-packages/mxnet/module/executor_group.py in __init__(self, symbol, contexts, workload, data_shapes, label_shapes, param_names, for_training, inputs_need_grad, shared_group, logger, fixed_param_names, grad_req, state_names, group2ctxs)
    277         self.num_outputs = len(self.symbol.list_outputs())
    278 
--> 279         self.bind_exec(data_shapes, label_shapes, shared_group)
    280 
    281     def decide_slices(self, data_shapes):

~/anaconda3/envs/python3/lib/python3.6/site-packages/mxnet/module/executor_group.py in bind_exec(self, data_shapes, label_shapes, shared_group, reshape)
    373             else:
    374                 self.execs.append(self._bind_ith_exec(i, data_shapes_i, label_shapes_i,
--> 375                                                       shared_group))
    376 
    377         self.data_shapes = data_shapes

~/anaconda3/envs/python3/lib/python3.6/site-packages/mxnet/module/executor_group.py in _bind_ith_exec(self, i, data_shapes, label_shapes, shared_group)
    660                                            type_dict=input_types, shared_arg_names=self.param_names,
    661                                            shared_exec=shared_exec, group2ctx=group2ctx,
--> 662                                            shared_buffer=shared_data_arrays, **input_shapes)
    663         self._total_exec_bytes += int(executor.debug_str().split('\n')[-3].split()[1])
    664         return executor

~/anaconda3/envs/python3/lib/python3.6/site-packages/mxnet/symbol/symbol.py in simple_bind(self, ctx, grad_req, type_dict, stype_dict, group2ctx, shared_arg_names, shared_exec, shared_buffer, **kwargs)
   1627                 error_msg += "%s: %s\n" % (k, v)
   1628             error_msg += "%s" % e
-> 1629             raise RuntimeError(error_msg)
   1630 
   1631         # update shared_buffer

RuntimeError: simple_bind error. Arguments:
data: (1, 52908)
Error in operator sample_normal0: vector::_M_range_insert

From the model architecture (https://arxiv.org/pdf/1809.02687.pdf), I know that the input data shape is a vector with VOCAB_SIZE length.

Any ideas what I am doing wrong?

[lanking520]

@salmanmashayekh is there any code piece available to reproduce?
@ThomasDelteil Could you please take a look here?

[salmanmashayekh]

Thank you both @lanking520 and @ThomasDelteil!

Here is a MWE:

from sklearn.feature_extraction.text import CountVectorizer
import s3fs
from sagemaker.amazon.amazon_estimator import get_image_uri
from sagemaker import get_execution_role

fs = s3fs.S3FileSystem()


# create the document corpus
docs = [
    "Python is an interpreted, high-level, general-purpose programming language.",
    "Created by Guido van Rossum and first released in 1991, Python's design philosophy emphasizes code readability with its notable use of significant whitespace.",
    "Its language constructs and object-oriented approach aim to help programmers write clear, logical code for small and large-scale projects.",
    "Python is dynamically typed and garbage-collected.",
    "It supports multiple programming paradigms, including procedural, object-oriented, and functional programming.",
    "Python is often described as a batteries included language due to its comprehensive standard library.",
    "Python was conceived in the late 1980s as a successor to the ABC language.",
    "Python 2.0, released 2000, introduced features like list comprehensions and a garbage collection system capable of collecting reference cycles.",
    "Python 3.0, released 2008, was a major revision of the language that is not completely backward-compatible, and much Python 2 code does not run unmodified on Python 3.",
    "Due to concern about the amount of code written for Python 2, support for Python 2.7 (the last release in the 2.x series) was extended to 2020.",
    "Language developer Guido van Rossum shouldered sole responsibility for the project until July 2018 but now shares his leadership as a member of a five-person steering council.",
    "The Python 2 language, i.e. Python 2.7.x, is sunsetting on January 1, 2020, and the Python team of volunteers will not fix security issues, or improve in other ways after that date.",
    "With the end-of-life, only Python 3.6.x and later, e.g.",
    "Python 3.8 which should be released in October 2019 (currently in beta), will be supported.",
    "Python interpreters are available for many operating systems.",
    "A global community of programmers develops and maintains CPython, an open source reference implementation.",
    "A non-profit organization, the Python Software Foundation, manages and directs resources for Python and CPython development.",
]
DOCS_NO = len(docs)

# fir the vectorizer and transform the docs
count_vectorizer = CountVectorizer(input="content", max_df=0.8, min_df=5, max_features=10)
transformed_docs = count_vectorizer.fit_transform(docs)
transformed_docs = transformed_docs.astype(np.float32) # transformed_docs is a 17x8 sparse matrix

# s3 params
BUCKET = "my-bucket"
PATH = "mxnet-mwe"
TRAIN_DIR  = f"{PATH}/train"
AUX_DIR    = f"{PATH}/auxiliary"
OUTPUT_DIR = f"{PATH}/output"

# store the transformed docs into s3
with fs.open(f"s3://{BUCKET}/{TRAIN_DIR}/transformed_docs.protobuf", "wb") as buf:
    smac.write_spmatrix_to_sparse_tensor(buf, transformed_docs)
    
# store the vocab into s3
vocab_dict = {i:w for w,i in count_vectorizer.vocabulary_.items()}
vocab = [str(vocab_dict[i]) for i in range(len(vocab_dict))]
with fs.open(f"s3://{BUCKET}/{AUX_DIR}/vocab.txt", "wb") as f:
    f.write(("\n".join(vocab)).encode())

# get the training image, role, and session
container = get_image_uri(boto3.Session().region_name, 'ntm')
role = get_execution_role()
session = sagemaker.Session()

# instantiate the estimator
ntm = sagemaker.estimator.Estimator(
    container,
    role, 
    train_instance_count = 1, 
    train_instance_type = "ml.p2.xlarge",
    output_path = f"s3://{BUCKET}/{OUTPUT_DIR}",
    sagemaker_session = session,
)

# set hyper-params
ntm.set_hyperparameters(
    num_topics = 3,
    feature_dim = len(vocab),
)

# set training and auxiliary channels
train_channel = sagemaker.session.s3_input(
    f"s3://{BUCKET}/{TRAIN_DIR}/transformed_docs.protobuf",
    content_type = 'application/x-recordio-protobuf'
)
auxiliary_channel = sagemaker.session.s3_input(
    f"s3://{BUCKET}/{AUX_DIR}/vocab.txt",
    content_type = 'text/plain'
)

# fit the model
ntm.fit(
    {
        'train': train_channel,
        'auxiliary': auxiliary_channel,
    }
)

The training log is as follows:

2019-09-25 21:56:29 Starting - Starting the training job...
2019-09-25 21:56:35 Starting - Launching requested ML instances......
2019-09-25 21:57:34 Starting - Preparing the instances for training......
2019-09-25 21:58:45 Downloading - Downloading input data...
2019-09-25 21:59:04 Training - Downloading the training image..Docker entrypoint called with argument(s): train
/opt/amazon/lib/python2.7/site-packages/pandas/util/nosetester.py:13: DeprecationWarning: Importing from numpy.testing.nosetester is deprecated, import from numpy.testing instead.
  from numpy.testing import nosetester
[09/25/2019 21:59:36 INFO 140704525621056] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/default-input.json: {u'num_patience_epochs': u'3', u'clip_gradient': u'Inf', u'encoder_layers': u'auto', u'optimizer': u'adadelta', u'_kvstore': u'auto_gpu', u'rescale_gradient': u'1.0', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'learning_rate': u'0.01', u'_data_format': u'record', u'sub_sample': u'1.0', u'epochs': u'50', u'weight_decay': u'0.0', u'_num_kv_servers': u'auto', u'encoder_layers_activation': u'sigmoid', u'mini_batch_size': u'256', u'tolerance': u'0.001', u'batch_norm': u'false'}
[09/25/2019 21:59:36 INFO 140704525621056] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'feature_dim': u'8', u'num_topics': u'3'}
[09/25/2019 21:59:36 INFO 140704525621056] Final configuration: {u'num_patience_epochs': u'3', u'clip_gradient': u'Inf', u'encoder_layers': u'auto', u'optimizer': u'adadelta', u'_kvstore': u'auto_gpu', u'rescale_gradient': u'1.0', u'_tuning_objective_metric': u'', u'_num_gpus': u'auto', u'learning_rate': u'0.01', u'_data_format': u'record', u'sub_sample': u'1.0', u'epochs': u'50', u'feature_dim': u'8', u'weight_decay': u'0.0', u'num_topics': u'3', u'_num_kv_servers': u'auto', u'encoder_layers_activation': u'sigmoid', u'mini_batch_size': u'256', u'tolerance': u'0.001', u'batch_norm': u'false'}
[09/25/2019 21:59:37 INFO 140704525621056] nvidia-smi took: 0.0503778457642 secs to identify 1 gpus
Process 1 is a worker.
[09/25/2019 21:59:37 INFO 140704525621056] Using default worker.
[09/25/2019 21:59:37 INFO 140704525621056] Initializing
/opt/amazon/lib/python2.7/site-packages/ai_algorithms_sdk/config/config_helper.py:122: DeprecationWarning: deprecated
  warnings.warn("deprecated", DeprecationWarning)
[09/25/2019 21:59:37 INFO 140704525621056] /opt/ml/input/data/auxiliary
[09/25/2019 21:59:37 INFO 140704525621056] vocab.txt
[09/25/2019 21:59:37 INFO 140704525621056] Vocab file vocab.txt is expected at /opt/ml/input/data/auxiliary
[09/25/2019 21:59:37 INFO 140704525621056] Loading pre-trained token embedding vectors from /opt/amazon/lib/python2.7/site-packages/algorithm/s3_binary/glove.6B.50d.txt

2019-09-25 21:59:58 Uploading - Uploading generated training model
2019-09-25 21:59:58 Completed - Training job completed
[09/25/2019 21:59:49 WARNING 140704525621056] 0 out of 8 in vocabulary do not have embeddings! Default vector used for unknown embedding!
[09/25/2019 21:59:49 INFO 140704525621056] Vocab embedding shape
[09/25/2019 21:59:49 INFO 140704525621056] Number of GPUs being used: 1
[09/25/2019 21:59:53 INFO 140704525621056] Create Store: device
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Batches Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Number of Records Since Last Reset": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Batches Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Total Records Seen": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Max Records Seen Between Resets": {"count": 1, "max": 0, "sum": 0.0, "min": 0}, "Reset Count": {"count": 1, "max": 0, "sum": 0.0, "min": 0}}, "EndTime": 1569448793.550828, "Dimensions": {"Host": "algo-1", "Meta": "init_train_data_iter", "Operation": "training", "Algorithm": "AWS/NTM"}, "StartTime": 1569448793.55079}

[2019-09-25 21:59:53.551] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 0, "duration": 16503, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] 
[09/25/2019 21:59:53 INFO 140704525621056] # Starting training for epoch 1
[2019-09-25 21:59:53.571] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 2, "duration": 19, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] # Finished training epoch 1 on 17 examples from 1 batches, each of size 256.
[09/25/2019 21:59:53 INFO 140704525621056] Metrics for Training:
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) total: 0.13865429163
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) kld: 0.000402930745622
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) recons: 0.13825134933
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) logppx: 0.13865429163
[09/25/2019 21:59:53 INFO 140704525621056] #quality_metric: host=algo-1, epoch=1, train total_loss <loss>=0.13865429163
[09/25/2019 21:59:53 INFO 140704525621056] Timing: train: 0.03s, val: 0.00s, epoch: 0.03s
[09/25/2019 21:59:53 INFO 140704525621056] #progress_metric: host=algo-1, completed 2 % of epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Total Batches Seen": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Total Records Seen": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Max Records Seen Between Resets": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Reset Count": {"count": 1, "max": 2, "sum": 2.0, "min": 2}}, "EndTime": 1569448793.577662, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/NTM", "epoch": 0}, "StartTime": 1569448793.551103}

[09/25/2019 21:59:53 INFO 140704525621056] #throughput_metric: host=algo-1, train throughput=636.442222897 records/second
[09/25/2019 21:59:53 INFO 140704525621056] 
[09/25/2019 21:59:53 INFO 140704525621056] # Starting training for epoch 2
[2019-09-25 21:59:53.591] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 5, "duration": 13, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] # Finished training epoch 2 on 17 examples from 1 batches, each of size 256.
[09/25/2019 21:59:53 INFO 140704525621056] Metrics for Training:
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) total: 0.137792155147
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) kld: 0.000175397624844
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) recons: 0.137616753578
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) logppx: 0.137792155147
[09/25/2019 21:59:53 INFO 140704525621056] #quality_metric: host=algo-1, epoch=2, train total_loss <loss>=0.137792155147
[09/25/2019 21:59:53 INFO 140704525621056] Timing: train: 0.01s, val: 0.00s, epoch: 0.02s
[09/25/2019 21:59:53 INFO 140704525621056] #progress_metric: host=algo-1, completed 4 % of epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Total Batches Seen": {"count": 1, "max": 2, "sum": 2.0, "min": 2}, "Total Records Seen": {"count": 1, "max": 34, "sum": 34.0, "min": 34}, "Max Records Seen Between Resets": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Reset Count": {"count": 1, "max": 4, "sum": 4.0, "min": 4}}, "EndTime": 1569448793.597192, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/NTM", "epoch": 1}, "StartTime": 1569448793.577978}

[09/25/2019 21:59:53 INFO 140704525621056] #throughput_metric: host=algo-1, train throughput=876.919088438 records/second
[09/25/2019 21:59:53 INFO 140704525621056] 
[09/25/2019 21:59:53 INFO 140704525621056] # Starting training for epoch 3
[2019-09-25 21:59:53.609] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 8, "duration": 11, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] # Finished training epoch 3 on 17 examples from 1 batches, each of size 256.
[09/25/2019 21:59:53 INFO 140704525621056] Metrics for Training:
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) total: 0.136722803116
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) kld: 5.14156054123e-05
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) recons: 0.136671379209
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) logppx: 0.136722803116
[09/25/2019 21:59:53 INFO 140704525621056] #quality_metric: host=algo-1, epoch=3, train total_loss <loss>=0.136722803116
[09/25/2019 21:59:53 INFO 140704525621056] Timing: train: 0.01s, val: 0.00s, epoch: 0.02s
[09/25/2019 21:59:53 INFO 140704525621056] #progress_metric: host=algo-1, completed 6 % of epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Total Batches Seen": {"count": 1, "max": 3, "sum": 3.0, "min": 3}, "Total Records Seen": {"count": 1, "max": 51, "sum": 51.0, "min": 51}, "Max Records Seen Between Resets": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Reset Count": {"count": 1, "max": 6, "sum": 6.0, "min": 6}}, "EndTime": 1569448793.615088, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/NTM", "epoch": 2}, "StartTime": 1569448793.597521}

[09/25/2019 21:59:53 INFO 140704525621056] #throughput_metric: host=algo-1, train throughput=958.182731976 records/second
[09/25/2019 21:59:53 INFO 140704525621056] 
[09/25/2019 21:59:53 INFO 140704525621056] # Starting training for epoch 4
[2019-09-25 21:59:53.628] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 11, "duration": 12, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] # Finished training epoch 4 on 17 examples from 1 batches, each of size 256.
[09/25/2019 21:59:53 INFO 140704525621056] Metrics for Training:
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) total: 0.136693164706
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) kld: 4.85915334139e-05
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) recons: 0.13664457202
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) logppx: 0.136693164706
[09/25/2019 21:59:53 INFO 140704525621056] #quality_metric: host=algo-1, epoch=4, train total_loss <loss>=0.136693164706
[09/25/2019 21:59:53 INFO 140704525621056] patience losses:[0.13865429162979126, 0.13779215514659882, 0.13672280311584473] min patience loss:0.136722803116 current loss:0.136693164706 absolute loss difference:2.96384096146e-05
[09/25/2019 21:59:53 INFO 140704525621056] Bad epoch: loss has not improved (enough). Bad count:1
[09/25/2019 21:59:53 INFO 140704525621056] Timing: train: 0.01s, val: 0.00s, epoch: 0.02s
[09/25/2019 21:59:53 INFO 140704525621056] #progress_metric: host=algo-1, completed 8 % of epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Total Batches Seen": {"count": 1, "max": 4, "sum": 4.0, "min": 4}, "Total Records Seen": {"count": 1, "max": 68, "sum": 68.0, "min": 68}, "Max Records Seen Between Resets": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Reset Count": {"count": 1, "max": 8, "sum": 8.0, "min": 8}}, "EndTime": 1569448793.633979, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/NTM", "epoch": 3}, "StartTime": 1569448793.615416}

[09/25/2019 21:59:53 INFO 140704525621056] #throughput_metric: host=algo-1, train throughput=908.019866032 records/second
[09/25/2019 21:59:53 INFO 140704525621056] 
[09/25/2019 21:59:53 INFO 140704525621056] # Starting training for epoch 5
[2019-09-25 21:59:53.648] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 14, "duration": 14, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] # Finished training epoch 5 on 17 examples from 1 batches, each of size 256.
[09/25/2019 21:59:53 INFO 140704525621056] Metrics for Training:
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) total: 0.136731252074
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) kld: 3.38535865012e-05
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) recons: 0.136697411537
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) logppx: 0.136731252074
[09/25/2019 21:59:53 INFO 140704525621056] #quality_metric: host=algo-1, epoch=5, train total_loss <loss>=0.136731252074
[09/25/2019 21:59:53 INFO 140704525621056] patience losses:[0.13779215514659882, 0.13672280311584473, 0.13669316470623016] min patience loss:0.136693164706 current loss:0.136731252074 absolute loss difference:3.80873680115e-05
[09/25/2019 21:59:53 INFO 140704525621056] Bad epoch: loss has not improved (enough). Bad count:2
[09/25/2019 21:59:53 INFO 140704525621056] Timing: train: 0.02s, val: 0.00s, epoch: 0.02s
[09/25/2019 21:59:53 INFO 140704525621056] #progress_metric: host=algo-1, completed 10 % of epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Total Batches Seen": {"count": 1, "max": 5, "sum": 5.0, "min": 5}, "Total Records Seen": {"count": 1, "max": 85, "sum": 85.0, "min": 85}, "Max Records Seen Between Resets": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Reset Count": {"count": 1, "max": 10, "sum": 10.0, "min": 10}}, "EndTime": 1569448793.649995, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/NTM", "epoch": 4}, "StartTime": 1569448793.634279}

[09/25/2019 21:59:53 INFO 140704525621056] #throughput_metric: host=algo-1, train throughput=1073.22869442 records/second
[09/25/2019 21:59:53 INFO 140704525621056] 
[09/25/2019 21:59:53 INFO 140704525621056] # Starting training for epoch 6
[2019-09-25 21:59:53.665] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 17, "duration": 14, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] # Finished training epoch 6 on 17 examples from 1 batches, each of size 256.
[09/25/2019 21:59:53 INFO 140704525621056] Metrics for Training:
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) total: 0.135983735323
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) kld: 2.08785422728e-05
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) recons: 0.135962858796
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) logppx: 0.135983735323
[09/25/2019 21:59:53 INFO 140704525621056] #quality_metric: host=algo-1, epoch=6, train total_loss <loss>=0.135983735323
[09/25/2019 21:59:53 INFO 140704525621056] patience losses:[0.13672280311584473, 0.13669316470623016, 0.13673125207424164] min patience loss:0.136693164706 current loss:0.135983735323 absolute loss difference:0.000709429383278
[09/25/2019 21:59:53 INFO 140704525621056] Bad epoch: loss has not improved (enough). Bad count:3
[09/25/2019 21:59:53 INFO 140704525621056] Timing: train: 0.02s, val: 0.01s, epoch: 0.02s
[09/25/2019 21:59:53 INFO 140704525621056] #progress_metric: host=algo-1, completed 12 % of epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Total Batches Seen": {"count": 1, "max": 6, "sum": 6.0, "min": 6}, "Total Records Seen": {"count": 1, "max": 102, "sum": 102.0, "min": 102}, "Max Records Seen Between Resets": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Reset Count": {"count": 1, "max": 12, "sum": 12.0, "min": 12}}, "EndTime": 1569448793.673135, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/NTM", "epoch": 5}, "StartTime": 1569448793.650263}

[09/25/2019 21:59:53 INFO 140704525621056] #throughput_metric: host=algo-1, train throughput=737.784344767 records/second
[09/25/2019 21:59:53 INFO 140704525621056] 
[09/25/2019 21:59:53 INFO 140704525621056] # Starting training for epoch 7
[2019-09-25 21:59:53.691] [tensorio] [info] epoch_stats={"data_pipeline": "/opt/ml/input/data/train", "epoch": 20, "duration": 17, "num_examples": 1, "num_bytes": 816}
[09/25/2019 21:59:53 INFO 140704525621056] # Finished training epoch 7 on 17 examples from 1 batches, each of size 256.
[09/25/2019 21:59:53 INFO 140704525621056] Metrics for Training:
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) total: 0.135596558452
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) kld: 5.20053035871e-05
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) recons: 0.135544538498
[09/25/2019 21:59:53 INFO 140704525621056] Loss (name: value) logppx: 0.135596558452
[09/25/2019 21:59:53 INFO 140704525621056] #quality_metric: host=algo-1, epoch=7, train total_loss <loss>=0.135596558452
[09/25/2019 21:59:53 INFO 140704525621056] patience losses:[0.13669316470623016, 0.13673125207424164, 0.13598373532295227] min patience loss:0.135983735323 current loss:0.135596558452 absolute loss difference:0.0003871768713
[09/25/2019 21:59:53 INFO 140704525621056] Bad epoch: loss has not improved (enough). Bad count:4
[09/25/2019 21:59:53 INFO 140704525621056] Bad epochs exceeded patience. Stopping training early!
[09/25/2019 21:59:53 INFO 140704525621056] Timing: train: 0.02s, val: 0.00s, epoch: 0.02s
[09/25/2019 21:59:53 INFO 140704525621056] Early stop condition met. Stopping training.
[09/25/2019 21:59:53 INFO 140704525621056] #progress_metric: host=algo-1, completed 100 % epochs
#metrics {"Metrics": {"Max Batches Seen Between Resets": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Batches Since Last Reset": {"count": 1, "max": 1, "sum": 1.0, "min": 1}, "Number of Records Since Last Reset": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Total Batches Seen": {"count": 1, "max": 7, "sum": 7.0, "min": 7}, "Total Records Seen": {"count": 1, "max": 119, "sum": 119.0, "min": 119}, "Max Records Seen Between Resets": {"count": 1, "max": 17, "sum": 17.0, "min": 17}, "Reset Count": {"count": 1, "max": 14, "sum": 14.0, "min": 14}}, "EndTime": 1569448793.696538, "Dimensions": {"Host": "algo-1", "Meta": "training_data_iter", "Operation": "training", "Algorithm": "AWS/NTM", "epoch": 6}, "StartTime": 1569448793.673541}

[09/25/2019 21:59:53 INFO 140704525621056] #throughput_metric: host=algo-1, train throughput=733.51886181 records/second
[09/25/2019 21:59:53 WARNING 140704525621056] wait_for_all_workers will not sync workers since the kv store is not running distributed
[09/25/2019 21:59:53 INFO 140704525621056] Best model based on early stopping at epoch 7. Best loss: 0.135596558452
[09/25/2019 21:59:53 INFO 140704525621056] Topics from epoch:final (num_topics:3) [wetc 0.63, tu 0.33]:
[09/25/2019 21:59:53 INFO 140704525621056] [0.63, 0.33] python and the for in is language of
[09/25/2019 21:59:53 INFO 140704525621056] [0.63, 0.33] in and of python the for is language
[09/25/2019 21:59:53 INFO 140704525621056] [0.63, 0.33] python and language for of the in is
[09/25/2019 21:59:53 INFO 140704525621056] Serializing model to /opt/ml/model/model_algo-1
[09/25/2019 21:59:53 INFO 140704525621056] Saved checkpoint to "/tmp/tmp3kswyI/state-0001.params"
[09/25/2019 21:59:53 INFO 140704525621056] Test data is not provided.
#metrics {"Metrics": {"totaltime": {"count": 1, "max": 16767.024993896484, "sum": 16767.024993896484, "min": 16767.024993896484}, "finalize.time": {"count": 1, "max": 12.12000846862793, "sum": 12.12000846862793, "min": 12.12000846862793}, "initialize.time": {"count": 1, "max": 16501.816987991333, "sum": 16501.816987991333, "min": 16501.816987991333}, "model.serialize.time": {"count": 1, "max": 3.983020782470703, "sum": 3.983020782470703, "min": 3.983020782470703}, "setuptime": {"count": 1, "max": 66.67494773864746, "sum": 66.67494773864746, "min": 66.67494773864746}, "early_stop.time": {"count": 7, "max": 6.479024887084961, "sum": 24.256229400634766, "min": 0.23102760314941406}, "update.time": {"count": 7, "max": 26.340961456298828, "sum": 141.98827743530273, "min": 15.537023544311523}, "epochs": {"count": 1, "max": 50, "sum": 50.0, "min": 50}}, "EndTime": 1569448793.714689, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/NTM"}, "StartTime": 1569448777.046916}

When the training is done, it generates the following zip file, which includes a metadata, a symbol, and a parameters file: https://drive.google.com/open?id=1TLnIrnmB1SzPqN7cgyECql1Ri74isPKG

I am trying to load the symbol/parameters file into an mxnet model, so that I can make predictions locally (outside of Sagemaker).

Can you share the code snippet to create an mxnet model with the artifacts and predict on the transformed_docs?

[ydennisy]

Hey @salmanmashayekh did you ever get this working?

Using your code:

sym, arg_params, aux_params = mx.model.load_checkpoint('model_algo-1', 0)
module_model = mx.mod.Module(symbol=sym, label_names=None, context=mx.cpu())
module_model.bind(
    for_training = False,
    data_shapes = [('data', (1, VOCAB_SIZE))]
)

Runs fine for me, the next step for inference I assume is to set:

module_model.set_params(arg_params=arg_params, aux_params=aux_params)

But this throws an error, which requests epochs, but I am not sure why this is needed for prediction...

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-65-e655955661fb> in <module>
----> 1 module_model.set_params(arg_params=arg_params, aux_params=aux_params)

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/mxnet/module/module.py in set_params(self, arg_params, aux_params, allow_missing, force_init, allow_extra)
    348             self.init_params(initializer=None, arg_params=arg_params, aux_params=aux_params,
    349                              allow_missing=allow_missing, force_init=force_init,
--> 350                              allow_extra=allow_extra)
    351             return
    352 

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/mxnet/module/module.py in init_params(self, initializer, arg_params, aux_params, allow_missing, force_init, allow_extra)
    307         for name, arr in sorted(self._arg_params.items()):
    308             desc = InitDesc(name, attrs.get(name, None))
--> 309             _impl(desc, arr, arg_params)
    310 
    311         for name, arr in sorted(self._aux_params.items()):

~/.pyenv/versions/3.7.8/lib/python3.7/site-packages/mxnet/module/module.py in _impl(name, arr, cache)
    298                 else:
    299                     if not allow_missing:
--> 300                         raise RuntimeError("%s is not presented" % name)
    301                     if initializer is not None:
    302                         initializer(name, arr)

RuntimeError: n_epoch is not presented

Small update: setting allow_missing=True seems to set the model params:

module_model.set_params(arg_params=arg_params, aux_params=aux_params, allow_missing=True)

But calling predict, gives an odd output!

module_model.predict(x)

OUTPUT:

[11.285855]
<NDArray 1 @cpu(0)>

[ydennisy]

@lanking520 @ThomasDelteil would appreciate if you could take a look!