In mxnet.monitor, does "outputs" means the intermediate activations in the forward() pass? Is there an efficient method to obtain activations of intermediate layers without extra computation?

[wydwww]

Dear developers,

I am trying to use mxnet.monitor to get the intermediate activations of a DNN with Module API v1.x (learned from #4805 (comment); though it is simpler than CustomOp). I suppose the printed names that match ".*output" are intermediate activations of those layers. For a full training loop of forward(), backward() and update(), I can get ".*output" (e.g., stage1_unit3_conv3_output) with a monitor.

However, for a model that is not for training, the monitor cannot obtain those outputs values, only print stage1_unit3_conv3.

I found another method to extract features by reconstructing a Module from the first layer to a certain layer from an example that was last updated in 2017 (https://github.com/dmlc/mxnet-notebooks/blob/master/python/how_to/predict.ipynb). This method runs an extra forward() for one intermediate result, which seems not efficient, especially when extracting intermediate results of multiple layers.

Another method mentioned in #4805 and #1538 is using BlockGrad and mx.sym.Group but I am not clear how it works.

My questions are:

1. Does the "outputs" (e.g., stage1_unit3_conv3_output) that caught by the monitor mean the intermediate activations in the forward operation?
2. Is there a more efficient method to obtain the intermediate activations of multiple layers, like the hook in PyTorch? mx.symbol.identity looks good but can I achieve this without modifying the DNN architecture?

Related previous discussion:

#4805
#1538
#5811

Thanks.

[ptrendx]

Could you clarify what it is that you try to accomplish? If you want to debug an output inside the graph then mxnet.monitor is a debug tool that lets you do that. It is typically used once per many iterations due to the high overhead. If instead what you want is to get some specific activations and use it in some other network for example, then you can group the activations you want together using mx.sym.Group and return as the result of the network - this will make the network produce multiple outputs.

[wydwww]

My goal is to get the activations of some layers.
Previously I was looking for a way that requires no modification to the original neural network definition. I will try mx.sym.Group.

[wydwww]

Since mx.sym.Group gives multiple outputs (say 2 outputs), the following update_metric() will give an error:
ValueError: Shape of labels 1 does not match shape of predictions 2.
I searched and the solution seems to be writing a custom mx.metric.EvalMetric class with its own update() implementation to bypass check_label_shapes().
I made a wrapper metric and use get_outputs() to get the intermediate activations.
Thanks for pointing out that the monitor is for debugging.