Visualizing NAS experiments using Google Colab

[Ekhao]

Hello,

I am trying use this library to do some NAS experiments in Google Colab. For that I would like to be able to access the experiments web interface while it is running.

I have found this tutorial on your documentation web page explaining a way to access the web interface of a normal experiment running in Google Colab. From my understanding this method requires that I first start the experiment and then use ngrok to publish the experiments web interface to the web. This works well in the example as it is possible to use the CLI to start an experiment and run it as a background process.

However, if I try to do the same in the NAS tutorial here I run into problems. According to my understanding of the retiarii experment documentation found here it is not possible to run a retiarii experiment in the background. Does this mean that I am unable to visualize this kind of experiment when I am using Google Colab?

[ultmaster]

Maybe you can start ngrok first. See #4779.

[Ekhao]

Hello,

I did try to start ngrok first as described in that issue discussion thread. I can however not reliably make ngrok work when starting it before the NNI experiment. My best guess is that there is some timing involved in when to run the ngrok commands to successfully start the server - I am however not at all an expert on the tool.

I have created two Jupyter Notebooks that you can try to run in Google Colab to see the issue. As can be seen in the notebooks I was able to start the server in the simple test case, however after 5-6 tries of starting the server for the NAS experiment I gave up on trying to make it run there.

NNIWebInterfaceTest.zip