TensorBoardOutputConfig not emitting TensorBoard loads to s3

[nsorros]

I am trying to send the logs created from TensorBoard to an s3 location so that I can monitor the training but I have not managed to make TensorBoardOutputConfig work. This is what I am trying:

from sagemaker.debugger import TensorBoardOutputConfig
    tensorboard_output_config = TensorBoardOutputConfig(
        s3_output_path=f"s3://{PROJECTS_BUCKET}/{PROJECT_NAME}/logs/",
        container_local_output_path='/logs/'
    )

    es = Estimator(
        ...,
        tensorboard_output_config=tensorboard_output_config
    )

I have tested the docker container and the logs are being created in /logs. Additionally the s3 location is correct and the same as used for saving models, data and outputs with the difference of the last directory which is logs in this case. No logs show up in s3 though.

In case it makes any difference I am only storing scalar logs, essentially metrics in tensorflow not tensors. so all the metrics are saved in /logs/scalar/....