Description
Environment information
Suggestion: Fix conflicting installations
Namely:
pip uninstall tensorboard tensorflow tensorflow-estimator tensorflow-gpu
pip install tensorflow # or tensorflow-gpu, or tf-nightly, ...
Issue description
System:
- Ubuntu 18.04
- TF2.0
Error:
I am using tf.keras and the model.fit() function to train my model. I have added the tf.keras.callbacks.TensorBoard callback to my .fit() call.
I am encountering an issue where if:
- profile_batch=2 --> Segmentation Fault (core dumped) when a call is made to
.fit() - profile_batch=0 --> No segfault (disabled profiling)
Note: This error only occurs sometimes; Run the model.fit() a couple of times to reproduce the error; This error happens independent of the directory being logged to
Background Info:
Currently, the input to my model is in the form of a tf.data.Dataset. The documentation for the callback says:
profile_batch: Profile the batch to sample compute characteristics. By default, it will profile the second batch. Set profile_batch=0 to disable profiling. Must run in TensorFlow eager mode.
I use the dataset.map() function in my input pipeline to transform my input data. However, since .map() does not execute eagerly, I wrap it around a tf.py_function, which should make it execute eagerly (I've verified that the map function does in fact run eagerly after using py_function).
However, sometimes, I still get a Segmentation Fault error, as described above. On the occasion where the Segmentation Fault does not occur, the profile logged to the Tensorboard is a .profile-empty file.
#2084 may be related.