Description
It seems like run_experiment in GCP takes longer to generate a report. It has been more than 5+ hours since I started executing following command:
PYTHONPATH=. python3 experiment/run_experiment.py \
--experiment-config experiment-config.yaml \
--benchmarks lcms_cms_transform_fuzzer \
--experiment-name $EXPERIMENT_NAME \
--fuzzers aflplusplus
for 2 trials, each of them lasting for 30 minutes, but it hasn’t finished generating report data. Running the same command and configuration in the local experiment already terminated.
The run_experiment is looping in this loop:
fuzzbench/experiment/dispatcher.py
Line 166 in ba22647
It seems like the processes are sleeping for the majority of time. htop output also shows that measuring processes are getting scheduled, but there’s less than 1% usage of CPU for each scheduled process.
Is there any solution to make it generate report faster?
run_experiment script also steadily outputs this warning in case it is relevant.
/usr/local/lib/python3.10/site-packages/jinja2/runtime.py:298: FutureWarning: this method is deprecated in favour of `Styler.to_html()`
return __obj(*args, **kwargs)
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:202: RuntimeWarning: All-NaN slice encountered
vmin = np.nanmin(calc_data)
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:207: RuntimeWarning: All-NaN slice encountered
vmax = np.nanmax(calc_data)
in the loop DEBUG
/usr/local/lib/python3.10/site-packages/jinja2/runtime.py:298: FutureWarning: this method is deprecated in favour of `Styler.to_html()`
return __obj(*args, **kwargs)
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:202: RuntimeWarning: All-NaN slice encountered
vmin = np.nanmin(calc_data)
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:207: RuntimeWarning: All-NaN slice encountered
vmax = np.nanmax(calc_data)