Fix the smoke test (except the benchmark runs)

Fix the smoke test (except the benchmark runs) #1187