Make TensorFlow e2e for flwr-datasets consistent with flwr #6124

Job	Run time
Build, test and upload wheel	1m 30s
Strategy / FedMedian	1m 7s
Framework / bare	1m 37s
Strategy / FedTrimmedAvg	1m 6s
Framework / bare-https	1m 23s
Strategy / QFedAvg	1m 13s
Framework / jax	2m 8s
Strategy / FaultTolerantFedAvg	1m 9s
Framework / pytorch	2m 37s
Strategy / FedAvgM	1m 9s
Framework / tensorflow	3m 11s
Strategy / FedAdam	1m 20s
Framework / tabnet	2m 43s
Strategy / FedAdagrad	1m 7s
Framework / opacus	2m 43s
Strategy / FedYogi	1m 5s
Framework / pytorch-lightning	2m 43s
Framework / mxnet	1m 42s
Framework / scikit-learn	2m 25s
Framework / fastai	4m 57s
Framework / pandas	1m 37s
	40m 32s

Provide feedback