Add percentage based threshold function and unit tests (#1679) #5867

Job	Run time
build_test (linux.2xlarge, 3.10, py310) / linux-job	8m 28s
build_test (linux.2xlarge, 3.11, py311) / linux-job	7m 56s
build_test (linux.2xlarge, 3.8, py38) / linux-job	8m 14s
build_test (linux.2xlarge, 3.9, py39) / linux-job	8m 20s
	32m 58s

Provide feedback