AP_NavEKF3: fix computation of rho #43514

Job	Run time
build (stm32f7, chibios, 10)	15m 20s
build (stm32h7, chibios, 10)	15m 48s
build (fmuv2-plane, chibios, 10)	8m 7s
build (periph-build, chibios, 10)	14m 19s
build (iofirmware, chibios, 10)	4m 35s
build (CubeOrange-bootloader, chibios, 10)	3m 23s
build (fmuv3-bootloader, chibios, 10)	3m 31s
build (revo-bootloader, chibios, 10)	3m 19s
build (stm32h7-debug, chibios, 10)	9m 21s
build (fmuv3, chibios, 10)	15m 54s
build (revo-mini, chibios, 10)	12m 25s
build (MatekF405-Wing, chibios, 10)	12m 59s
build (CubeOrange-ODID, chibios, 10)	10m 33s
build (CubeRedPrimary-bootloader, chibios, 10)	3m 18s
build (configure-all, chibios, 10)	13m 41s
build (build-options-defaults-test, chibios, 10)	25m 51s
build (signing, chibios, 10)	11m 22s
build (CubeOrange-PPP, chibios, 10)	8m 46s
build (CubeRed-EKF2, chibios, 10)	9m 2s
build (SOHW, chibios, 10)	13m 17s
build (Pixhawk6X-PPPGW, chibios, 10)	4m 59s
build (new-check, chibios, 10)	15m 31s
	3h 55m 21s

Provide feedback