Clean up SklImpute #2421

Job	Run time
lint	1m 7s
Released (ubuntu-20.04, py3.8)	19m 25s
Released (ubuntu-20.04, py3.9)	12m 47s
Released (ubuntu-20.04, py3.10)	14m 30s
Released (ubuntu-20.04, py3.11)	13m 12s
Latest (ubuntu-20.04, py3.10)	9m 23s
Oldest dependencies (ubuntu-20.04, py3.8)	19m 20s
PyQt6 (ubuntu-20.04, py3.9)	4m 53s
Released (macos-latest, py3.8)	22m 52s
Released (macos-latest, py3.9)	30m 13s
Released (macos-latest, py3.10)	20m 37s
Released (macos-latest, py3.11)	21m 13s
Released (windows-latest, py3.8)	22m 41s
Released (windows-latest, py3.9)	28m 38s
Released (windows-latest, py3.10)	27m 28s
Released (windows-latest, py3.11)	16m 55s
Latest (windows-latest, py3.10)	25m 1s
Latest (macos-latest, py3.10)	23m 30s
PyQt6 (windows-latest, py3.10)	11m 44s
PyQt6 (macos-latest, py3.9)	11m 47s
	5h 57m 16s

Provide feedback