shell: assignment of tasks to shells contradicts taskmap on partially allocated elcap nodes

[garlick]

Problem: on MI300 nodes with 96 cores, running a job that requires 97 thru 191 cores without specifying a node count seems to cause the shell to assign the wrong ranks to nodes compared to the taskmap.

Example:

[garlick@tuolumne1005:~]$ flux run --label-io -n 191 hostname | grep 190:
190: tuolumne1005
[garlick@tuolumne1005:~]$ flux job taskmap --to=hosts $(flux job last)
tuolumne1005: 0-95
tuolumne1006: 96-190

Adding a userrc that prints the shell hostname confirms this:

garlick@elcap1063:~]$ flux run -n192 -o mpibind=off -o spindle.level=off -o verbose -o userrc=rc.lua true 2>&1 |grep elcap
0.058s: flux-shell[0]: elcap1063
0.066s: flux-shell[1]: elcap1064
[garlick@elcap1063:~]$ flux run -n191 -o mpibind=off -o spindle.level=off -o verbose -o userrc=rc.lua true 2>&1 |grep elcap
0.066s: flux-shell[0]: elcap1064
0.058s: flux-shell[1]: elcap1063
[garlick@elcap1063:~]$ flux job taskmap --to=hosts $(flux job last)
elcap1063: 0-95
elcap1064: 96-190

This does not reproduce if -N2 is added.

[garlick@elcap1063:~]$ flux run -n97 -N2 -o mpibind=off -o spindle.level=off -o verbose -o userrc=rc.lua true 2>&1 |grep elcap
0.059s: flux-shell[0]: elcap1063
0.092s: flux-shell[1]: elcap1064
[garlick@elcap1063:~]$ flux job taskmap --to=hosts $(flux job last)
elcap1063: 0-48
elcap1064: 49-96

First noted in flux-framework/flux-coral2#222