cmd: add grow option to submission

[milroy]

To support job grow (#2791, and numerous issues in flux-sched), a user will need a way to specify the job type from the command line, e.g., flux mini submit --type=grow .... The attribute will be used by the flux-sched qmanager to take type-dependent actions.

@garlick suggested using

flux-core/src/cmd/flux-mini.py

Line 96 in a360528

"--setattr",

which I believe will insert the attr into the jobspec. @dongahn would prefer to have the attribute outside of the jobspec, because that will require the qmanager to inspect the jobspec to handle different types.

Is there another way to achieve the desired functionality?

[edited to reflect that the qmanager not the queue itself is taking actions]

[dongahn]

@dongahn would prefer to have the attribute outside of the jobspec, because that will require the flux-sched queue to inspect the jobspec to handle different types.

Yes, currently qmanager doesn't crack open the jobspec at all. Just toss it to resource for matching. And there is a performance advantage with that.

The only other option I can think of is to pass this (t_type?) to alloc request along with t_submit, t_prio etc.

Now that I think about this, though, it seems we will keep encountering the problems of the same class. For example a same issue will arise when we consider "partition or queue". If users want to specify what queue within qmanager they want to run a job, where should they go? If that goes to a jobspec again, the qmanager needs to crack it open to determine which queue of the multiple queues so that it will be triaged...

Despite my preference, it seems maybe good time to consider making qmanager jobspec aware.

[grondo]

Despite my preference, it seems maybe good time to consider making qmanager jobspec aware.

This is probably wise. We should try to avoid moving user-specified job data out of the jobspec as much as possible IMO, since the design of Jobspec and J is to encapsulate this information with authenticity and integrity on behalf of the user.

I don't think we will find too much of a performance hit. If so, there might be future mitigations, e.g. a scheduler-provided validator plugin or script could stash these values since it has to parse the entire jobspec anyway.

Will qmanager also need to examine walltime (or is it only resource) ? This is also defined within jobspec.

[dongahn]

Will qmanager also need to examine walltime (or is it only resource) ? This is also defined within jobspec

No wall time is used by resource which has been the only service that examined it.