forked from SchedMD/slurm
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathNEWS
3726 lines (3621 loc) · 210 KB
/
NEWS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
This file describes changes in recent versions of SLURM. It primarily
documents those changes that are of interest to users and admins.
* Changes in Slurm 14.03.0pre5
==============================
-- Added squeue format option of "%X" (core specialization count).
-- Added core specialization web page (just a start for now).
-- Added the SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID
in epilogi slurmctld environment.
-- Fix bug in job step allocation failing due to memory limit.
-- Modify the pbsnodes script to reflect its output on a TORQUE system.
-- Add ability to clear a node's DRAIN flag using scontrol or sview by setting
it's state to "UNDRAIN". The node's base state (e.g. "DOWN" or "IDLE") will
not be changed.
-- Modify the output of 'scontrol show partition' by displaying
DefMemPerCPU=UNLIMITED and MaxMemPerCPU=UNLIMITED when these limits are
configured as 0.
-- mpirun-mic - Major re-write of the command wrapper for Xeon Phi use.
-- Add new configuration parameter of AuthInfo to specify port used by
authentication plugin.
-- Fixed conditional rpm compiling.
-- Corrected slurmstepd ident name when loggind to syslog.
-- Fixed sh5util loop when there are no node-step files.
-- Add SLURM_CLUSTER_NAME to environment variables passed to PrologSlurmctld,
Prolog, EpilogSlurmctld, and Epilog
-- Add the idea of running a prolog right when an allocation happens
instead of when running on the node for the first time.
-- If user runs 'scontrol reconfig' but hostnames or the host count changes
the slurmctld throws a fatal error.
* Changes in Slurm 14.03.0pre4
==============================
-- Remove the ThreadID documentation from slurm.conf. This functionality has
been obsoleted by the LogTimeFormat.
-- Sched plugins - rename global and plugin functions names for consistency
with other plugin types.
-- BGQ - Added RebootQOSList option to bluegene.conf to allow an implicate
reboot of a block if only jobs in the list are running on it when cnodes
go into a failure state.
-- Correct task count of pending job steps.
-- Improve limit enforcement for jobs, set RLIMIT_RSS, RLIMIT_AS and/or
RLIMIT_DATA to enforce memory limit.
-- Pending job steps will have step_id of INFINITE rather than NO_VAL and
will be reported as "TBD" by scontrol and squeue commands.
-- Add logic so PMI_Abort or PMI2_Abort can propagate an exit code.
-- Added SlurmdPlugstack configuration parameter.
-- Added PriorityFlag DEPTH_OBLIVIOUS to have the depth of an association
not effect it's priorty.
-- Multi-thread the sinfo command (one thread per partition).
-- Added sgather tool to gather files from a job's compute nodes into a
central location.
-- Added configuration parameter FairShareDampeningFactor to offer a greater
priority range based upon utilization.
-- Change MaxArraySize and job's array_task_id from 16-bit to 32-bit field.
Additional Slurm enhancements are be required to support larger job arrays.
-- Added -S/--core-spec option to salloc, sbatch and srun commands to reserve
specialized cores for system use. Modify scontrol and sview to get/set
the new field. No enforcement exists yet for these new options.
struct job_info / slurm_job_info_t: Added core_spec
struct job_descriptorjob_desc_msg_t: Added core_spec
* Changes in Slurm 14.03.0pre3
==============================
-- Do not set SLURM_NODEID environment variable on front-end systems.
-- Convert bitmap functions to use int32_t instead of int in data structures
and function arguments. This is to reliably enable use of bitmaps containing
up to 4 billion elements. Several data structures containing index values
were also changed from data type int to int32_t:
- Struct job_info / slurm_job_info_t: Changed exc_node_inx, node_inx, and
req_node_inx from type int to type int32_t
- job_step_info_t: Changed node_inx from type int to type int32_t
- Struct partition_info / partition_info_t: Changed node_inx from type int
to type int32_t
- block_job_info_t: Changed cnode_inx from type int to type int32_t
- block_info_t: Changed ionode_inx and mp_inx from type int to type int32_t
- Struct reserve_info / reserve_info_t: Changed node_inx from type int to
type int32_t
-- Modify qsub wrapper output to match torque command output, just print the
job ID rather than "Submitted batch job #"
-- Change Slurm error string for ESLURM_MISSING_TIME_LIMIT from
"Missing time limit" to
"Time limit specification required, but not provided"
-- Change salloc job_allocate error message header from
"Failed to allocate resources" to
"Job submit/allocate failed"
-- Modify slurmctld message retry logic to support Cray cold-standby SDB.
* Changes in Slurm 14.03.0pre2
==============================
-- Added "JobAcctGatherParams" configuration parameter. Value of "NoShare"
disables accounting for shared memory.
-- Added fields to "scontrol show job" output: boards_per_node,
sockets_per_board, ntasks_per_node, ntasks_per_board, ntasks_per_socket,
ntasks_per_core, and nice.
-- Add squeue output format options for job command and working directory
(%o and %Z respectively).
-- Add stdin/out/err to sview job output.
-- Add new job_state of JOB_BOOT_FAIL for job terminations due to failure to
boot it's allocated nodes or BlueGene block.
-- CRAY - Add SelectTypeParameters NHC_NO_STEPS and NHC_NO which will disable
the node health check script for steps and allocations respectfully.
-- Reservation with CoreCnt: Avoid possible invalid memory reference.
-- Add new error code for attempt to create a reservation with duplicate name.
-- Validate that a hostlist file contains text (i.e. not a binary).
-- switch/generic - propagate switch information from srun down to slurmd and
slurmstepd.
-- CRAY - Do not package Slurm's libpmi or libpmi2 libraries. The Cray version
of those libraries must be used.
-- Added a new option to the scontrol command to view licenses that are
configured in use and avalable. 'scontrol show licenses'.
-- MySQL - Made Slurm compatible with 5.6
* Changes in Slurm 14.03.0pre1
==============================
-- sview - improve scalability
-- Add task pointer to the task_post_term() function in task plugins. The
terminating task's PID is available in task->pid.
-- Move select/cray to select/alps
-- Defer sending SIGKILL signal to processes while core dump in progress.
-- Added JobContainerPlugin configuration parameter and plugin infrastructure.
-- Added partition configuration parameters AllowAccounts, AllowQOS,
DenyAccounts and DenyQOS.
-- The rpmbuild option for a cray system with ALPS has changed from
%_with_cray to %_with_cray_alps.
-- The log file timestamp format can now be selected at runtime via the
LogTimeFormat configuration option. See the slurm.conf and slurmdbd.conf
man pages for details.
-- Added switch/generic plugin to a job's convey network topology.
-- BLUEGENE - If block is in 'D' state or has more cnodes in error than
MaxBlockInError set the job wait reason appropriately.
-- API use: Generate an error return rather than fatal error and exit if the
configuraiton file is absent or invalid. This will permit Slurm APIs to be
more reliably used by other programs.
-- Add support for load-based scheduling, allocate jobs to nodes with the
largest number of available CPUs. Added SchedulingParameters paramter of
"CR_LLN" and partition parameter of "LLN=yes|no".
-- Added job_info() and step_info() functions to the gres plugins to extract
plugin specific fields from the job's or step's GRES data structure.
-- Added sbatch --signal option of "B:" to signal the batch shell rather than
only the spawned job steps.
-- Added sinfo and squeue format option of "%all" to print all fields available
for the data type with a vertical bar separating each field.
-- Add mechanism for job_submit plugin to generate error message for srun,
salloc or sbatch to stderr. New argument added to job_submit function in
the plugin.
-- Add StdIn, StdOut, and StdErr paths to job information dumped with
"scontrol show job".
-- Permit Slurm administrator to submit a batch job as any user.
-- Set a job's RLIMIT_AS limit based upon it's memory limit and VsizeFactor
configuration value.
-- Remove Postgres plugins
-- Make jobacct_gather/cgroup work correctly and also make all jobacct_gather
plugins more maintainable.
-- Proctrack/pgid - Add support for proctrack_p_plugin_get_pids() function.
-- Sched/backfill - Change default max_job_bf parameter from 50 to 100.
-- Added -I|--item-extract option to sh5util to extract data item from series.
* Changes in Slurm 2.6.5
========================
-- Correction to hostlist parsing bug introduced in v2.6.4 for hostlists with
more than one numeric range in brackets (e.g. rack[0-3]_blade[0-63]").
-- Add notification if using proctrack/cgroup and task/cgroup when oom hits.
-- Corrections to advanced reservation logic with overlapping jobs.
-- job_submit/lua - add cpus_per_task field to those available.
-- Add cpu_load to the node information available using the Perl API.
-- Correct a job's GRES allocation data in accounting records for non-Cray
systems.
-- Substantial performance improvement for systems with Shared=YES or FORCE
and large numbers of running jobs (replace bubble sort with quick sort).
-- proctrack/cgroup - Add locking to prevent race condition where one job step
is ending for a user or job at the same time another job stepsis starting
and the user or job container is deleted from under the starting job step.
-- Fixed sh5util loop when there are no node-step files.
-- Fix race condition on batch job termination that could result in a job exit
code of 0xfffffffe if the slurmd on node zero registers its active jobs at
the same time that slurmstepd is recording the job's exit code.
-- Correct logic returning remaining job dependencies in job information
reported by scontrol and squeue. Eliminates vestigial descriptors with
no job ID values (e.g. "afterany").
-- Improve performance of REQUEST_JOB_INFO_SINGLE RPC by removing unnecessary
locks and use hash function to find the desired job.
-- jobcomp/filetxt - Reopen the file when slurmctld daemon is reconfigured
or gets SIGHUP.
-- Remove notice of CVE with very old/deprecated versions of Slurm in
news.html.
-- Fix if hwloc_get_nbobjs_by_type() returns zero core count (set to 1).
-- Added ApbasilTimeout parameter to the cray.conf configuration file.
-- Handle in the API if parts of the node structure are NULL.
-- Fix srun hang when IO fails to start at launch.
-- Fix for GRES bitmap not matching the GRES count resulting in abort
(requires manual resetting of GRES count, changes to gres.conf file,
and slurmd restarts).
-- Modify sview to better support job arrays.
-- Modify squeue to support longer job ID values (for many job array tasks).
* Changes in Slurm 2.6.4
========================
-- Fixed sh5util to print its usage.
-- Corrected commit f9a3c7e4e8ec.
-- Honor ntasks-per-node option with exclusive node allocations.
-- sched/backfill - Prevent invalid memory reference if bf_continue option is
configured and slurm is reconfigured during one of the sleep cycles or if
there are any changes to the partition configuration or if the normal
scheduler runs and starts a job that the backfill scheduler is actively
working on.
-- Update man pages information about acct-freq and JobAcctGatherFrequency
to reflect only the latest supported format.
-- Minor document update to include note about PrivateData=Usage for the
slurm.conf when using the DBD.
-- Expand information reported with DebugFlags=backfill.
-- Initiate jobs pending to run in a reservation as soon as the reservation
becomes active.
-- Purged expired reservation even if it has pending jobs.
-- Corrections to calculation of a pending job's expected start time.
-- Remove some vestigial logic treating job priority of 1 as a special case.
-- Memory freeing up to avoid minor memory leaks at close of daemons
-- Updated documentation to give correct units being displayed.
-- Report AccountingStorageBackupHost with "scontrol show config".
-- init scripts ignore quotes around Pid file name specifications.
-- Fixed typo about command case in quickstart.html.
-- task/cgroup - handle new cpuset files, similar to commit c4223940.
-- Replace the tempname() function call with mkstemp().
-- Fix for --cpu_bind=map_cpu/mask_cpu/map_ldom/mask_ldom plus
--mem_bind=map_mem/mask_mem options, broken in 2.6.2.
-- Restore default behavior of allocating cores to jobs on a cyclic basis
across the sockets unless SelectTypeParameters=CR_CORE_DEFAULT_DIST_BLOCK
or user specifies other distribution options.
-- Enforce JobRequeue configuration parameter on node failure. Previously
always requeued the job.
-- acct_gather_energy/ipmi - Add delay before retry on read error.
-- select/cons_res with GRES and multiple threads per core, fix possible
infinite loop.
-- proctrack/cgroup - Add cgroup create retry logic in case one step is
starting at the same time as another step is ending and the logic to create
and delete cgroups overlaps.
-- Improve setting of job wait "Reason" field.
-- Correct sbatch documentation and job_submit/pbs plugin "%j" is job ID,
not "%J" (which is job_id.step_id).
-- Improvements to sinfo performance, especially for large numbers of
partitions.
-- SlurmdDebug - Permit changes to slurmd debug level with "scontrol reconfig"
-- smap - Avoid invalid memory reference with hidden nodes.
-- Fix sacctmgr modify qos set preempt+/-=.
-- BLUEGENE - fix issue where node count wasn't set up correctly when srun
preforms the allocation, regression in 2.6.3.
-- Add support for dependencies of job array elements (e.g.
"sbatch --depend=afterok:123_4 ...") or all elements of a job array (e.g.
"sbatch --depend=afterok:123 ...").
-- Add support for new options in sbatch qsub wrapper:
-W block=true (wait for job completion)
Clear PBS_NODEFILE environment variable
-- Fixed the MaxSubmitJobsPerUser limit in QOS which limited submissions
a job too early.
-- sched/wiki, sched/wiki2 - Fix to work with change logic introduced in
version 2.6.3 preventing Maui/Moab from starting jobs.
-- Updated the QOS limits documentation and man page.
* Changes in Slurm 2.6.3
========================
-- Add support for some new #PBS options in sbatch scripts and qsub wrapper:
-l accelerator=true|false (GPU use)
-l mpiprocs=# (processors per node)
-l naccelerators=# (GPU count)
-l select=# (node count)
-l ncpus=# (task count)
-v key=value (environment variable)
-W depend=opts (job dependencies, including "on" and "before" options)
-W umask=# (set job's umask)
-- Added qalter and qrerun commands to torque package.
-- Corrections to qstat logic: job CPU count and partition time format.
-- Add job_submit/pbs plugin to translate PBS job dependency options to the
extend possible (no support for PBS "before" options) and set some PBS
environment variables.
-- Add spank/pbs plugin to set a bunch of PBS environment variables.
-- Backported sh5util from master to 2.6 as there are some important
bugfixes and the new item extraction feature.
-- select/cons_res - Correct MacCPUsPerNode partition constraint for CR_Socket.
-- scontrol - for setdebugflags command, avoid parsing "-flagname" as an
scontrol command line option.
-- Fix issue with step accounting if a job is requeued.
-- Close file descriptors on exec of prolog, epilog, etc.
-- Fix issue when a user has held a job and then sets the begin time
into the future.
-- Scontrol - Enable changing a job's stdout file.
-- Fix issues where memory or node count of a srun job is altered while the
srun is pending. The step creation would use the old values and possibly
hang srun since the step wouldn't be able to be created in the modified
allocation.
-- Add support for new SchedulerParameters value of "bf_max_job_part", the
maximum depth the backfill scheduler should go in any single partition.
-- acct_gather/infiniband plugin - Correct packets_in/out values.
-- BLUEGENE - Don't ignore a conn-type request from the user.
-- BGQ - Force a request on a Q for a MESH to be a TORUS in a dimension that
can only be a TORUS (1).
-- Change max message length from 100MB to 1GB before generating "Insane
message length" error.
-- sched/backfill - Prevent possible memory corruption due to use of
bf_continue option and long running scheduling cycle (pending jobs could
have been cancelled and purged).
-- CRAY - fix AcceleratorAllocation depth correctly for basil 1.3
-- Created the environment variable SLURM_JOB_NUM_NODES for srun jobs and
updated the srun man page.
-- BLUEGENE/CRAY - Don't set env variables that pertain to a node when Slurm
isn't doing the launching.
-- gres/gpu and gres/mic - Do not treat the existence of an empty gres.conf
file as a fatal error.
-- Fixed for if hours are specified as 0 the time days-0:min specification
is not parsed correctly.
-- switch/nrt - Fix for memory leak.
-- Subtract the PMII_COMMANDLEN_SIZE in contribs/pmi2/pmi2_api.c to prevent
certain implementation of snprintf() to segfault.
* Changes in Slurm 2.6.2
========================
-- Fix issue with reconfig and GrpCPURunMins
-- Fix of wrong node/job state problem after reconfig
-- Allow users who are coordinators update their own limits in the accounts
they are coordinators over.
-- BackupController - Make sure we have a connection to the DBD first thing
to avoid it thinking we don't have a cluster name.
-- Correct value of min_nodes returned by loading job information to consider
the job's task count and maximum CPUs per node.
-- If running jobacct_gather/none fix issue on unpacking step completion.
-- Reservation with CoreCnt: Avoid possible invalid memory reference.
-- sjstat - Add man page when generating rpms.
-- Make sure GrpCPURunMins is added when creating a user, account or QOS with
sacctmgr.
-- Fix for invalid memory reference due to multiple free calls caused by
job arrays submitted to multiple partitions.
-- Enforce --ntasks-per-socket=1 job option when allocating by socket.
-- Validate permissions of key directories at slurmctld startup. Report
anything that is world writable.
-- Improve GRES support for CPU topology. Previous logic would pick CPUs then
reject jobs that can not match GRES to the allocated CPUs. New logic first
filters out CPUs that can not use the GRES, next picks CPUs for the job,
and finally picks the GRES that best match those CPUs.
-- Switch/nrt - Prevent invalid memory reference when allocating single adapter
per node of specific adapter type
-- CRAY - Make Slurm work with CLE 5.1.1
-- Fix segfault if submitting to multiple partitions and holding the job.
-- Use MAXPATHLEN instead of the hardcoded value 1024 for maximum file path
lengths.
-- If OverTimeLimit is defined do not declare failed those jobs that ended
in the OverTimeLimit interval.
* Changes in Slurm 2.6.1
========================
-- slurmdbd - Allow job derived ec and comments to be modified by non-root
users.
-- Fix issue with job name being truncated to 24 chars when sending a mail
message.
-- Fix minor issues with spec file, missing files and including files
erroneously on a bluegene system.
-- sacct - fix --name and --partition options when using
accounting_storage/filetxt.
-- squeue - Remove extra whitespace of default printout.
-- BGQ - added head ppcfloor as an include dir when building.
-- BGQ - Better debug messages in runjob_mux plugin.
-- PMI2 Updated the Makefile.am to build a versioned library.
-- CRAY - Fix srun --mem_bind=local option with launch/aprun.
-- PMI2 Corrected buffer size computation in the pmi2_api.c module.
-- GRES accounting data wrong in database: gres_alloc, gres_req, and gres_used
fields were empty if the job was not started immediately.
-- Fix sbatch and srun task count logic when --ntasks-per-node specified,
but no explicit task count.
-- Corrected the hdf5 profile user guide and the acct_gather.conf
documentation.
-- IPMI - Fix Math bug getting new wattage.
-- Corrected the AcctGatherProfileType documentation in slurm.conf
-- Corrected the sh5util program to print the header in the csv file
only once, set the debug messages at debug() level, make the argument
check case insensitive and avoid printing duplicate \n.
-- If cannot collect energy values send message to the controller
to drain the node and log error slurmd log file.
-- Handle complete removal of CPURunMins time at the end of the job instead
of at multifactor poll.
-- sview - Add missing debug_flag options.
-- PGSQL - Notes about Postgres functionality being removed in the next
version of Slurm.
-- MYSQL - fix issue when rolling up usage and events happened when a cluster
was down (slurmctld not running) during that time period.
-- sched/wiki2 - Insure that Moab gets current CPU load information.
-- Prevent infinite loop in parsing configuration if including file containing
one blank line.
-- Fix pack and unpack between 2.6 and 2.5.
-- Fix job state recovery logic in which a job's accounting frequency was
not set. This would result in a value of 65534 seconds being used (the
equivalent of NO_VAL in uint16_t), which could result in the job being
requeued or aborted.
-- Validate a job's accounting frequency at submission time rather than
waiting for it's initiation to possibly fail.
-- Fix CPURunMins if a job is requeued from a failed launch.
-- Fix in accounting_storage/filetxt to correct start times which sometimes
could end up before the job started.
-- Fix issue with potentially referencing past an array in parse_time()
-- CRAY - fix issue with accelerators on a cray when parsing BASIL 1.3 XML.
-- Fix issue with a 2.5 slurmstepd locking up when talking to a 2.6 slurmd.
-- Add argument to priority plugin's priority_p_reconfig function to note
when the association and QOS used_cpu_run_secs field has been reset.
* Changes in Slurm 2.6.0
========================
-- Fix it so bluegene and serial systems don't get warnings over new NODEDATA
enum.
-- When a job is aborted send a message for any tasks that have completed.
-- Correction to memory per CPU calculation on system with threads and
allocating cores or sockets.
-- Requeue batch job if it's node reboots (used to abort the job).
-- Enlarge maximum size of srun's hostlist file.
-- IPMI - Fix first poll to get correct consumed_energy for a step.
-- Correction to job state recovery logic that could result in assert failure.
-- Record partial step accounting record if allocated nodes fail abnormally.
-- Accounting - fix issue where PrivateData=jobs or users could potentially
show information to users that had no associations on the system.
-- Make PrivateData in slurmdbd.conf case insensitive.
-- sacct/sstat - Add format option ConsumedEnergyRaw to print full energy
values.
* Changes in Slurm 2.6.0rc2
===========================
-- HDF5 - Fix issue with Ubuntu where HDF5 development headers are
overwritten by the parallel versions thus making it so we need handle
both cases.
-- ACCT_GATHER - handle suspending correctly for polling threads.
-- Make SLURM_DISTRIBUTION env var hold both types of distribution if
specified.
-- Remove hardcoded /usr/local from slurm.spec.
-- Modify slurmctld locking to improve performance under heavy load with
very large numbers of batch job submissions or job cancellations.
-- sstat - Fix issue where if -j wasn't given allow last argument to be checked
for as the job/step id.
-- IPMI - fix adjustment on poll when using EnergyIPMICalcAdjustment.
* Changes in Slurm 2.6.0rc1
===========================
-- Added helper script for launching symmetric and MIC-only MPI tasks within
SLURM (in contribs/mic/mpirun-mic).
-- Change maximum delay for state save from 2 secs to 5 secs. Make timeout
configurable at build time by defining SAVE_MAX_WAIT.
-- Modify slurmctld data structure locking to interleave read and write
locks rather than always favor write locks over read locks.
-- Added sacct format option of "ALL" to print all fields.
-- Deprecate the SchedulerParameters value of "interval" use "bf_interval"
instead as documented.
-- Add acct_gather_profile/hdf5 to profile jobs with hdf5
-- Added MaxCPUsPerNode partition configuration parameter. This can be
especially useful to schedule systems with GPUs.
-- Permit "scontrol reboot_node" for nodes in MAINT reservation.
-- Added "PriorityFlags" value of "SMALL_RELATIVE_TO_TIME". If set, the job's
size component will be based upon not the job size alone, but the job's
size divided by it's time limit.
-- Added sbatch option "--ignore-pbs" to ignore "#PBS" options in the batch
script.
-- Rename slurm_step_ctx_params_t field from "mem_per_cpu" to "pn_min_memory".
Job step now accepts memory specification in either per-cpu or per-node
basis.
-- Add ability to specify host repitition count in the srun hostfile (e.g.
"host1*2" is equivalent to "host1,host1").
* Changes in Slurm 2.6.0pre3
============================
-- Add milliseconds to default log message header (both RFC 5424 and ISO 8601
time formats). Disable milliseconds logging using the configure
parameter "--disable-log-time-msec". Default time format changes to
ISO 8601 (without time zone information). Specify "--enable-rfc5424time"
to restore the time zone information.
-- Add username (%u) to the filename pattern in the batch script.
-- Added options for front end nodes of AllowGroups, AllowUsers, DenyGroups,
and DenyUsers.
-- Fix sched/backfill logic to initiate jobs with maximum time limit over the
partition limit, but the minimum time limit permits it to start.
-- gres/gpu - Fix for gres.conf file with multiple files on a single line
using a slurm expression (e.g. "File=/dev/nvidia[0-1]").
-- Replaced ipmi.conf with generic acct_gather.conf file for all acct_gather
plugins. For those doing development to use this follow the model set
forth in the acct_gather_energy_ipmi plugin.
-- Added more options to update a step's information
-- Add DebugFlags=ThreadID which will print the thread id of the calling
thread.
-- CRAY - Allocate whole node (CPUs) in reservation despite what the
user requests. We have found any srun/aprun afterwards will work on a
subset of resources.
* Changes in Slurm 2.6.0pre2
============================
-- Do not purge inactive interactive jobs that lack a port to ping (added
for MR+ operation).
-- Advanced reservations with hostname and core counts now supports asymetric
reservations (e.g. specific different core count for each node).
-- Added slurmctld/dynalloc plugin for MapReduce+ support.
-- Added "DynAllocPort" configuration parameter.
-- Added partition paramter of SelectTypeParameters to override system-wide
value.
-- Added cr_type to partition_info data structure.
-- Added allocated memory to node information available (within the existing
select_nodeinfo field of the node_info_t data structure). Added Allocated
Memory to node information displayed by sview and scontrol commands.
-- Make sched/backfill the default scheduling plugin rather than sched/builtin
(FIFO).
-- Added support for a job having different priorities in different partitions.
-- Added new SchedulerParameters configuration parameter of "bf_continue"
which permits the backfill scheduler to continue considering jobs for
backfill scheduling after yielding locks even if new jobs have been
submitted. This can result in lower priority jobs from being backfill
scheduled instead of newly arrived higher priority jobs, but will permit
more queued jobs to be considered for backfill scheduling.
-- Added support to purge reservation records from accounting.
-- Cray - Add support for Basil 1.3
* Changes in SLURM 2.6.0pre1
============================
-- Add "state" field to job step information reported by scontrol.
-- Notify srun to retry step creation upon completion of other job steps
rather than polling. This results in much faster throughput for job step
execution with --exclusive option.
-- Added "ResvEpilog" and "ResvProlog" configuration parameters to execute a
program at the beginning and end of each reservation.
-- Added "slurm_load_job_user" function. This is a variation of
"slurm_load_jobs", but accepts a user ID argument, potentially resulting
in substantial performance improvement for "squeue --user=ID"
-- Added "slurm_load_node_single" function. This is a variation of
"slurm_load_nodes", but accepts a node name argument, potentially resulting
in substantial performance improvement for "sinfo --nodes=NAME".
-- Added "HealthCheckNodeState" configuration parameter identify node states
on which HealthCheckProgram should be executed.
-- Remove sacct --dump --formatted-dump options which were deprecated in
2.5.
-- Added support for job arrays (phase 1 of effort). See "man sbatch" option
-a/--array for details.
-- Add new AccountStorageEnforce options of 'nojobs' and 'nosteps' which will
allow the use of accounting features like associations, qos and limits but
not keep track of jobs or steps in accounting.
-- Cray - Add new cray.conf parameter of "AlpsEngine" to specify the
communication protocol to be used for ALPS/BASIL.
-- select/cons_res plugin: Correction to CPU allocation count logic in for
cores without hyperthreading.
-- Added new SelectTypeParameter value of "CR_ALLOCATE_FULL_SOCKET".
-- Added PriorityFlags value of "TICKET_BASED" and merged priority/multifactor2
plugin into priority/multifactor plugin.
-- Add "KeepAliveTime" configuration parameter controlling how long sockets
used for srun/slurmstepd communications are kept alive after disconnect.
-- Added SLURM_SUBMIT_HOST to salloc, sbatch and srun job environment.
-- Added SLURM_ARRAY_TASK_ID to environment of job array.
-- Added squeue --array/-r option to optimize output for job arrays.
-- Added "SlurmctldPlugstack" configuration parameter for generic stack of
slurmctld daemon plugins.
-- Removed contribs/arrayrun tool. Use native support for job arrays.
-- Modify default installation locations for RPMs to match "make install":
_prefix /usr/local
_slurm_sysconfdir %{_prefix}/etc/slurm
_mandir %{_prefix}/share/man
_infodir %{_prefix}/share/info
-- Add acct_gather_energy/ipmi which works off freeipmi for energy gathering
* Changes in Slurm 2.5.8
========================
-- Fix for slurmctld segfault on NULL front-end reason field.
-- Avoid gres step allocation errors when a job shrinks in size due to either
down nodes or explicit resizing. Generated slurmctld errors of this type:
"step_test ... gres_bit_alloc is NULL"
-- Fix bug that would leak memory and over-write the AllowGroups field if on
"scontrol reconfig" when AllowNodes is manually changed using scontrol.
-- Get html/man files to install in correct places with rpms.
-- Remove --program-prefix from spec file since it appears to be added by
default and appeared to break other things.
-- Updated the automake min version in autogen.sh to be correct.
-- Select/cons_res - Correct total CPU count allocated to a job with
--exclusive and --cpus-per-task options
-- switch/nrt - Don't allocate network resources unless job step has 2+ nodes.
-- select/cons_res - Avoid extraneous "oversubscribe" error messages.
-- Reorder get config logic to avoid deadlock.
-- Enforce QOS MaxCPUsMin limit when job submission contains no user-specified
time limit.
-- EpilogSlurmctld pthread is passed required arguments rather than a pointer
to the job record, which under some conditions could be purged and result
in an invalid memory reference.
* Changes in Slurm 2.5.7
========================
-- Fix for linking to the select/cray plugin to not give warning about
undefined variable.
-- Add missing symbols to the xlator.h
-- Avoid placing pending jobs in AdminHold state due to backfill scheduler
interactions with advanced reservation.
-- Accounting - make average by task not cpu.
-- CRAY - Change logging of transient ALPS errors from error() to debug().
-- POE - Correct logic to support poe option "-euidevice sn_all" and
"-euidevice sn_single".
-- Accounting - Fix minor initialization error.
-- POE - Correct logic to support srun network instances count with POE.
-- POE - With the srun --launch-cmd option, report proper task count when
the --cpus-per-task option is used without the --ntasks option.
-- POE - Fix logic binding tasks to CPUs.
-- sview - Fix race condition where new information could of slipped past
the node tab and we didn't notice.
-- Accounting - Fix an invalid memory read when slurmctld sends data about
start job to slurmdbd.
-- If a prolog or epilog failure occurs, drain the node rather than setting it
down and killing all of its jobs.
-- Priority/multifactor - Avoid underflow in half-life calculation.
-- POE - pack missing variable to allow fanout (more than 32 nodes)
-- Prevent clearing reason field for pending jobs. This bug was introduced in
v2.5.5 (see "Reject job at submit time ...").
-- BGQ - Fix issue with preemption on sub-block jobs where a job would kill
all preemptable jobs on the midplane instead of just the ones it needed to.
-- switch/nrt - Validate dynamic window allocation size.
-- BGQ - When --geo is requested do not impose the default conn_types.
-- CRAY - Support CLE 4.2.0
-- RebootNode logic - Defers (rather than forgets) reboot request with job
running on the node within a reservation.
-- switch/nrt - Correct network_id use logic. Correct support for user sn_all
and sn_single options.
-- sched/backfill - Modify logic to reduce overhead under heavy load.
-- Fix job step allocation with --exclusive and --hostlist option.
-- Select/cons_res - Fix bug resulting in error of "cons_res: sync loop not
progressing, holding job #"
-- checkpoint/blcr - Reset max_nodes from zero to NO_VAL on job restart.
-- launch/poe - Fix for hostlist file support with repeated host names.
-- priority/multifactor2 - Prevent possible divide by zero.
-- srun - Don't check for executable if --test-only flag is used.
-- energy - On a single node only use the last task for gathering energy.
Since we don't currently track energy usage per task (only per step).
Otherwise we get double the energy.
* Changes in Slurm 2.5.6
========================
-- Gres fix for requeued jobs.
-- Gres accounting - Fix regression in 2.5.5 for keeping track of gres
requested and allocated.
* Changes in Slurm 2.5.5
========================
-- Fix for sacctmgr add qos to handle the 'flags' option.
-- Export SLURM_ environment variables from sbatch, even if "--export"
option does not explicitly list them.
-- If node is in more than one partition, correct counting of allocated CPUs.
-- If step requests more CPUs than possible in specified node count of job
allocation then return ESLURM_TOO_MANY_REQUESTED_CPUS rather than
ESLURM_NODES_BUSY and retrying.
-- CRAY - Fix SLURM_TASKS_PER_NODE to be set correctly.
-- Accounting - more checks for strings with a possible `'` in it.
-- sreport - Fix by adding planned down time to utilization reports.
-- Do not report an error when sstat identifies job steps terminated during
its execution, but log using debug type message.
-- Select/cons_res - Permit node removed from job by going down to be returned
to service and re-used by another job.
-- Select/cons_res - Tighter packing of job allocations on sockets.
-- SlurmDBD - fix to allow user root along with the slurm user to register a
cluster.
-- Select/cons_res - Fix for support of consecutive node option.
-- Select/cray - Modify build to enable direct use of libslurm library.
-- Bug fixes related to job step allocation logic.
-- Cray - Disable enforcement of MaxTasksPerNode, which is not applicable
with launch/aprun.
-- Accounting - When rolling up data from past usage ignore "idle" time from
a reservation when it has the "Ignore_Jobs" flag set. Since jobs could run
outside of the reservation in it's nodes without this you could have
double time.
-- Accounting - Minor fix to avoid reuse of variable erroneously.
-- Reject job at submit time if the node count is invalid. Previously such a
job submitted to a DOWN partition would be queued.
-- Purge vestigial job scripts when the slurmd cold starts or slurmstepd
terminates abnormally.
-- Add support for FreeBSD.
-- Add sanity check for NULL cluster names trying to register.
-- BGQ - Push action 'D' info to scontrol for admins.
-- Reset a job's reason from PartitionDown when the partition is set up.
-- BGQ - Handle issue where blocks would have a pending job on them and
while it was free cnodes would go into software error and kill the job.
-- BGQ - Fix issue where if for some reason we are freeing a block with
a pending job on it we don't kill the job.
-- BGQ - Fix race condition were a job could of been removed from a block
without it still existing there. This is extremely rare.
-- BGQ - Fix for when a step completes in Slurm before the runjob_mux notifies
the slurmctld there were software errors on some nodes.
-- BGQ - Fix issue on state recover if block states are not around
and when reading in state from DB2 we find a block that can't be created.
You can now do a clean start to rid the bad block.
-- Modify slurmdbd to retransmit to slurmctld daemon if it is not responding.
-- BLUEGENE - Fix issue where when doing backfill preemptable jobs were
never looked at to determine eligibility of backfillable job.
-- Cray/BlueGene - Disable srun --pty option unless LaunchType=launch/slurm.
-- CRAY - Fix sanity check for systems with more than 32 cores per node.
-- CRAY - Remove other objects from MySQL query that are available from
the XML.
-- BLUEGENE - Set the geometry of a job when a block is picked and the job
isn't a sub-block job.
-- Cray - avoid check of macro versions of CLE for version 5.0.
-- CRAY - Fix memory issue with reading in the cray.conf file.
-- CRAY - If hostlist is given with srun make sure the node count is the same
as the hosts given.
-- CRAY - If task count specified, but no tasks-per-node, then set the tasks
per node in the BASIL reservation request.
-- CRAY - fix issue with --mem option not giving correct amount of memory
per cpu.
-- CRAY - Fix if srun --mem is given outside an allocation to set the
APRUN_DEFAULT_MEMORY env var for aprun. This scenario will not display
the option when used with --launch-cmd.
-- Change sview to use GMutex instead of GStaticMutex
-- CRAY - set APRUN_DEFAULT_MEMROY instead of CRAY_AUTO_APRUN_OPTIONS
-- sview - fix issue where if a partition was completely in one state the
cpu count would be reflected correctly.
-- BGQ - fix for handling half rack system in STATIC of OVERLAP mode to
implicitly create full system block.
-- CRAY - Dynamically create BASIL XML buffer to resize as needed.
-- Fix checking if QOS limit MaxCPUMinsPJ is set along with DenyOnLimit to
deny the job instead of holding it.
-- Make sure on systems that use a different launcher than launch/slurm not
to attempt to signal tasks on the frontend node.
-- Cray - when a step is requested count other steps running on nodes in the
allocation as taking up the entire node instead of just part of the node
allocated. And always enforce exclusive on a step request.
-- Cray - display correct nodelist, node/cpu count on steps.
* Changes in Slurm 2.5.4
========================
-- Fix bug in PrologSlurmctld use that would block job steps until node
responds.
-- CRAY - If a partition has MinNodes=0 and a batch job doesn't request nodes
put the allocation to 1 instead of 0 which prevents the allocation to
happen.
-- Better debug when the database is down and using the --cluster option in
the user commands.
-- When asking for job states with sacct, default to 'now' instead of midnight
of the current day.
-- Fix for handling a test-only job or immediate job that fails while being
built.
-- Comment out all of the logic in the job_submit/defaults plugin. The logic
is only an example and not meant for actual use.
-- Eliminate configuration file 4096 character line limitation.
-- More robust logic for tree message forward
-- BGQ - When cnodes fail in a timeout fashion correctly look up parent
midplane.
-- Correct sinfo "%c" (node's CPU count) output value for Bluegene systems.
-- Backfill - Responsive improvements for systems with large numbers of jobs
(>5000) and using the SchedulerParameters option bf_max_job_user.
-- slurmstepd: ensure that IO redirection openings from/to files correctly
handle interruption
-- BGQ - Able to handle when midplanes go into Hardware::SoftwareFailure
-- GRES - Correct tracking of specific resources used after slurmctld restart.
Counts would previously go negative as jobs terminate and decrement from
a base value of zero.
-- Fix for priority/multifactor2 plugin to not assert when configured with
--enable-debug.
-- Select/cons_res - If the job request specified --ntasks-per-socket and the
allocation using is cores, then pack the tasks onto the sockets up to the
specified value.
-- BGQ - If a cnode goes into an 'error' state and the block containing the
cnode does not have a job running on it do not resume the block.
-- BGQ - Handle blocks that don't free themselves in a reasonable time better.
-- BGQ - Fix for signaling steps when allocation ends before step.
-- Fix for backfill scheduling logic with job preemption; starts more jobs.
-- xcgroup - remove bugs with EINTR management in write calls
-- jobacct_gather - fix total values to not always == the max values.
-- Fix for handling node registration messages from older versions without
energy data.
-- BGQ - Allow user to request full dimensional mesh.
-- sdiag command - Correction to jobs started value reported.
-- Prevent slurmctld assert when invalid change to reservation with running
jobs is made.
-- BGQ - If signal is NODE_FAIL allow forward even if job is completing
and timeout in the runjob_mux trying to send in this situation.
-- BGQ - More robust checking for correct node, task, and ntasks-per-node
options in srun, and push that logic to salloc and sbatch.
-- GRES topology bug in core selection logic fixed.
-- Fix to handle init.d script for querying status and not return 1 on
success.
* Changes in SLURM 2.5.3
========================
-- Gres/gpu plugin - If no GPUs requested, set CUDA_VISIBLE_DEVICES=NoDevFiles.
This bug was introduced in 2.5.2 for the case where a GPU count was
configured, but without device files.
-- task/affinity plugin - Fix bug in CPU masks for some processors.
-- Modify sacct command to get format from SACCT_FORMAT environment variable.
-- BGQ - Changed order of library inclusions and fixed incorrect declaration
to compile correctly on newer compilers
-- Fix for not building sview if glib exists on a system but not the gtk libs.
-- BGQ - Fix for handling a job cleanup on a small block if the job has long
since left the system.
-- Fix race condition in job dependency logic which can result in invalid
memory reference.
* Changes in SLURM 2.5.2
========================
-- Fix advanced reservation recovery logic when upgrading from version 2.4.
-- BLUEGENE - fix for QOS/Association node limits.
-- Add missing "safe" flag from print of AccountStorageEnforce option.
-- Fix logic to optimize GRES topology with respect to allocated CPUs.
-- Add job_submit/all_partitions plugin to set a job's default partition
to ALL available partitions in the cluster.
-- Modify switch/nrt logic to permit build without libnrt.so library.
-- Handle srun task launch failure without duplicate error messages or abort.
-- Fix bug in QoS limits enforcement when slurmctld restarts and user not yet
added to the QOS list.
-- Fix issue where sjstat and sjobexitmod was installed in 2 different RPMs.
-- Fix for job request of multiple partitions in which some partitions lack
nodes with required features.
-- Permit a job to use a QOS they do not have access to if an administrator
manually set the job's QOS (previously the job would be rejected).
-- Make more variables available to job_submit/lua plugin: slurm.MEM_PER_CPU,
slurm.NO_VAL, etc.
-- Fix topology/tree logic when nodes defined in slurm.conf get re-ordered.
-- In select/cons_res, correct logic to allocate whole sockets to jobs. Work
by Magnus Jonsson, Umea University.
-- In select/cons_res, correct logic when job removed from only some nodes.
-- Avoid apparent kernel bug in 2.6.32 which apparently is solved in
at least 3.5.0. This avoids a stack overflow when running jobs on
more than 120k nodes.
-- BLUEGENE - If we made a block that isn't runnable because of a overlapping
block, destroy it correctly.
-- Switch/nrt - Dynamically load libnrt.so from within the plugin as needed.
This eliminates the need for libnrt.so on the head node.
-- BLUEGENE - Fix in reservation logic that could cause abort.
* Changes in SLURM 2.5.1
========================
-- Correction to hostlist sorting for hostnames that contain two numeric
components and the first numeric component has various sizes (e.g.
"rack9blade1" should come before "rack10blade1")
-- BGQ - Only poll on initialized blocks instead of calling getBlocks on
each block independently.
-- Fix of task/affinity plugin logic for Power7 processors having hyper-
threading disabled (cpu mask has gaps).
-- Fix of job priority ordering with sched/builtin and priority/multifactor.
Patch from Chris Read.
-- CRAY - Fix for setting up the aprun for a large job (+2000 nodes).
-- Fix for race condition related to compute node boot resulting in node being
set down with reason of "Node <name> unexpectedly rebooted"
-- RAPL - Fix for handling errors when opening msr files.
-- BGQ - Fix for salloc/sbatch to do the correct allocation when asking for
-N1 -n#.
-- BGQ - in emulation make it so we can pretend to run large jobs (>64k nodes)
-- BLUEGENE - Correct method to update conn_type of a job.
-- BLUEGENE - Fix issue with preemption when needing to preempt multiple jobs
to make one job run.
-- Fixed issue where if an srun dies inside of an allocation abnormally it
would of also killed the allocation.
-- FRONTEND - fixed issue where if a systems nodes weren't defined in the
slurm.conf with NodeAddr's signals going to a step could be handled
incorrectly.
-- If sched/backfill starts a job with a QOS having NO_RESERVE and not job
time limit, start it with the partition time limit (or one year if the
partition has no time limit) rather than NO_VAL (140 year time limit);
-- Alter hostlist logic to allocate large grid dynamically instead of on
stack.
-- Change RPC version checks to support version 2.5 slurmctld with version 2.4
slurmd daemons.
-- Correct core reservation logic for use with select/serial plugin.
-- Exit scontrol command on stdin EOF.
-- Disable job --exclusive option with select/serial plugin.
* Changes in SLURM 2.5.0
========================
-- Add DenyOnLimit flag for QOS to deny jobs at submission time if they
request resources that reach a 'Max' limit.
-- Permit SlurmUser or operator to change QOS of non-pending jobs (e.g.
running jobs).
-- BGQ - move initial poll to beginning of realtime interaction, which will
also cause it to run if the realtime server ever goes away.
* Changes in SLURM 2.5.0-rc2
============================
-- Modify sbcast logic to survive slurmd daemon restart while file a
transmission is in progress.
-- Add retry logic to munge encode/decode calls. This is needed if the munge
deamon is under very heavy load (e.g. with 1000 slurmd daemons per compute
node).
-- Add launch and acct_gather_energy plugins to RPMs.
-- Restore support for srun "--mpi=list" option.
-- CRAY - Introduce step accounting for a Cray.
-- Modify srun to abandon I/O 60 seconds after the last task ends. Otherwise
an aborted slurmstepd can cause the srun process to hang indefinitely.
-- ENERGY - RAPL - alter code to close open files (and only open them once
where needed)
-- If the PrologSlurmctld fails, then requeue the job an indefinite number
of times instead of only one time.
* Changes in SLURM 2.5.0-rc1
============================
-- Added Prolog and Epilog Guide (web page). Based upon work by Jason Sollom,
Cray Inc. and used by permission.
-- Restore gang scheduling functionality. Preemptor was not being scheduled.
Fix for bugzilla #3.
-- Add "cpu_load" to node information. Populate CPULOAD in node information
reported to Moab cluster manager.
-- Preempt jobs only when insufficient idle resources exist to start job,
regardless of the node weight.
-- Added priority/multifactor2 plugin based upon ticket distribution system.
Work by Janne Blomqvist, Aalto University.
-- Add SLURM_NODELIST to environment variables available to Prolog and Epilog.
-- Permit reservations to allow or deny access by account and/or user.
-- Add ReconfigFlags value of KeepPartState. See "man slurm.conf" for details.
-- Modify the task/cgroup plugin adding a task_pre_launch_priv function and
move slurmstepd outside of the step's cgroup. Work by Matthieu Hautreux.
-- Intel MIC processor support added using gres/mic plugin. BIG thanks to
Olli-Pekka Lehto, CSC-IT Center for Science Ltd.
-- Accounting - Change empty jobacctinfo structs to not actually be used
instead of putting 0's into the database we put NO_VALS and have sacct
figure out jobacct_gather wasn't used.
-- Cray - Prevent calling basil_confirm more than once per job using a flag.
-- Fix bug with topology/tree and job with min-max node count. Now try to
get max node count rather than minimizing leaf switches used.
-- Add AccountingStorageEnforce=safe option to provide method to avoid jobs
launching that wouldn't be able to run to completion because of a
GrpCPUMins limit.
-- Add support for RFC 5424 timestamps in logfiles. Disable with configuration
option of "--disable-rfc5424time". By Janne Blomqvist, Aalto University.
-- CRAY - Replace srun.pl with launch/aprun plugin to use srun to wrap the
aprun process instead of a perl script.
-- srun - Rename --runjob-opts to --launcher-opts to be used on systems other
than BGQ.
-- Added new DebugFlags - Energy for AcctGatherEnergy plugins.
-- start deprecation of sacct --dump --fdump
-- BGQ - added --verbose=OFF when srun --quiet is used
-- Added acct_gather_energy/rapl plugin to record power consumption by job.
Work by Yiannis Georgiou, Martin Perry, et. al., Bull
* Changes in SLURM 2.5.0.pre3
=============================
-- Add Google search to all web pages.
-- Add sinfo -T option to print reservation information. Work by Bill Brophy,
Bull.
-- Force slurmd exit after 2 minute wait, even if threads are hung.
-- Change node_req field in struct job_resources from 8 to 32 bits so we can
run more than 256 jobs per node.
-- sched/backfill: Improve accuracy of expected job start with respect to
reservations.
-- sinfo partition field size will be set the the length of the longest
partition name by default.
-- Make it so the parse_time will return a valid 0 if given epoch time and
set errno == ESLURM_INVALID_TIME_VALUE on error instead.
-- Correct srun --no-alloc logic when node count exceeds node list or task
task count is not a multiple of the node count. Work by Hongjia Cao, NUDT.
-- Completed integration with IBM Parallel Environment including POE and IBM's
NRT switch library.
* Changes in SLURM 2.5.0.pre2
=============================
-- When running with multiple slurmd daemons per node, enable specifying a
range of ports on a single line of the node configuration in slurm.conf.
-- Add reservation flag of Part_Nodes to allocate all nodes in a partition to
a reservation and automatically change the reservation when nodes are
added to or removed from the reservation. Based upon work by
Bill Brophy, Bull.
-- Add support for advanced reservation for specific cores rather than whole
nodes. Current limiations: homogeneous cluster, nodes idle when reservation
created, and no more than one reservation per node. Code is still under
development. Work by Alejandro Lucero Palau, et. al, BSC.
-- Add DebugFlag of Switch to log switch plugin details.
-- Correct job node_cnt value in job completion plugin when job fails due to
down node. Previously was too low by one.
-- Add new srun option --cpu-freq to enable user control over the job's CPU
frequency and thus it's power consumption. NOTE: cpu frequency is not
currently preserved for jobs being suspended and later resumed. Work by
Don Albert, Bull.
-- Add node configuration information about "boards" and optimize task
placement on minimum number of boards. Work by Rod Schultz, Bull.
* Changes in SLURM 2.5.0.pre1
=============================
-- Add new output to "scontrol show configuration" of LicensesUsed. Output is
"name:used/total"
-- Changed jobacct_gather plugin infrastructure to be cleaner and easier to
maintain.
-- Change license option count separator from "*" to ":" for consistency with
the gres option (e.g. "--licenses=foo:2 --gres=gpu:2"). The "*" will still
be accepted, but is no longer documented.
-- Permit more than 100 jobs to be scheduled per node (new limit is 250
jobs).
-- Restructure of srun code to allow outside programs to utilize existing
logic.
* Changes in SLURM 2.4.6
========================
-- Correct WillRun authentication logic when issued for non-job owner.
-- BGQ - fix memory leak
-- BGQ - Fix to check block for action 'D' if it also has nodes in error.
* Changes in SLURM 2.4.5
========================
-- Cray - On job kill requeust, send SIGCONT, SIGTERM, wait KillWait and send
SIGKILL. Previously just sent SIGKILL to tasks.
-- BGQ - Fix issue when running srun outside of an allocation and only
specifying the number of tasks and not the number of nodes.
-- BGQ - validate correct ntasks_per_node
-- BGQ - when srun -Q is given make runjob be quiet
-- Modify use of OOM (out of memory protection) for Linux 2.6.36 kernel
or later. NOTE: If you were setting the environment variable
SLURMSTEPD_OOM_ADJ=-17, it should be set to -1000 for Linux 2.6.36 kernel
or later.
-- BGQ - Fix job step timeout actually happen when done from within an
allocation.
-- Reset node MAINT state flag when a reservation's nodes or flags change.
-- Accounting - Fix issue where QOS usage was being zeroed out on a
slurmctld restart.
-- BGQ - Add 64 tasks per node as a valid option for srun when used with
overcommit.
-- BLUEGENE - With Dynamic layout mode - Fix issue where if a larger block