forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
2871 lines (2134 loc) · 108 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v7r0-pre19]
FIX: removed __init__ files from test directories to avoid importing
NEW: Change the schema of the Resources description in CS, use new Resources helper to
interpret this information
NEW: Base RSS resources representation on the new CS Resources schema
NEW: RSS Pilots Efficiency Policy
NEW: Added Licence statement available in all the commands through --license switch
*Core
FIX: multiple fixes to remove pylint complaints
CHANGE: Moved log coloring to utilities so it can be used consistently across DIRAC
FIX: Workflow package - fix wild wild card imports resulting in circular imports
NEW: MySQL - assign one MySQL connection per thread
FIX: LocalConfiguration - if short option is not set, do not print "-" in help message,
closes #1371
FIX: dirac-install - update creation of bashrc/cshrc to allow multiplatform installations
NEW: InstallTools - check inheritance of modules to be loaded to determine their types;
executors can now be handled by dirac-setup-site
FIX: LocalConfiguration - allow '-' character in long script options, e.g. "--long-script-option"
CHANGE: Removed "import *" from DIRAC.__init__
NEW: dirac-install allows projects to define mirrors from where to download other projects
(DIRAC mirrors for instance)
FIX: Removed deprecation warning when importing MySQLdb package
FIX: BaseClient - take into account DISET decorator
CHANGE: MySQL - added okIfTableExists flag to the _createTables() method, in case of OK
returns a list of created tables
*Accounting
FIX: AccountingDB - align properly days with MySQL bucketing. Closes #1219
FIX: DataStoreClient - make a deep copy of the records' values
NEW: Accounting selectors will allow registered users to choose from all possible values
except from users and groups
FIX: AccountingDB - randomize the order type insertion in order to avoid type starvation
NEW: AccountingDB - add a monitoring record for each type in IN tables
*Framework
FIX: ProxyDB - prevent duplicate key errors on writing VOMSProxies to DB. Closes #1228
BUGFIX: NotificationDB - missing "," in SQL, closes #1373
FIX: dirac-proxy-get-uploaded-info.py - typo Value -> Message
*Configuration
NEW: Resources helper class to work with the new /Resources structure according to RFC #5
NEW: dirac-configuration-convert-resources-schema - command to convert old /Resources schema
to the new one
*Interfaces
CHANGE: Job.py - setPlatform renamed to setSubmitPools
*DMS
CHANGE: MigrationMonitoringAgent - removed obsoleted
NEW: FileCatalogClient - getFileUserMetadata to get both file and directory metadata for a given file
CHANGE: FileCatalogClient - list meta data sets
FIX: FileCatalog - extra check while new user registration for already existing entry
FIX: FileManagerBase - clean the FC_FileAncestors table when removing a file
BUGFIX: ReplicaManager - pass catalog argument in putAndRegister to registerFile call, closes #1369
FIX: SRM2Storage - in getCurrentStatus() use pythonCall to impose a timeout on top of
lcg_util.lcg_stmd call
NEW: IRODSStorageElement.py - first import
NEW: DFC - use ObjectLoader to instantiate catalog component plug-ins
NEW: DFC - createTables according to the in-class schema definitions
NEW: FileCatalogClientCLI - added -q (quite) option to the find command
*WMS
CHANGE: JobScheduling - is now extensible. Added unit test
NEW: SiteDirector - define which extensions pilot should install in the options, do not take from globals
FIX: SandboxStoreClient - change default from False to None ( to suite pylint )
NEW: JobAgent - send Accounting record for Jobs Rescheduled
CHANGE: JobDB, JobLoggingDB and PilotAgentsDB - use python table description
FIX: Fixed JobDB.JobJDLs BLOB columns ( they had defaults, but BLOB does not accept default values )
CHANGE: JobLoggingDB schema - StatusTimeOrder DOUBLE(12,3) instead of (11,3)
CHANGE: JobLoggingDB schema - a new field has been added, SeqNum (INTEGER), which is part of the
new PK (JobID, SeqNum). SeqNum is generated by a trigger at every insert and behave as a counter
within a given JobID
*Transformation
NEW: TaskManager - if a site is specified in the job definition, it is now taken into account
when submitting the task
NEW: Speeding up the getTransformationSummary call, using an UpdateTransformationCounters agent
FIX: Multiple fixes to please pylint
FIX: TransformationDB - removed bad index on TaskID
NEW: TransformationDB - added methods to DELETE rows from TransformationFIleTasks table,
minor changes in operation order to be ready for FK.
CHANGE: field definition for TaskID in TransformationFiles table (from
VARCHAR to INT, in order to be ready for FK definition: TaskID in
TransformationTasks and TransformationFileTasks is INT),
!!!!! needs update of the MySQL schema on already installed databases
CHANGE: TransformationDB - in DataFiles table removed LFN field from the Primary Key,
tt was already UNIQUE, Primary key is FileID only now.
*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'
*Resources
FIX: ComputingElement - properly check if the pilot proxy has VOMS before adding it to the
payload when updating it
NEW: StorageElement - enable Storage Element proxy configuration by protocol name
CHANGE: glexecComputingElement - updated to reflect new glexec evolution, added DIRAC_GLEXEC
to the environment when running under glexec
NEW: XXXStorage - added getCurrentStatus() method
NEW: SSHOSGComputingElement and condorgce script
*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs,
dirac-stager-monitor-requests, dirac-stager-show-stats
*SMS
NEW: StorageManagementDB - use python table description
[v6r10-pre9]
*Core
FIX: Several fixes on DB classes(AccountingDB, SystemLoggingDB, UserProfileDB, TransformationDB,
JobDB, PilotAgentsDB) after the new movement to the new MySQL implementation with a persistent
connection per running thread
NEW: SystemAdministratorCLI - better support for executing remote commands
FIX: DIRAC.__init__.py - avoid re-definition of platform variable
NEW: Graphs - added CurveGraph class to draw non-stacked lines with markers
NEW: Graphs - allow graphs with negative Y values
NEW: Graphs - allow to provide errors with the data and display them in the CurveGraph
*Framework
FIX: SystemAdministratorIntegrator - use Host option to get the host address in addition to the section name, closes #1628
*DMS
CHANGE: DFC - optimization and bug fixes of the bulk file addition
FIX: TransferAgent - protection against badly defined LFNs in collectFiles()
NEW: DFC - added getDirectoryReplicas() service method support similar to the LFC
CHANGE: DFC - added new option VisibleReplicaStatus which is used in replica getting commands
CHANGE: FileCatalogClientCLI client shows number of replicas in the 2nd column rather than
unimplemented number of links
CHANGE: DFC - optimizations for the bulk replica look-up
CHANGE: DFC updated scalability testing tool FC_Scaling_test.py
NEW: DFC - methods returning replicas provide also SE definitions instead of PFNs to construct PFNs on the client side
NEW: DFC - added getReplicasByMetadata() interface
CHANGE: DFC - optimized getDirectoryReplicas()
CHANGE: FileCatalogClient - treat the reduced output from various service queries restoring LFNs and PFNs on the fly
NEW: DFC - LFNPFNConvention flag can be None, Weak or Strong to facilitate compatibility with LFC data
CHANGE: FileCatalog - do not return PFNs, construct them on the client side
CHANGE: FileCatalog - simplified FC_Scaling_test.py script
NEW: FileCatalog/DatasetManager class to define and manipulate datasets corresponding to meta queries
NEW: FileCatalogHandler - new interface methods to expose DatasetManager functionality
NEW: FileCatalogClientCLI - new dataset family of commands
FIX: StorageFactory, ReplicaManager - resolve SE alias name recursively
FIX: FTSRequest, ReplicaManager, SRM2Storage - use current proxy owner as user name in accounting reports, closes #1602
BUGFIX: FileCatalogClientCLI - bug fix in do_ls, missing argument to addFile() call, closes #1658
NEW: FileCatalog - added new setMetadataBulk() interface, closes #1358
FIX: FileCatalog - initial argument check strips off leading lfn:, LFN:, /grid, closes #448
NEW: FileCatalog - added new setFileStatus() interface, closes #170, valid and visible file and replica statuses can be defined in respective options.
CHANGE: multiple new FTS system fixes
CHANGE: uniform argument checking with checkArgumentFormat() in multiple modules
*WMS
CHANGE: JobPolicy - optimization for bulk job verification
NEW: JobPolicy - added getControlledUsers() to get users which jobs can be accessed for
a given operation
CHANGE: JobMonitoringHandler - Avoid doing a selection of all Jobs, first count matching jobs
and then use "limit" to select only the required JobIDs.
NEW: JobMonitoringHandler - use JobPolicy to filter jobs in getJobSummaryWeb()
NEW: new Operations option /Services/JobMonitoring/GlobalJobsInfo ( True by default ) to
allow or not job info lookup by anybody, used in JobMonitoringHandler
BUGFIX: SiteDirector - take into account the target queue Platform
BUGFIX: JobDB - bug in __insertNewJDL()
*RSS
FIX: DowntimeCommand - take the latest Downtime that fits
*Resources
NEW: added SSHOARComputingElement class
*TS
CHANGE: TransformationClient(DB,Manager) - set file status for transformation as bulk operation
CHANGE: TransformationClient - applying state machine when changing transformation status
BUGFIX: TransformationClient(Handler) - few minor fixes
NEW: TransformationDB - backported __deleteTransformationFileTask(s) methods
*RMS
NEW: Migrate to use the new Request Management by all the clients
CHANGE: RequestContainer - Retry failed transfers 10 times and avoid sub-requests to be set Done
when the files are failed
CHANGE: Use a unique name for storing the proxy as processes may use the same "random" name and
give conflicts
NEW: RequestClient(Handler) - add new method readRequest( requestname)
*Workflow
NEW: Porting the LHCb Workflow package to DIRAC to make the use of general purpose modules and
simplify construction of workflows
[v6r9p21]
*TS
FIX: Transformation(Client,DB,Manager) - restored FileCatalog compliant interface
FIX: TransformationDB - fix in __insertIntoExistingTransformationFiles()
[v6r9p20]
*Core
BUGFIX: ProxyUpload - an on the fly upload does not require a proxy to exist
*DMS
CHANGE: TransferAgent - use compareAdler() for checking checksum
FIX: FailoverTransfer - recording the sourceSE in case of failover transfer request
*WMS
FIX: ProcessMonitor - some fixes added, printout when <1 s of consumed CPU is found
*Transformation
BUGFIX: TransformationClient - fixed return value in moveFilesToDerivedTransformation()
*RMS
BUGFIX: CleanReqDBAgent - now() -> utcnow() in initialize()
*Resources
FIX: ARCComputingElement - fix the parsing of CE status if no jobs are available
[v6r9p19]
*DMS
FIX: FileCatalog/DirectoryMetadata - inherited metadata is used while selecting directories
in findDirIDsByMetadata()
[v6r9p18]
*DMS
FIX: FTSSubmitAgent, FTSRequest - fixes the staging mechanism in the FTS transfer submission
NEW: TransferDBMonitoringHandler - added getFilesForChannel(), resetFileChannelStatus()
[v6r9p17]
*Accounting
FIX: DataStoreClient - send accounting records in batches of 1000 records instead of 100
*DMS:
FIX: FailoverTransfer - catalog name from list to string
FIX: FTSSubmitAgent, FTSRequest - handle FTS3 as new protocol and fix bad submission time
FIX: FTSSubmitAgent, FTSRequest - do not submit FTS transfers for staging files
*WMS
FIX: TaskQueueDB - do not check enabled when TQs are requested from Directors
FIX: TaskQueueDB - check for Enabled in the TaskQueues when inserting jobs to print an alert
NEW: TaskQueueDB - each TQ can have at most 5k jobs, if beyond the limit create a new TQ
to prevent long matching times when there are way too many jobs in a single TQ
[v6r9p16]
*TS
BUGFIX: typos in TransformationCleaningAgent.py
*DMS
CHANGE: DownloadInputData - check the available disk space in the right input data directory
FIX: DownloadInputData - try to download only Cached replicas
[v6r9p15]
*Core
FIX: MySQL - do not decrease the retry counter after ping failure
*DMS
CHANGE: FC/DirectoryMetadata - Speed up findFilesByMetadataWeb when many files match
FIX: RemovalTask - fix error string when removing a non existing file (was incompatible
with the LHCb BK client).
*WMS
FIX: JobReport - minor fix ( removed unused imports )
FIX: JobMonitoring(JobStateUpdate)Handler - jobID argument can be either string, int or long
*TS
CHANGE: TransformationClient - change status of Moved files to a deterministic value
FIX: FileReport - minor fix ( inherits object )
[v6r9p14]
*DMS
CHANGE: FTSDB - changed schema: removing FTSSite table. From now on FTS sites
would be read from CS Resources
[v6r9p13]
FIX: included fixes from v6r8p26 patch release
[v6r9p12]
FIX: included fixes from v6r8p25 patch release
[v6r9p11]
*DMS
BUGFIX: FTSRequest - in __resolveFTSServer() type "=" -> "=="
[v6r9p10]
FIX: included fixes from v6r8p24 patch release
*Core
NEW: StateMachine utility
*DMS
BUGFIX: in RegisterFile operation handler
*Interfaces
FIX: Dirac.py - in splitInputData() consider only Active replicas
[v6r9p9]
*RMS
FIX: RequestDB - added getRequestFileStatus(), getRequestName() methods
[v6r9p8]
*DMS
FIX: RequestDB - get correct digest ( short request description ) of a request
[v6r9p7]
FIX: included fixes from v6r8p23 patch release
*RSS
FIX: SpaceTokenOccupancyPolicy - SpaceToken Policy decision was based on
percentage by mistake
*RMS
NEW: new scripts dirac-dms-ftsdb-summary, dirac-dms-show-ftsjobs
FIX: FTSAgent - setting space tokens for newly created FTSJobs
[v6r9p6]
*DMS
BUGFIX: dirac-admin-add-ftssite - missing import
*RMS
NEW: RequestDB, ReqManagerHandler - added getRequestStatus() method
*TS
FIX: fixes when using new RequestClient with the TransformationCleaningAgent
*WMS
BUGFIX: typo in SandboxStoreHandler transfer_fromClient() method
[v6r9p5]
*DMS
BUGFIX: missing proxy in service env in the FTSManager service. By default service
will use DataManager proxy refreshed every 6 hours.
*Resources
NEW: StorageElement - new checkAccess policy: split the self.checkMethods in
self.okMethods. okMethods are the methods that do not use the physical SE.
The isValid returns S_OK for all those immediately
*RSS
FIX: SpaceTokenOccupancyPolicy - Policy that now takes into account absolute values
for the space left
*TS
FIX: TransformationCleaningAgent - will look for both old and new RMS
[v6r9p4]
*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs,
dirac-stager-monitor-requests, dirac-stager-show-stats
[v6r9p3]
*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'
[v6r9p2]
*RSS
NEW: Added Component family tables and statuses
FIX: removed old & unused code
NEW: allow RSS policies match wild cards on CS
*WMS
BUGFIX: FailoverTransfer,JobWrapper - proper propagation of file metadata
[v6r9p1]
*RMS
NEW: FTSAgent - update rwAccessValidStamp,
update ftsGraphValidStamp,
new option for staging files before submission,
better log handling here and there
CHANGE: FTSJob - add staging flag in in submitFTS2
CHANGE: Changes in WMS (FailoverTransfer, JobReport, JobWrapper, SandboxStoreHandler)
and TS (FileReport) to follow the new RMS.
NEW: Full CRUD support in RMS.
*RSS
NEW: ResourceManagementDB - new table ErrorReportBuffer
NEW: new ResourceManagementClient methods - insertErrorReportBuffer, selectErrorReportBuffer,
deleteErrorReportBuffer
[v6r9]
NEW: Refactored Request Management System, related DMS agents and FTS management
components
[v6r8p28]
*Core
BUGFIX: RequestHandler - the lock Name includes ActionType/Action
*DMS
FIX: dirac-dms-filecatalog-cli - prevent exception in case of missing proxy
[v6r8p27]
*DMS
BUGFIX: dirac-dms-add-file - fixed typo item -> items
[v6r8p26]
*Core
NEW: RequestHandler - added getServiceOption() to properly resolve inherited options
in the global service handler initialize method
NEW: FileCatalogHandler, StorageElementHandler - use getServiceOption()
[v6r8p25]
FIX: included fixes from v6r7p40 patch release
*Resources
FIX: SRM2Storage - do not account gfal_ls operations
[v6r8p24]
FIX: included fixes from v6r7p39 patch release
*Core
FIX: SiteSEMapping was returning wrong info
*DMS
FIX: FTSRequest - choose explicitly target FTS point for RAL and CERN
BUGFIX: StrategyHandler - wrong return value in __getRWAccessForSE()
*Resources
CHANGE: SRM2Storage - do not account gfal_ls operations any more
[v6r8p23]
FIX: included fixes from v6r7p37 patch release
*TS
FIX: TransformationDB - allow tasks made with ProbInFC files
FIX: TransformationCleaingAgent,Client - correct setting of transformation
status while cleaning
[v6r8p22]
FIX: included fixes from v6r7p36 patch release
[v6r8p21]
*DMS
FIX: FileCatalog/DirectoryMetadata - even if there is no meta Selection
the path should be considered when getting Compatible Metadata
FIX: FileCatalog/DirectoryNodeTree - findDir will return S_OK( '' ) if dir not
found, always return the same error from DirectoryMetadata in this case.
*RSS
FIX: DowntimeCommand - use UTC time stamps
*TS
FIX: TransformationAgent - in _getTransformationFiles() get also ProbInFC files in
addition to Used
[v6r8p20]
*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs,
dirac-stager-monitor-requests, dirac-stager-show-stats
[v6r8p19]
*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'
[v6r8p18]
*TS
BUGFIX: TransformationAgent - regression in __cleanCache()
[v6r8p17]
FIX: included fixes from v6r7p32 patch release
*WMS
FIX: StalledJobAgent - for accidentally stopped jobs ExecTime can be not set,
set it to CPUTime for the accounting purposes in this case
[v6r8p16]
FIX: included fixes from v6r7p31 patch release
*WMS
BUGFIX: TaskQueueDB - fixed a bug in the negative matching conditions SQL construction
*RSS
NEW: improved doc strings of PEP, PDP modules ( part of PolicySystem )
FIX: Minor changes to ensure consistency if ElementInspectorAgent and
users interact simultaneously with the same element
CHANGE: removed DatabaseCleanerAgent ( to be uninstalled if already installed )
FIX: SummarizeLogsAgent - the logic of the agent was wrong, the agent has been re-written.
[v6r8p15]
*Core
FIX: X509Chain - fix invalid information when doing dirac-proxy-info without CS
( in getCredentials() )
*RSS
NEW: PDP, PEP - added support for option "doNotCombineResult" on PDP
[v6r8p14]
*Core
FIX: dirac-deploy-scripts - can now work with the system python
*WMS
NEW: dirac-wms-cpu-normalization - added -R option to modify a given configuration file
FIX: Executor/InputData - Add extra check for LFns in InputData optimizer, closes #1472
*Transformation
CHANGE: TransformationAgent - add possibility to kick a transformation (not skip it if no
unused files), by touching a file in workDirectory
BUGFIX: TransformationAgent - bug in __cleanCache() dict modified in a loop
[v6r8p13]
*Transformation
BUGFIX: TransformationDB - restored import of StringType
[v6r8p12]
NEW: Applied patches from v6r7p29
*WMS
FIX: JobDB - check if SystemConfig is present in the job definition and convert it
into Platform
*DMS
FIX: ReplicaManager - do not get metadata of files when getting files in a directory
if not strictly necessary
*RSS
NEW: ported from LHCb PublisherHandler for RSS web views
[v6r8p11]
NEW: Applied patches from v6r7p27
*RSS
NEW: SpaceTokenOccupancyPolicy - ported from LHCbDIRAC
NEW: db._checkTable done on service initialization ( removed dirac-rss-setup script doing it )
*Transformation
FIX: TaskManager - reset oJob for each task in prepareTransformationTasks()
BUGFIX: ValidateOutputDataAgent - typo fixed in getTransformationDirectories()
FIX: TransformationManagerHandler - use CS to get files statuses not to include in
processed file fraction calculation for the web monitoring pages
[v6r8p10]
NEW: Applied patches from v6r7p27
[v6r8p9]
*DMS
FIX: TransferAgent,dirac-dms-show-se-status, ResourceStatus,TaskManager - fixes
needed for DMS components to use RSS status information
NEW: ReplicaManager - allow to get metadata for an LFN+SE as well as PFN+SE
[v6r8p8]
*RSS
BUGFIX: dirac-rss-setup - added missing return of S_OK() result
[v6r8p7]
NEW: Applied patches from v6r7p24
*DMS
BUGFIX: LcgFileCatalogClient - bug in addFile()
*RSS
BUGFIX: fixed script dirac-rss-set-token, broken in the current release.
NEW: Statistics module - will be used in the future to provide detailed information
from the History of the elements
[v6r8p6]
NEW: Applied patches from v6r7p23
*Transformation
FIX: TaskManager - allow prepareTransformationTasks to proceed if no OutputDataModule is defined
FIX: TransformationDB - remove INDEX(TaskID) from TransformationTasks. It produces a single counter
for the whole table instead of one per TransformationID
*WMS
FIX: WMSUtilities - to allow support for EMI UI's for pilot submission we drop support for glite 3.1
[v6r8p5]
NEW: Applied patches from v6r7p22
*RSS
CHANGE: removed old tests and commented out files
*WMS
FIX: PoolXMLCatalog - proper addFile usage
*Transformation
CHANGE: TransformationAgent - clear replica cache when flushing or setting a file in the workdirectory
[v6r8p4]
*Transformation
FIX: The connection to the jobManager is done only at submission time
FIX: Jenkins complaints fixes
*WMS
BUGFIX: JobDB - CPUtime -> CPUTime
FIX: Jenkins complaints fixes
[v6r8p3]
*DMS
BUGFIX: LcgFileCatalogClient
[v6r8p2]
*DMS:
FIX: LcgFileCatalogClient - remove check for opening a session in __init__ as credentials are not yet set
*Transformation
CHANGE: reuse RPC clients in Transformation System
[v6r8p1]
*Core
FIX: dirac-deploy-scripts - restored regression w.r.t. support of scripts starting with "d"
*DMS
BUGFIX: LcgFileCatalogClient - two typos fixed
[v6r8]
CHANGE: Several fixes backported from the v7r0 integration branch
*Core
CHANGE: DictCache - uses global LockRing to avoid locks in multiprocessing
FIX: X509Chain - proxy-info showing an error when there's no CS
*DMS
FIX: TransferAgent - inside loop filter out waiting files dictionary
BUGFIX: dirac-admin-allow-se - there was a continue that was skipping the complete loop for
ARCHIVE elements
NEW: LcgFileCatalogClient - test return code in startsess lfc calls
*WMS:
FIX: OptimizerExecutor, InputData, JobScheduling - check that site candidates have all the
replicas
*RSS:
BUGFIX: ResourceStatus, RSSCacheNoThread - ensure that locks are always released
*Transformation
FIX: TaskManager - site in the job definition is taken into account when submitting
NEW: Transformation - get the allowed plugins from the CS /Operations/Transformations/AllowedPlugins
FIX: ValidateOutputDataAgent - self not needed for static methods
[v6r7p40]
*Resources
FIX: StorageElement class was not properly passing the lifetime argument for prestageFile method
[v6r7p39]
*Core
CHANGE: Grid - in executeGridCommand() allow environment script with arguments needed for ARC client
*DMS
FIX: DFC SEManager - DIP Storage can have a list of ports now
*Resources
FIX: ARCComputingElement - few fixes after debugging
[v6r7p38]
*Core
NEW: DISET FileHelper, TransferClient - possibility to switch off check sum
*Resources
NEW: ARCComputingElement - first version
NEW: StorageFactory - possibility to pass extra protocol parameters to storage object
NEW: DIPStorage - added CheckSum configuration option
BUGFIX: SSHComputingElement - use CE name in the pilot reference construction
*WMS
FIX: StalledJobAgent - if ExecTime < CPUTime make it equal to CPUTime
[v6r7p37]
*Framework
BUGFIX: NotificationDB - typos in SQL statement in purgeExpiredNotifications()
*WMS
NEW: JobCleaningAgent - added scheduling sandbox LFN removal request
when deleting jobs
CHANGE: JobWrapper - report only error code as ApplicationError parameter
when payload finishes with errors
NEW: SiteDirector - possibility to specify extensions to be installed in
pilots in /Operations/Pilots/Extensions option in order not to install
all the server side extensions
*DMS
CHANGE: FileCatalogFactory - use service path as default URL
CHANGE: FileCatalogFactory - use ObjectLoader to import catalog clients
*SMS
BUGFIX: StorageManagementDB, dirac-stager-monitor-jobs - small bug fixes ( sic, Daniela )
*Resources
CHANGE: DIPStorage - added possibility to specify a list of ports for multiple
service end-points
CHANGE: InProcessComputingElement - demote log message when payload failure
to warning, the job will fail anyway
FIX: StalledJobAgent - if pilot reference is not registered, this is not an
error of the StalledJobAgent, no log.error() in this case
*RMS
CHANGE: RequestTask - ensure that tasks are executed with user credentials
even with respect to queries to DIRAC services ( useServerCertificate
flag set to false )
[v6r7p36]
*WMS
FIX: CREAMCE, SiteDirector - make sure that the tmp executable is removed
CHANGE: JobWrapper - remove sending mails via Notification Service in case
of job rescheduling
*SMS
FIX: StorageManagementDB - fix a race condition when old tasks are set failed
between stage submission and update.
[v6r7p35]
*Stager
NEW: Stager API: dirac-stager-monitor-file, dirac-stager-monitor-jobs,
dirac-stager-monitor-requests, dirac-stager-show-stats
[v6r7p34]
*Transformation
FIX: TransformationCleaning Agent status was set to 'Deleted' instead of 'Cleaned'
[v6r7p33]
*Interfaces
FIX: Job.py - in setExecutable() - prevent changing the log file name string type
*StorageManagement
NEW: StorageManagementDB(Handler) - kill staging requests at the same time as
killing related jobs, closes #1510
FIX: StorageManagementDB - demote the level of several log messages
[v6r7p32]
*DMS
FIX: StorageElementHandler - do not use getDiskSpace utility, use os.statvfs instead
CHANGE: StorageManagementDB - in getStageRequests() make MySQL do an UNIQUE selection
and use implicit loop to speed up queries for large results
*Resources
FIX: lsfce remote script - use re.search instead of re.match in submitJob() to cope with
multipline output
[v6r7p31]
*WMS
FIX: SiteDirector - make possible more than one SiteDirector (with different pilot identity) attached
to a CE, ie sgm and pilot roles. Otherwise one is declaring Aborted the pilots from the other.
[v6r7p30]
*Core
CHANGE: X509Chain - added groupProperties field to the getCredentials() report
BUGFIX: InstallTools - in getSetupComponents() typo fixed: agent -> executor
[v6r7p29]
*DMS
CHANGE: FileCatalog - selection metadata is also returned as compatible metadata in the result
of getCompatibleMetadata() call
NEW: FileCatalog - added path argument to getCompatibleMetadata() call
NEW: FileCatalogClient - added getFileUserMetadata()
BUGFIX: dirac-dms-fts-monitor - exit with code -1 in case of error
*Resources
FIX: CREAMComputingElement - check globus-url-copy result for errors when retrieving job output
[v6r7p28]
*DMS
BUGFIX: FileCatalog/DirectoryMetadata - wrong MySQL syntax
[v6r7p27]
*Core
FIX: Mail.py - fix of the problem of colons in the mail's body
*Interfaces
NEW: Job API - added setSubmitPools(), setPlatform() sets ... "Platform"
*WMS
FIX: TaskQueueDB - use SystemConfig as Platform for matching ( if Platform is not set explicitly
*Resources
FIX: SSHComputingElement - use ssh host ( and not CE name ) in the pilot reference
BUGFIX: SSHGEComputingElement - forgotten return statement in _getJobOutputFiles()
*Framework
NEW: dirac-sys-sendmail - email's body can be taken from pipe. Command's argument
in this case will be interpreted as a destination address
[v6r7p26]
*DMS
FIX: ReplicaManager - status names Read/Write -> ReadAccess/WriteAccess
[v6r7p25]
*Core
CHANGE: X509Chain - in getCredentials() failure to contact CS is not fatal,
can happen when calling dirac-proxy-init -x, for example
[v6r7p24]
*DMS
NEW: FileCatalog - added getFilesByMetadataWeb() to allow pagination in the Web
catalog browser
*WMS
CHANGE: WMSAdministrator, DiracAdmin - get banned sites list by specifying the status
to the respective jobDB call
[v6r7p23]
*Transformation
BUGFIX: TransformationDB - badly formatted error log message
*RMS
CHANGE: RequestDBMySQL - speedup the lookup of requests
*WMS
BUGFIX: dirac-dms-job-delete - in job selection by group
*DMS
FIX: LcgFileCatalogClient - getDirectorySize made compatible with DFC
BUGFIX: LcgFileCatalogClient - proper call of __getClientCertInfo()
[v6r7p22]
*Transformation
CHANGE: InputDataAgent - treats only suitable transformations, e.g. not the extendable ones.
CHANGE: TransformationAgent - make some methods more public for easy overload
[v6r7p21]
*Core
FIX: Shifter - pass filePath argument when downloading proxy
[v6r7p20]
*DMS
CHANGE: StrategyHandler - move out SourceSE checking to TransferAgent
CHANGE: ReplicaManager, InputDataAgent - get active replicas
FIX: StorageElement, SRM2Storage - support for '*Access' statuses, checking results
of return structures
*RSS
NEW: set configurable email address on the CS to send the RSS emails
NEW: RSSCache without thread in background
FIX: Synchronizer - moved to ResourceManager handler
[v6r7p19]
*DMS
BUGFIX: ReplicaManager - in putAndRegister() SE.putFile() singleFile argument not used explicitly
[v6r7p18]
*WMS
FIX: StalledJobAgent - do not exit the loop over Completed jobs if accounting sending fails
NEW: dirac-wms-job-delete - allow to specify jobs to delete by job group and/or in a file
FIX: JobManifest - If CPUTime is not set, set it to MaxCPUTime value
[v6r7p17]
*Resources
FIX: SRM2Storage - treat properly "22 SRM_REQUEST_QUEUED" result code
[v6r7p16]
*DMS
FIX: StrategyHandler - do not proceed when the source SE is not valid for read
BUGFIX: StorageElement - putFile can take an optional sourceSize argument
BUGFIX: ReplicaManager - in removeFile() proper loop on failed replicas
*RSS
FIX: SpaceTokenOccupancyCommand, CacheFeederAgent - add timeout when calling lcg_util commands
*WMS
FIX: JobManifest - take all the SubmitPools defined in the TaskQueueAgent
NEW: StalledJobAgent - declare jobs stuck in Completed status as Failed
[v6r7p15]
*Core
BUGFIX: SocketInfo - in host identity evaluation
*DMS
BUGFIX: FileCatalogHandler - missing import os
*Transformation
CHANGE: JobManifest - getting allowed job types from operations() section
[v6r7p14]
*DMS
CHANGE: StorageElementProxy - removed getParameters(), closes #1280
FIX: StorageElementProxy - free the getFile space before the next file
FIX: StorageElement - added getPFNBase() to comply with the interface
*Interfaces
CHANGE: Dirac API - allow lists of LFNs in removeFile() and removeReplica()
*WMS
CHANGE: JobSchedulingAgent(Executor) - allow both BannedSite and BannedSites JDL option
*RSS
FIX: ElementInspectorAgent - should only pick elements with rss token ( rs_svc ).
FIX: TokenAgent - using 4th element instead of the 5th. Added option to set admin email on the CS.
[v6r7p13]
*Core
FIX: Resources - in getStorageElementSiteMapping() return only sites with non-empty list of SEs
*DMS
FIX: StorageElement - restored the dropped logic of using proxy SEs
FIX: FileCatalog - fix the UseProxy /LocalSite/Catalog option
*Transformation
FIX: TransformationDB - use lower() string comparison in extendTransformation()
[v6r7p12]
*WMS
BUGFIX: JobManifest - get AllowedSubmitPools from the /Systems section, not from /Operations
*Core
NEW: Resources helper - added getSites(), getStorageElementSiteMapping()
*DMS
CHANGE: StrategyHandler - use getStorageElementSiteMapping helper function
BUGFIX: ReplicaManager - do not modify the loop dictionary inside the loop
[v6r7p11]
*Core
CHANGE: Subprocess - put the use of watchdog in flagging
[v6r7p10]
*Core
NEW: Logger - added getLevel() method, closes #1292
FIX: Subprocess - returns correct structure in case of timeout, closes #1295, #1294
CHANGE: TimeOutExec - dropped unused utility
FIX: Logger - cleaned unused imports
*RSS
CHANGE: ElementInspectorAgent - do not use mangled name and removed shifterProxy agentOption
[v6r7p9]
*Core
BUGFIX: InstallTools - MySQL Port should be an integer
[v6r7p8]
*Core
FIX: Subprocess - consistent timeout error message
*DMS
NEW: RemovalTask - added bulk removal
FIX: StrategyHandler - check file source CEs