MSOutput created 2 tape rules for a GEN output

[haozturk]

Impact of the bug
MSOutput

Describe the bug
We had a request which had multiple workflows writing to the same output [1]. For this output, I saw that MSOutput created 2 tape rules most probably because there were multiple workflows producing the same output:

$ rucio list-rules cms:/InclusiveDileptonMinBias_TuneCP5Plus_13p6TeV_pythia8/GenericNoSmearGEN-124X_mcRun3_2022_realistic_v12-v2/GEN
ID                                ACCOUNT        SCOPE:NAME                                                                                                         STATE[OK/REPL/STUCK]     RSE_EXPRESSION      COPIES    SIZE    EXPIRES (UTC)    CREATED (UTC)
--------------------------------  -------------  -----------------------------------------------------------------------------------------------------------------  -----------------------  ------------------  --------  ------  ---------------  -------------------
24e8ea184aeb49e189af495b5c13622c  transfer_ops   cms:/InclusiveDileptonMinBias_TuneCP5Plus_13p6TeV_pythia8/GenericNoSmearGEN-124X_mcRun3_2022_realistic_v12-v2/GEN  SUSPENDED[161655/0/0]    T1_FR_CCIN2P3_Tape  1         N/A                      2023-08-08 08:53:30
76c69d47a254457f97636ebdff381ec1  wmcore_output  cms:/InclusiveDileptonMinBias_TuneCP5Plus_13p6TeV_pythia8/GenericNoSmearGEN-124X_mcRun3_2022_realistic_v12-v2/GEN  WAITING_APPROVAL[0/0/0]  T1_US_FNAL_Tape     1         N/A                      2024-08-05 14:18:34
53d7c108f92c435ea66dcf8dd92f8f8e  wmcore_output  cms:/InclusiveDileptonMinBias_TuneCP5Plus_13p6TeV_pythia8/GenericNoSmearGEN-124X_mcRun3_2022_realistic_v12-v2/GEN  OK[180979/0/0]           T1_FR_CCIN2P3_Tape  1         N/A                      2024-08-05 14:18:35

[1] https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_BPH-GenericNoSmearGEN-00001

How to reproduce it
Create multiple workflows which share the same prepid and write to the same output and let MSOutput handle their output placement.

Expected behavior
MSOutput should create a single tape rule for a given output dataset regardless of how many workflows produce it

Additional context and error message
I'll delete the FNAL tape rule. Is this going to create issues for workflow archival?

[amaltaro]

That is an interesting case that I had never thought about.

If we want to ensure a single tape rule for each output dataset, we need to add extra logic to MSOutput to check if there is any Tape rule (under wmcore_output account). In case we find a rule, we can either:
a) skip the Tape rule creation; or
b) proceed with the Tape rule creation, but make sure it goes to the same Tape RSE. This way, it will hit a duplicate rule condition and just reuse the previous rule id.

I am very much in favor of option b).

[haozturk]

To be able to preserve the expected behavior, it should be able to track the existing rule before archiving the workflow IMO.

Btw, I deleted 76c69d47a254457f97636ebdff381ec1