Missing "hltScoutingMuonPacker" in ScoutingNANO using recent Run-3 data

[silviodonato]

I tried to run NANO:@Scout using recent 2024 data, but I got the following two errors:

----- Begin Fatal Exception 23-Apr-2024 14:25:05 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 379617 lumi: 9 event: 16054775 stream: 0
   [1] Running path 'NANOAODoutput_step'
   [2] Prefetching for module NanoAODOutputModule/'NANOAODoutput'
   [3] Calling method for module SimpleRun3ScoutingMuonFlatTableProducer/'muonScoutingTable'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: Run3ScoutingMuon
Looking for module label: hltScoutingMuonPacker
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

and

----- Begin Fatal Exception 23-Apr-2024 14:32:26 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 379617 lumi: 9 event: 16054775 stream: 0
   [1] Running path 'NANOAODoutput_step'
   [2] Prefetching for module NanoAODOutputModule/'NANOAODoutput'
   [3] Calling method for module SimpleRun3ScoutingVertexFlatTableProducer/'displacedvertexScoutingTable'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: Run3ScoutingVertex
Looking for module label: hltScoutingMuonPacker
Looking for productInstanceName: displacedVtx

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "TryToContinue = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

I used
cmsDriver.py step2 -s NANO:@Scout --process NANO --data --eventcontent NANOAOD,DQM --datatier NANOAOD,DQMIO -n 10000 --customise "Configuration/DataProcessing/Utils.addMonitoring" --era Run3 --conditions auto:run3_data --filein file:xxx in CMSSW_14_0_5_patch1 (taken from runTheMatrix.py -l 2500.5).
You should be able to reproduce the error using a recent scouting data

I managed to run Nano adding :

process.muonScoutingTable.src = "hltScoutingMuonPackerNoVtx"
process.displacedvertexScoutingTable.src = "hltScoutingMuonPackerNoVtx","displacedVtx"

to the configuration.

My suggestion is to add a modifier, or something similar, able to run with 2022-23 and 2024 data, creating a new workflow in runTheMatrix.
The cause of the crash is CMSHLT-3089

cc: @Martin-Grunewald @mmusich @cms-sw/hlt-l2 @elfontan @karim El-Morabit

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @silviodonato.

@rappoccio, @smuzaffar, @makortel, @Dr15Jones, @sextonkennedy, @antoniovilela can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

assign xpog

[cmsbuild]

New categories assigned: xpog

@vlimant,@hqucms you have been requested to review this Pull request/Issue and eventually sign? Thanks

[elfontan]

Tagging @patinkaew as well

[vlimant]

@alintulu can you please prepare a PR introducing a run3_2024 era

[mmusich]

can you please prepare a PR introducing a run3_2024 era

just for the record, TSG has been in favor of this since some time and already provided a PR for that: #43761.
This was rejected by PPD on the grounds that after passing the HCal PF thresholds via DB, the whole of Run 3 could be treated by a single era (Run3) to be applied uniformly to all of its years.
Maybe this doesn't apply to nano , or maybe the situation changed in the meanwhile.
More reading material: #43756 (comment)
Tagging @malbouis

[malbouis]

I think that in order to proceed with the Run3_2024 Era, the only request is to not carry over the HCal PFRecHit thresholds as defined in Run3_2023. IIUC, this could be done by using the Run3 Era as a base. We have always said from the beginning that if a new Era is really needed, then it could be implemented, one should just be careful with the the call to the configuration with hardcoded hcal thresholds in the Run3_2023 Era, as this should be avoided.

[patinkaew]

@vlimant Adelina has graduated and I'm currently taking over her work.

I have prepared the fix in my private branch. This add run3_scouting_nanoAOD_2024 era modifier and introduce Run3_2024 using Run3 as base, i.e.

Run3_2024 = cms.ModifierChain(Run3, run3_scouting_nanoAOD_2024)

Without introducing Run3_2024, it is also possible to call --era Run3,run3_scouting_nanoAOD_2024.

It is not clear to me whether we want to introduce Run3_2024 according to the current discussion. Please let me know which option we prefer and I can make a PR.

[vlimant]

I don't think you need a run3_scouting_nanoAOD_2024 era, and a top level one will do.
I would recommend to call it run3_post23 as there will be a run3 in 2025, leading to confusion with something named Run3_2024.
Please make a PR along these lines for master and and 14.0