We have migrated the entire HEX computer farm to a new operating system, AlmaLinux 8, which will require some changes when running your setup scripts or submitting jobs to condor. For CMS users, there are some particularly important steps needed, depending on whether you are analyzing Run 2 or Run 3 data.
Everything below is shown assuming the bash shell, but the scripts all exist for tcsh as well. If you would like to change your default shell, you can run ypchsh
and specify /bin/bash
, or email Joey for help.
ssh your-username@hexcms.rutgers.edu
source /condor/HTCondor/alma8/condor.sh
source /osg/alma8/setup.sh
If your .bashrc
or .cshrc
already contains lines sourcing /condor/HTCondor/current/condor.sh
or /osg/current/setup.sh
, you *have to remove them*, since they they refer to the old Condor scheduler, and things will not work properly. Similarly, if you set a SCRAM_ARCH
in a script that is run on condor, be sure that it is consistent with the one that CMSSW sets for you locally (or better yet, try removing the SCRAM_ARCH
from your script--CMSSW should autodetect it).
The simplest thing to do to automatically have the OSG software and Condor work upon each login, without breaking apptainer usage (which is described in more detail in a later section of this page), is to put the following into your ~/.bash_profile, which is invoked each time you log on to hexcms:
- # Set up condor and OSG software automatically, but not inside of an apptainer
- if [ -z "$APPTAINER_NAME" ]; then
- source /condor/HTCondor/alma8/condor.sh
- source /osg/alma8/setup.sh
- fi
- source /cvmfs/cms.cern.ch/cmsset_default.sh
- cmsrel CMSSW_14_0_15 # replace "14_0_15" with the desired CMSSW version. You only need to do cmsrel once, not everytime you login
- cd CMSSW_14_0_15/src
- cmsenv
- source /cvmfs/cms.cern.ch/common/crab-setup.sh # only needed if you run crab jobs
source /home/joey/alma8_setups/setup_alma8_no_apptainer.sh
An additional complication arises for those analyzing CMS Run 2 data, because the CMSSW releases for Run 2 data (e.g. CMSSW_10_6_X) were built against CentOS7, with no corresponding builds for Alma8. The Run 3 CMSSW releases (e.g. CMSSW 12, 13, and 14) are built against Alma8, which is why this complication is not relevant for the Run 3 setup documentation above. As a result, for the Run 2 analyses, the current recommendation is to use apptainer (formerly known as "singularity") containers, which create a virtual environment for us to run CMSSW code from older releases on newer operating systems. These containers have a minimal set of items installed, and do not even have emacs (though vim and nano are both available); it is probably best to do any text editing in a separate shell that is not running apptainer, while using the container primarily for running code. For those interested, more detail can be found at https://cms-sw.github.io/singularity.html. For our purposes, we can do the following to set up the apptainer compatible with Run 2 data:
- source /home/joey/alma8_setups/setup_cmssw-el7_apptainer.sh
From there, you should go to your existing Run 2 CMSSW working area, run cmsenv
, and compile via scram b
. If you don't yet have a CMSSW working area, you can make one in the usual way via:
cmsrel CMSSW_10_6_40 # or replace with your version of choice
For 10_6_X releases, in my experience, it is not necessary to set the SCRAM_ARCH
, because the container automatically falls back to slc7_amd64_gcc700
. However, for older releases, you may need to set SCRAM_ARCH
to an appropriate value before attempting cmsrel.
The final complication we have to worry about is that since the condor nodes are on Alma8, our CentOS7-compatible condor jobs will also need to run within a container! To do so, we must add the following to our condor_submit file:
+SingularityImage = "/cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/el7:x86_64"
As mentioned earlier in this page, it's important to also be sure to remove any lines pertaining to "C7".
To transfer centrally produced CMS samples to Rutgers, you can use Rucio. First, you will need a quota on our Rucio area (ask Joey to set this up for you, and how much space you anticipate needing). Then, you should run:
- # add this alias to your .bash_profile or .bashrc to make life easier!
- alias setuprucio='source /cvmfs/cms.cern.ch/cmsset_default.sh; source /cvmfs/cms.cern.ch/rucio/setup-py3.sh'
- setuprucio
- rucio add-rule cms:/QCD_PT-1800to2400_TuneCP5_13p6TeV_pythia8/Run3Summer22EEDRPremix-124X_mcRun3_2022_realistic_postEE_v1-v2/AODSIM 1 T3_US_Rutgers --notify Y
Of course, adjust the sample to the one that you actually want! Just note that cms:/
must be at the start of the sample name when calling rucio add-rule
. You can get sample names via DAS with commands such as the one below:
- dasgoclient -query "dataset dataset=/GJets/UL*/MINIAODSIM"
To monitor the status of your rucio transfers, check our Rucio status page. Once the samples have transferred, you can find the files in /cms/se/phedex/store/
in the data
and mc
subdirectories. To list all of your rules, use
- rucio list-rules --account YOUR-CERN-USERNAME
rucio delete-rule cd2188f0f97e4e08be07659bb8b8be0d
(for the appropriate rule ID that the list-rules command printed out for you).