Setup on Alma8 machines to use Condor, OSG, and CMSSW on the HEX Computer Farm

We have migrated the entire HEX computer farm to a new operating system, AlmaLinux 8, which will require some changes when running your setup scripts or submitting jobs to condor. For CMS users, there are some particularly important steps needed, depending on whether you are analyzing Run 2 or Run 3 data.

Everything below is shown assuming the bash shell, but the scripts all exist for tcsh as well. If you would like to change your default shell, you can run ypchsh and specify /bin/bash, or email Joey for help.

Connect to hexcms:

ssh your-username@hexcms.rutgers.edu

Condor setup for non-CMS users or for those analyzing Run 3 CMS data:

source /condor/HTCondor/alma8/condor.sh

When submitting condor jobs, be sure that your condor_submit file does not have any lines related to "C7". This was a flag to force the jobs to run on CentOS7 machines, which no longer exist.

OSG setup for non-CMS users or for those analyzing Run 3 CMS data:

source /osg/alma8/setup.sh

If your .bashrc or .cshrc already contains lines sourcing /condor/HTCondor/current/condor.sh or /osg/current/setup.sh, you *have to remove them*, since they they refer to the old Condor scheduler, and things will not work properly. Similarly, if you set a SCRAM_ARCH in a script that is run on condor, be sure that it is consistent with the one that CMSSW sets for you locally (or better yet, try removing the SCRAM_ARCH from your script--CMSSW should autodetect it).

The simplest thing to do to automatically have the OSG software and Condor work upon each login, without breaking apptainer usage (which is described in more detail in a later section of this page), is to put the following into your ~/.bash_profile, which is invoked each time you log on to hexcms:

# Set up condor and OSG software automatically, but not inside of an apptainer

if [ -z "$APPTAINER_NAME" ]; then

  source /condor/HTCondor/alma8/condor.sh

  source /osg/alma8/setup.sh

fi

For those analyzing CMS Run 3 data

CMSSW release setup for Run 3 data:

source /cvmfs/cms.cern.ch/cmsset_default.sh

cmsrel CMSSW_14_0_15 # replace "14_0_15" with the desired CMSSW version. You only need to do cmsrel once, not everytime you login

cd CMSSW_14_0_15/src

cmsenv

source /cvmfs/cms.cern.ch/common/crab-setup.sh # only needed if you run crab jobs

Summary for Run 3 CMS analyses:

To simplify things, all of the above steps *except* for the CMSSW release setup have been added to a single script:

source /home/joey/alma8_setups/setup_alma8_no_apptainer.sh

For those analyzing CMS Run 2 data

An additional complication arises for those analyzing CMS Run 2 data, because the CMSSW releases for Run 2 data (e.g. CMSSW_10_6_X) were built against CentOS7, with no corresponding builds for Alma8. The Run 3 CMSSW releases (e.g. CMSSW 12, 13, and 14) are built against Alma8, which is why this complication is not relevant for the Run 3 setup documentation above. As a result, for the Run 2 analyses, the current recommendation is to use apptainer (formerly known as "singularity") containers, which create a virtual environment for us to run CMSSW code from older releases on newer operating systems. These containers have a minimal set of items installed, and do not even have emacs (though vim and nano are both available); it is probably best to do any text editing in a separate shell that is not running apptainer, while using the container primarily for running code. For those interested, more detail can be found at https://cms-sw.github.io/singularity.html. For our purposes, we can do the following to set up the apptainer compatible with Run 2 data:

Set up cmssw-el7 container

source /home/joey/alma8_setups/setup_cmssw-el7_apptainer.sh

From there, you should go to your existing Run 2 CMSSW working area, run cmsenv, and compile via scram b. If you don't yet have a CMSSW working area, you can make one in the usual way via:

For 10_6_X releases, in my experience, it is not necessary to set the SCRAM_ARCH, because the container automatically falls back to slc7_amd64_gcc700. However, for older releases, you may need to set SCRAM_ARCH to an appropriate value before attempting cmsrel.

The final complication we have to worry about is that since the condor nodes are on Alma8, our CentOS7-compatible condor jobs will also need to run within a container! To do so, we must add the following to our condor_submit file:

As mentioned earlier in this page, it's important to also be sure to remove any lines pertaining to "C7".

Setup for rucio transfers

To transfer centrally produced CMS samples to Rutgers, you can use Rucio. First, you will need a quota on our Rucio area (ask Joey to set this up for you, and how much space you anticipate needing). Then, you should run:

# add this alias to your .bash_profile or .bashrc to make life easier!

alias setuprucio='source /cvmfs/cms.cern.ch/cmsset_default.sh; source /cvmfs/cms.cern.ch/rucio/setup-py3.sh'

setuprucio

rucio add-rule cms:/QCD_PT-1800to2400_TuneCP5_13p6TeV_pythia8/Run3Summer22EEDRPremix-124X_mcRun3_2022_realistic_postEE_v1-v2/AODSIM 1 T3_US_Rutgers --notify Y

Of course, adjust the sample to the one that you actually want! Just note that cms:/ must be at the start of the sample name when calling rucio add-rule. You can get sample names via DAS with commands such as the one below:

dasgoclient -query "dataset dataset=/GJets/UL*/MINIAODSIM"

To monitor the status of your rucio transfers, check our Rucio status page. Once the samples have transferred, you can find the files in /cms/se/phedex/store/ in the data and mc subdirectories. To list all of your rules, use

rucio list-rules --account YOUR-CERN-USERNAME