Setup on Alma8 machines to use Condor, OSG, and CMSSW on the HEX Computer Farm

In the very near future, we will be migrating the entire HEX computer farm to a new operating system, Almalinux 8, which will require some changes when running your setup scripts or submitting jobs to condor. For CMS users, there are some particularly important steps needed, depending on whether you are analyzing Run 2 or Run 3 data.

Connect to our test Alma8 machine, while logged in to hexcms:

ssh hexcms-alma8

Condor setup for non-CMS users or for those analyzing Run 3 CMS data:

source /condor/HTCondor/alma8/condor.(c)sh

# Note! When submitting condor jobs, be sure that your condor_submit file does not have any lines related to "C7", which was a flag to force the jobs to run on CentOS7 machines, which will soon no longer exist.

OSG setup for non-CMS users or for those analyzing Run 3 CMS data:

source /osg/alma8/setup.(c)sh

Note that if your .bashrc or .cshrc already contains lines sourcing /condor/HTCondor/current/condor.(c)sh or /osg/current/setup.(c)sh, it would be good to remove them, to avoid conflicts when logging on to the alma8 machine. Similarly, if you set a SCRAM_ARCH in a script that is run on condor, be sure that it is consistent with the one you set in the following steps.

For those analyzing CMS Run 3 data

CMSSW setup for bash shell for Run 3 data

export VO_CMS_SW_DIR=/cvmfs/cms.cern.ch

export SCRAM_ARCH=slc8_amd64_gccXXX # (replace "XXX" with the desired version of gcc; 10, 11, and 12 are currently valid options)

source $VO_CMS_SW_DIR/cmsset_default.sh

CMSSW setup for tcsh shell for Run 3 data

setenv VO_CMS_SW_DIR /cvmfs/cms.cern.ch

setenv SCRAM_ARCH slc8_amd64_gccXXX (replace "XXX" with the desired version of gcc; 10, 11, and 12 are currently valid options)

source $VO_CMS_SW_DIR/cmsset_default.csh

CMSSW release setup for Run 3 data:

# First, follow the "CMSSW setup" instructions above for either tcsh or bash.

# Then do:

cmsrel CMSSW_RRR (replace "RRR" with the desired CMSSW version)

# Note that you only need to do cmsrel once, not everytime you login.

# Then do:

cd CMSSW_RRR/src (replace "RRR" with the desired CMSSW version)

cmsenv

cd

# Note for those who intend to submit crab jobs, an additional step is needed:

source /cvmfs/cms.cern.ch/common/crab-setup.sh

Summary for Run 3 CMS analyses:

# To simplify things, all of the above steps *except* for the CMSSW release setup and setting of SCRAM_ARCH have been added to a single script:

source /home/joey/alma8_setups/setup_alma8_no_apptainer.(c)sh

For those analyzing CMS Run 2 data

An additional complication arises for those analyzing CMS Run 2 data, because the CMSSW releases for Run 2 data (e.g. CMSSW_10_6_X) were built against CentOS7, with no corresponding builds for Alma8. The Run 3 CMSSW releases (e.g. CMSSW_12_X_Y) are built against Alma8, which is why this complication is not relevant for the Run 3 setup documentation above. As a result, for the Run 2 analyses, the current recommendation is to use apptainer (formerly known as "singularity") containers, which create a virtual environment for us to run CMSSW code from older releases on newer operating systems. These containers have a minimal set of items installed, and do not even have emacs or vim (though vi and nano are both available); it is probably best to do any text editing in a separate shell that is not running apptainer, while using the container primarily for running code. For those interested, more detail can be found at https://cms-sw.github.io/singularity.html. For our purposes, we can do the following to set up the apptainer compatible with Run 2 data:

Set up cmssw-el7 container

source /home/joey/alma8_setups/setup_cmssw-el7_apptainer.(c)sh

# you're now in a singularity shell with a CentOS 7 environment

source /home/joey/alma8_setups/setup_inside_cmssw-el7_apptainer.(c)sh # this sets up the correct condor and OSG paths, as well as sourcing the cmsset_default and crab-setup scripts

From there, you should go to your existing Run 2 CMSSW working area, run `cmsenv`, and compile via `scram b`. If you don't yet have a CMSSW working area, you can make one in the usual way via:

cmsrel CMSSW_10_6_40 # or replace with your version of choice

For 10_6_X releases, in my experience, it is not necessary to set the SCRAM_ARCH, because the container automatically falls back to slc7_amd64_gcc700. However, for older releases, you may need to set SCRAM_ARCH to an appropriate value before attempting cmsrel.

Submitting condor jobs from the apptainer:

The final complication we have to worry about is that since we are migrating the condor nodes to Alma8, our CentOS7-compatible condor jobs will also need to run within a container! To do so, we must add the following to our condor_submit file:

+SingularityImage = "/cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmssw/el7:x86_64"

As mentioned earlier in this page, it's important to also be sure to remove any lines pertaining to "C7", which was an option that forces the jobs to run on CentOS7 machines, which will soon be deprecated.