HTCondor¶

TLDR

What: HTCondor manages jobs on unused ICP workstations - your default tool instead of manually ssh-ing onto machines.
Best for: Jobs that fit on a single workstation (a few cores at most); use Ant/Bee/other clusters for >~8 cores.
Submit: condor_submit <job script>.
Monitor: condor_q, condor_q -run, condor_q -hold.
Remove: condor_rm <job_id/user_name>.
Cluster status: condor_status, condor_userprio.
Critical: Jobs can be evicted anytime (no guaranteed contiguous runtime) - implement checkpointing.
Storage: Avoid heavy/frequent I/O and don’t fill your home directory; use /data or /work.

On the ICP workstations, we use HTCondor to efficiently manage high-throughput computing (HTC) jobs that require extended runtimes. HTCondor allows users to submit jobs to a centralized scheduler, which queues and distributes them across available resources. It’s best suited for jobs fitting at most a single workstation.

This should be your default for using compute-resources at the ICP. Do not ssh onto other machines and run jobs locally - this tool exists exactly to solve this issue.

Things to remember¶

HTCondor is a system to run multiple jobs (high throughput computing) on unused ICP compute resources.
All workstations, including the CIP pool at the ICP, are part of the Condor cluster.
Condor is best suited for jobs needing resources a single workstation provides.
This means parallel jobs up to a couple of cores are possible.
Generally, make sure your jobs do not write to disk more often than necessary.
Extensive I/O hurts compute performance and can slow down the storage servers as well, ruining the usability for everyone. So write what is necessary, and try to bundle small outputs in memory and dump them to disk periodically.
Don’t write too often or in large amounts to the home directory from a Condor job, as it can fill up your home directory.
Simulation data should generally be stored in /data or /work directories. More info about data storage can be found on the wiki.
HTCondor does not promise you a contiguous runtime. It evicts jobs if a user needs more resources on the computer where the job is running, if another user has a higher priority (i.e., has used Condor less), or if it has to be restarted due to pending updates. So having checkpointing implemented in your job is CRUCIAL.
For jobs running more than a couple of minutes, you need to implement checkpointing to resume simulation from the point when the job was evicted. You can write checkpoints periodically or react when receiving a signal. Condor sends SIGTERM (15) signal to a job before kicking it out. You can catch that signal to perform checkpointing, however, be aware that you’re only having a couple of minutes from here.

A summary of important commands for using HTCondor can be found in this quick start guide.

Submitting a Job¶

To submit a simulation on Condor, you need a Condor job script, which you can submit using the condor_submit <job script> command. You can add more switches to condor_submit if you wish to customize the process. Make sure to properly specify request_cpus and request_memory to reasonable sizes, these limits are enforced automatically. If you e.g. exceed the memory requested, your job will be put on hold.

Job Script¶

An example job script:

# Simple HTCondor submit description file
# Everything with a leading # is a comment

executable   = myexe      # Path to your executable file
arguments    = arguments  # Arguments you want to pass through the command line

output       = OUTPUT # Path for output file, which records the console output
error        = ERROR  # Path for error file, which records error signals
log          = LOG    # Path for Condor log file

request_cpus   = 1       # Number of CPUs required for your job
request_memory = 1024M   # RAM allocated to your job, you can also use G for GB ex. 1G instead of 1024M
request_disk   = 10240K  # Disk space required for your job

BATCH_NAME = my_investigation

queue # Queues the Condor job

HTCondor website explains this in more detail. You can ignore everything there about file transfers, as we’re on a shared-filesystem. However, be aware that e.g. /ssd is a local directory, so avoid it where possible. However, if you need a temporary directory per job e.g. for caches, Condor maintains one for you and cleans it up afterwards for you, which can be found as an environment variable _CONDOR_SCRATCH_DIR; others you can find here. For submitting multiple jobs with varying arguments, you can use the following script:

# Condor submission file for running a program with different arguments

executable     = <path_to_executable>
arguments = $(ARG)
request_memory = 1G
request_cpus   = 1

# Specify the output, error, and log files
output = output_$(ARG).out
error  = error_$(ARG).err
log    = log_$(Cluster).log

BATCH_NAME = my_investigation

# Queue jobs with different arguments
queue ARG from args.txt

It loads arguments from the file args.txt, which is in the same directory from where you submit the job. Otherwise, you need to specify the full path. The args.txt file looks like this:

argument1
argument2
argument3
.
.
.
argumentN

For submitting GPU jobs, you have to additionally request a GPU, so extend your submit-file with

request_gpus = 1
gpus_minimum_memory = 1G

Because large GPUs are sparse in our cluster, HTCondor is configured such that machines with large GPUs prefer jobs requiring large GPUs, at the cost of a larger slot-weight, influencing your user-priority.

Monitoring a Simulation¶

As soon as you submit a simulation on the cluster, a job id appears on the screen. You can track your job with condor_q.

condor_q shows information on jobs submitted to the queue.
condor_q <user_name> shows information on jobs you submitted to the queue.
condor_q -run shows information on which machines your jobs are running.
condor_q -hold shows information on held jobs, which also includes reasons why the job is held.

Refer to condor_q for more details.

Removing a Job¶

To remove a job from the cluster queue, use condor_rm <job_id/user_name>. You can retrieve your <job_id> by running condor_q. condor_rm <user_name> removes all the jobs submitted by the user.

Why is my job not running? Why got my job kicked?¶

Not running / not starting: Run condor_q <job_id> --better-analyze to check for a mis-configuration in your submission, e.g. requesting more resources than any machine can provide.
Kicked / evicted while running: Check your user-priority with condor_userprio. If another user with a lower priority (i.e., who has used Condor less) needed resources, your job may have been preempted in their favor.
Effective priority: This is the mechanism HTCondor uses to distribute resources fairly among users over time. Resources have different weights - a large GPU is roughly worth 8 CPU cores, depending on its compute-capability. Run condor_userprio to see how many resources everyone is currently using (column Wghted In Use) and everyone’s effective priority. For preemption to be considered, your effective priority needs to be at least 20% higher than theirs.
Why this happens: HTCondor jobs have no time-limit, so eviction is the mechanism that keeps resources shared fairly among users. This is exactly why implementing checkpointing in your job is essential - see Things to remember above.

Useful Commands¶

condor_status: Shows the status of all machines on Condor cluster.
condor_status -constraint 'State == "<State>"': Use to filter machines by their state: Claimed, Unclaimed, or Drained.
condor_userprio: Shows current resource utilization by each user and their corresponding Effective Priority.
condor_drain <machine_name>: Drains all Condor jobs on a particular machine. Use only if Condor jobs are inhibiting tasks on your own machine.
condor_drain -cancel <machine_name>: Use it to un-drain a machine. Only un-drain other machines if you are certain that no one is using them.
condor_vacate: Evicts all currently running Condor jobs from your machine. Condor will immediately fill it again, if it needs the resources.
condor_rm -constraint 'JobBatchName == "my_investigation"'': You can also constraint other commands, e.g. for removing jobs from the queue, if you have named them properly.

Additional Notes¶

To see why your job is not running, you can use condor_q <job_id> --better-analyze. It will output which machines fulfill each of your requirements, so you can check yourself why a job is not running.
For a list of options, refer to the requirements section in the manual.
You can specify requirements to machines as well, like their instruction set or other parameters. This is typically not necessary and if you have issues, please report them to the IT admin for it to be fixed. However, if you ever need to run a job on a specific machine or exclude one, you can use:
```
# include
requirements = (machine == "<machine_name>.icp.uni-stuttgart.de")
# exclude
requirements = (machine != "<machine_name>.icp.uni-stuttgart.de")
```

It is highly discouraged to include or exclude single machines if not necessary. If a machine is causing you problems, please always report it to the IT admins to get it fixed.

For MPI parallel simulations, you can request several processor cores:
```
executable     = /usr/bin/mpirun
arguments = -np <n-process> <path_to_your_program> <arguments_to_your_program>
request_cpus   = <n-process>
```
Note: For jobs with more than ~8 cores, it is probably better to use Ant, Bee, or one of the other clusters.
There is a possibility to use multiple conditions in the job script and use variables. Visit the conditionals page or the variables page for more information.