# Parallel Workers on High-Performance Compute Environments

# Introduction

This vignette illustrates how to launch parallel workers via job schedulers running in high-performance compute (HPC) environments.

# Examples

## Example: Launch parallel workers via the Grid Engine job scheduler

‘Grid Engine’ is a high-performance compute (HPC) job scheduler where one can request compute resources on multiple nodes, each running multiple cores. Examples of Grid Engine schedulers are Oracle Grid Engine (formerly Sun Grid Engine), Univa Grid Engine, and Son of Grid Engine - all commonly referred to as SGE schedulers. Each SGE cluster may have its own configuration with its own way of requesting parallel slots.

Consider the following two files: `script.sh` and `script.R`.

script.sh:

    #! /usr/bin/env bash
    #$ -cwd               ## Run in current working directory
    #$ -j y               ## Merge stdout and stderr
    #$ -l mem_free=100M   ## 100 MiB RAM per slot
    #$ -l h_rt=00:10:00   ## 10 minutes runtime
    #$ -pe mpi 8          ## 8 compute slots

    echo "Information on R:"
    Rscript --version

    echo "Running R script:"
    Rscript script.R

script.R:

    library(parallelly)
    library(parallel)

    cl <- makeClusterPSOCK(
      availableWorkers(),
      rshcmd = "qrsh", rshopts = c("-inherit", "-nostdin", "-V")
    )
    print(cl)

    # Perform calculations in parallel
    X <- 1:100
    y <- parLapply(cl = cl, X, fun = sqrt)
    y <- unlist(y)
    z <- sum(y)
    print(z)

    stopCluster(cl)

The `script.sh` file is a job script that we submit to the scheduler that runs the R script `script.R` when launched. If we submit `script.sh` as:

    $ qsub script.sh

it will by default request eight slots - on one or more machines, which then R and parallelly will set up a parallel cluster on. Exactly on which machines depends on where the job scheduler finds these requested slots.

Here is the output from one such run, where the scheduler happened to allot the slots across three machines:

    Information on R:
    Rscript (R) version 4.5.2 (2025-10-31)
    Running R script:
    Socket cluster with 8 nodes where 4 nodes are on host ‘localhost’
    (R version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu), 3
    nodes are on host ‘qb3-id130’ (R version 4.5.2 (2025-10-31),
    platform x86_64-pc-linux-gnu), 1 node is on host ‘qb3-as16’ (R
    version 4.5.2 (2025-10-31), platform x86_64-pc-linux-gnu)
    [1] 671.4629

## Example: Launch parallel workers via the Fujitsu Technical Computing Suite job scheduler

The ‘Fujitsu Technical Computing Suite’ is a high-performance compute (HPC) job scheduler where one can request compute resources on multiple nodes, each running multiple cores.

Consider the following two files: `script.sh` and `script.R`.

script.sh:

    #! /usr/bin/env bash

    echo "Information on R:"
    Rscript --version

    echo "Running R script:"
    Rscript script.R

script.R:

    library(parallelly)
    library(parallel)

    cl <- makeClusterPSOCK(
      availableWorkers(),
      rshcmd = "pjrsh"
    )
    print(cl)

    # Perform calculations in parallel
    X <- 1:100
    y <- parLapply(cl = cl, X, fun = sqrt)
    y <- unlist(y)
    z <- sum(y)
    print(z)

    stopCluster(cl)

The `script.sh` file is a job script that we submit to the scheduler that runs the R script `script.R` when launched. We can submit `script.sh` as:

    $ pjsub -L vnode=3 -L vnode-core=18 script.sh

to request 18 CPU cores on three compute nodes, which in total requests 3*18=54 compute slots.