In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The crew.cluster
package extends the mirai
-powered ‘crew’ package with worker launcher plugins for traditional high-performance computing systems. Inspiration also comes from packages mirai
, future
, rrq
, clustermq
, and batchtools
.
Type | Source | Command |
---|---|---|
Release | CRAN | install.packages("crew.cluster") |
Development | GitHub | remotes::install_github("wlandau/crew.cluster") |
Development | R-universe | install.packages("crew", repos = "https://crew.cluster.r-universe.dev") |
Please see https://wlandau.github.io/crew.cluster/ for documentation, including a full function reference and usage tutorial.
First, create a controller object appropriate for your platform. For example, to launch workers on a Sun Grid Engine (SGE) cluster, use crew_controller_sge()
.
library(crew.cluster)
controller <- crew_controller_sge(
name = "my_workflow", # for informative job names
workers = 16,
tasks_max = 2, # to avoid reaching wall time limits
seconds_idle = 10, # to release resources when they are not needed
script_lines = "module load R" # if R is an environment module
)
controller$start()
At this point, usage is exactly the same as basic crew
. The push()
method submits tasks and auto-scales SGE workers to meet demand.
controller$push(name = "do work", command = do_work())
The pop()
method retrieves available tasks.
controller$pop()
#> # A tibble: 1 × 11
#> name command result seconds seed error trace warni…¹ launc…² worker insta…³
#> <chr> <chr> <list> <dbl> <int> <chr> <chr> <chr> <chr> <int> <chr>
#> 1 do work … do_work… <int> 0 1.56e8 NA NA NA 79e71c… 1 7686b2…
#> # … with abbreviated variable names ¹warnings, ²launcher, ³instance
Remember to terminate the controller when you are done.
controller$terminate()
crew.cluster
submits jobs over the local network using system calls to the resource manager (e.g SGE or SLURM). Please invoke crew.cluster
on a node of the cluster, either a login node (head node) or a compute node.module load R
(or module load R/x.y.z
for a specific version) in order to use on the cluster. In crew.cluster
, you will most likely need to supply "module load R"
or similar to the script_lines
argument of e.g. crew_controller_sge()
.The risks of crew.cluster
are the same as those of crew
, plus the risks of traditional high-performance computing environments. These distributed systems typically operate inside a firewall and trust the local network. It is your responsibility to assess the security of these systems and use crew.cluster
in a safe manner. In addition, crew.cluster
automatically launches jobs on the cluster scheduler, and it may not always be able to terminate leftover jobs. It is your responsibility to monitor your jobs an manually terminate jobs that crew.cluster
may not be able to.
mirai
and nanonext
and graciously accommodated the complicated and demanding feature requests that made crew
and its ecosystem possible.clustermq
with the permissive Apache License 2.0. These scripts helped construct launcher plugins to clusters where direct access was not possible. See the NOTICE
file in this package.Please note that the crew
project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
citation("crew.cluster")