CRAN status check codecov lint

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The NNG-powered mirai R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The crew package extends mirai with a unifying interface for third-party worker launchers. Inspiration also comes from packages future, rrq, clustermq, and batchtools.

📣 Request for community contributions 📣

The launcher plugin framework aims to extend crew to modern platforms and services like Google Cloud Run, Kubernetes, and beyond. At the time of writing, plugins for cloud computing do not yet exist. So if you have access to these services and know how to use them, please consider contributing a package with plugins of your own. The tutorial at https://wlandau.github.io/crew/articles/plugins.html explains how. The maintainer of crew would love to work with you!

Installation

Type Source Command
Release CRAN install.packages("crew")
Development GitHub remotes::install_github("wlandau/crew")
Development R-universe install.packages("crew", repos = "https://wlandau.r-universe.dev")

Risks

The crew package has unavoidable risks, and the user is responsible for safety, security, and computational resources. Please read the software license and the vignette about specific known risks.

Similar work

  • mirai: a powerful R framework for asynchronous tasks built on NNG. The purpose of crew is to extend mirai to different computing platforms for distributed workers.
  • rrq: a task queue for R based on Redis.
  • rrqueue: predecessor of rrq.
  • clustermq: sends R function calls as jobs to computing clusters.
  • future: a unified interface for asynchronous evaluation of single tasks and map-reduce calls on a wide variety of backend technologies.
  • batchtools: tools for computation on batch systems.
  • targets: a Make-like pipeline tool for R.
  • later: delayed evaluation of synchronous tasks.
  • promises: minimally-invasive asynchronous programming for a small number of tasks within Shiny apps.
  • callr: initiates R process from other R processes.
  • High-performance computing CRAN task view.

Thanks

The crew package incorporates insightful ideas from the following people.

Code of Conduct

Please note that the crew project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Citation

To cite package ‘crew’ in publications use:

  Landau WM (2023). _crew: A Distributed Worker Launcher Framework_.
  https://wlandau.github.io/crew/, https://github.com/wlandau/crew.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {crew: A Distributed Worker Launcher Framework},
    author = {William Michael Landau},
    year = {2023},
    note = {https://wlandau.github.io/crew/, https://github.com/wlandau/crew},
  }