Controller groups

Each controller object only supports only one type of worker configuration which you set in advance. However, different controllers may have different types of workers, and crew supports controller groups to coordinate among these different worker types. With third-party launcher subclasses from other packages, this mechanism will allow you to e.g. send some tasks to GPU-capable or high-memory workers while other tasks go to low-spec workers.

We demonstrate with a controller of fully persistent workers which always stay running and a controller of semi-persistent workers which terminate after completing four tasks. We create controller objects with names.

library(crew)
persistent <- crew_controller_local(name = "persistent")
transient <- crew_controller_local(name = "semi-persistent", tasks_max = 4L)

crew uses a different TCP port for each controller you run, so please do not create hundreds of controllers. Please see the subsection on ports in the README.

We put these controller objects into a new controller group object.

group <- crew_controller_group(persistent, transient)

This controller group has a global connect() method to initialize both controllers.

group$start()

You can choose which worker pool to receive tasks.

group$push(name = "my task", command = sqrt(4), controller = "semi-persistent")

The controller group also supports global methods for wait(), pop(), and terminate(). These methods operate on all controllers at once by default, but the controllers argument allows you to select a subset of controllers to act on. Below in pop() the launcher column of the output indicates which controller ran the task.

group$wait(controllers = "semi-persistent")
group$pop()
#> # A tibble: 1 × 12
#>   name    command result seconds  seed algorithm error trace warnings
#>   <chr>   <chr>   <list>   <dbl> <int> <chr>     <chr> <chr> <chr>
#> 1 my task NA      <dbl>        0    NA NA        NA    NA    NA
#> # ℹ 3 more variables: launcher <chr>, worker <int>, instance <chr>

The map() method provides functional programming, and the controller argument lets you choose the controller to submit the tasks.

group$map(
  command = a + b + c + d,
  iterate = list(
    a = c(1, 3),
    b = c(2, 4)
  ),
  data = list(c = 5),
  globals = list(d = 6),
  controller = "persistent"
)
#> # A tibble: 2 × 12
#>   name  command result    seconds  seed algorithm error trace warnings
#>   <chr> <chr>   <list>      <dbl> <int> <chr>     <chr> <chr> <chr>
#> 1 1     NA      <dbl [1]>       0    NA NA        NA    NA    NA
#> 2 2     NA      <dbl [1]>       0    NA NA        NA    NA    NA
#> # ℹ 3 more variables: launcher <chr>, worker <int>, instance <chr>

The controller group has a summary() method which aggregates the summaries of one or more controllers.

group$summary()
#> # A tibble: 2 × 6
#>   controller      worker tasks seconds errors warnings
#>   <chr>            <int> <int>   <dbl>  <int>    <int>
#> 1 persistent           1     2       0      0        0
#> 2 semi-persistent      1     1       0      0        0

When you are finished, please call terminate() with no arguments to terminate all controllers in the controller group.

group$terminate()