Debugging targets pipelines

Will Landau

Why targets?

• Manage computationally demanding work in R:
  • Bayesian data analysis: JAGS, Stan, NIMBLE, greta
  • Deep learning: keras, tensorflow, torch
  • Machine learning: tidymodels
  • PK/PD: nlmixr, mrgsolve
  • Clinical trial simulation: rpact, Mediana
  • Statistical genomics
  • Social network analysis
  • Permutation tests
  • Database queries: DBI
  • Big data ETL

Typical notebook-based project

Messy reality: managing data

Messy reality: managing change

Pipeline tools

• Orchestrate moving parts.
• Scale the computation.
• Manage output data.

targets

• Designed for R.
• Encourages good programming habits.
• Automatic dependency detection.
• Behind-the-scenes data management.
• Distributed computing.

Resources

• Get started in four minutes: https://vimeo.com/700982360
• Example project: https://github.com/wlandau/targets-four-minutes
• Documentation website: https://docs.ropensci.org/targets/
• User manual: https://books.ropensci.org/targets/
• Debugging guide: https://books.ropensci.org/targets/debugging.html

Extensions to {targets}

• Ecosystem of packages to support literate programming, Bayesian data analysis, etc. in targets.
• Compatible with other tools such as renv, Quarto, R Markdown, Shiny, pins, and vetiver.

Live demo

• Code: https://github.com/wlandau/targets-four-minutes
• Helpful practice:
  1. Get to know R/functions.R.
  2. Start setting up a pipeline with use_targets().
  3. Customize the pipeline in _targets.R.
  4. Use tar_visnetwork(), tar_make(), etc. to use the pipeline.

Debugging: challenges

• R code is easiest to debug in the interactive console.
• To ensure reproducibility and to manage heavy computation, a pipeline is automated and non-interactive.
  • External callr::r() process
  • Data management
  • Environment management
  • High-performance computing
  • Error handling

Debugging: techniques

• Finish the pipeline anyway.
• Inspect error messages.
• Debug functions.
• Check for system issues.
• Pause the pipeline with browser().
• Pause the pipeline with the targets debug option.
• Save a targets workspace.

Live demo

• Code: https://github.com/wlandau/targets-debug
• Interactive tutorial scripts:
  1. demo_small.R
  2. demo_browser.R
  3. demo_workspace.R