Maestro Tag Reference

Maestro tags are roxygen2 comments for configuring the scheduling and execution of pipelines.

Maestro tags follow the format: `#' @maestroTagName value`

Some tags may not take a value.

Maestro tags must be written above the function that is to be included as a pipeline. A typical pipeline with tags could look like this:

    #' @maestroFrequency 1 hour
    #' @maestroStartTime 12:30:00
    #' @maestroLogLevel WARN
    my_pipeline <- function() {

      # Pipeline code
    }

Below are descriptions of all the tags currently available in maestro along with examples.

# Basic Tags

## maestroFrequency

How often to run the pipeline. This tag takes a time unit indicating how long to wait between subsequent runs of the pipeline. Acceptable values include an integer value followed by one of minute(s), hour(s), day(s), week(s), month(s), and year(s). Note that some combinations of integer + unit may be invalid. Adverbs like ‘hourly’, ‘daily’, ‘weekly’, etc. are also valid.

Default: `1 day`

Examples:

- `#' @maestroFrequency 1 hour`

- `#' @maestroFrequency daily`

- `#' @maestroFrequency 2 weeks`

- `#' @maestroFrequency 3 months`

## maestroStartTime

Timestamp, date, or time corresponding to the start time of the pipeline. This also sets the cadence of the pipeline in some cases. For instance, if the start time is `2025-03-18 03:00:00` and the frequency is daily, the pipeline will run at 03:00 every day. A value in the future prevents the pipeline from running until that time has been reached.

For pipelines running less frequently than daily, partial anchor formats are supported to make it easier to express natural cycle points:

- `HH:MM:SS` — time of day, for `minute`, `hour`, or `day` frequencies.
- `Mon HH:MM:SS` — weekday abbreviation + time, for `week` frequencies. Valid abbreviations: `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`, `Sun`.
- `DD HH:MM:SS` or `DD` — month-day (+ optional time), for `month` frequencies.

Default: Current date-time floored to the pipeline’s frequency unit.

Examples:

- `#' @maestroStartTime 2025-02-05 12:00:00`

- `#' @maestroStartTime 2025-01-01`

- `#' @maestroStartTime 08:00:00`

- `#' @maestroStartTime Mon 04:00:00`

- `#' @maestroStartTime Wed 09:30:00`

- `#' @maestroStartTime 15 04:00:00`

- `#' @maestroStartTime 1`

## maestroTz

Timezone in which the maestroStartTime is to be considered. Takes any valid timezone string that can be found in `OlsonNames()`.

Default: `UTC`

Examples:

- `#' @maestroTz Europe/Paris`

- `#' @maestroTz America/Halifax`

- `#' @maestroTz US/Pacific`

## maestroLogLevel

Minimum logging threshold for messages, warnings, and errors that come from the pipeline. These levels correspond to those in `logger:::log_levels_supported` but most commonly used are ERROR, WARN, INFO. For example, if you use `WARN` then any messages of lower urgency (i.e., INFO) will be suppressed, but errors will be logged.

Default: `INFO`

Examples:

- `#' @maestroLogLevel ERROR`

- `#' @maestroLogLevel WARN`

## maestroFlags

Arbitrary labeling tags which are then made accessible via `get_flags()`. A pipeline can have multiple tags separated by spaces.

Default:

Examples

- `#' @maestroFlags critical aviation`

- `#' @maestroFlags time-series`

# Specifier Tags

## maestroHours

Specific hours of the day in which to run the pipeline. This only applies for pipelines that run at an hourly or minutely frequency. Acceptable values are integers from 0-23 separated by spaces. If empty, pipeline runs at all hours. This tag uses the timezone specified by `maestroTz` (will be UTC if empty).

Default:

Examples:

- `#' @maestroHours 1 4 7 10`

- `#' @maestroHours 0 5 20`

## maestroDays

Specific days of week or days of month on which to run the pipeline. This only applies for pipelines that run at a minutely, hourly, or daily frequency. Acceptable values are either integers from 1-31 or day of week strings like Mon, Tue, Wed, etc. These two options cannot be used in combination.

Default:

Examples:

- `#' @maestroDays 1 11 21 31`

- `#' @maestroDays Mon Tue Wed Thu Fri`

## maestroMonths

Specific months of the year on which to run the pipeline. This only applies for pipelines that do run at least monthly. Acceptable values are integers (1-12) corresponding to the month of the year (e.g., 1 = January, 2 = February, etc.).

Default:

Examples:

- `#' @maestroMonths 3 8 12`

- `#' @maestroMonths 10`

# DAG Tags

## maestroInputs

For a DAG style pipeline, the names of pipelines that input into the pipeline. These names must match the function name defining the inputting pipeline. Multiple pipelines can be used as inputs and the input value is used in the target pipeline via the required `.input` parameter. Note that this tag could be redundant if the inputting pipeline uses `maestroOutputs`.

Default:

Examples:

- `#' @maestroInputs extract verify`

## maestroOutputs

For a DAG style pipeline, the names of pipelines that receive the return value of this pipeline as input. These names must match the function name defining the outputting pipeline. Multiple pipelines can be outputted into. The return value of the pipeline will be passed into the receiving pipeline. Note that this tag could be redundant if pipeline to be inputted into uses `maestroInputs`.

Default:

Examples:

- `#' @maestroOutputs transform`

# Advanced Tags

## maestroSkip

Flags a pipeline to never be executed even if it is scheduled to run. This can be useful when developing or testing a pipeline. This tag takes no value, instead the presence of the tag indicates whether it is skipped. This tag is ignored when `run_schedule(..., run_all = TRUE)` or when using `invoke()`.

Default:

Examples:

- `#' @maestroSkip`

## maestroPriority

Determines the order in which pipelines that run at the same scheduled instance are executed. Values are positive integers from 1 to N. Order is determined in descending order such that 1 indicates the highest priority. Pipelines with the same priority run in the order in which `build_schedule()` parses the pipeline (usually alphabetical according to file name and then line number within file). By default, all pipelines are given equal priority.

Default:

Examples:

- `#' @maestroPriority 1`
- `#' @maestroPriority 3`

## maestroRunIf

Sets the pipeline to be conditional based on a TRUE/FALSE value returned from an R expression. R code following the tag must evaluate to a single TRUE or FALSE, where TRUE causes the pipeline to run (assuming it meets scheduling criteria). If the pipeline is part of a DAG it can use the upstream return value as .input. Variables passed through the `resources` argument of `run_schedule()` are also accessible in expression following the tag.

Default:

Examples

- `#' @maestroRunIf sample(c(TRUE, FALSE), 1)`

-     #' @maestroRunIf
      #' rand_bool <- sample(c(TRUE, FALSE), 1)
      #' !rand_bool

-     #' @maestroInputs init_pipeline
      #' @maestroRunIf .input

## maestro

Generic tag for identifying a maestro pipeline with all the defaults. Useful when you want a pipeline to be scheduled via maestro that accepts all default tag values. Only use this tag if you have no other maestro tags. The tag takes no value.

Default:

Examples:

- `#' @maestro`