---
title: "STF Testing Frameworks"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{STF Testing Frameworks}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

```{r load, echo=FALSE}
library(test.assessr)
```
The `test.assessr` package checks package structure and test drivers to detect and classify testing configurations.

If it detects a standard testing framework (STF) (i.e. `testthat` or `testit`),
it runs a standard testing workflow involving a number of steps:

 `Step 1`
 
  - run `callr::r_safe` which calls `covr::package_coverage`. 
 
  - The `r_safe` function has the parameter `error = "error"`. 
 
  - This means if `package_coverage` encounters an error, 
 then `r_safe` returns an error to the main testing process.

  - If `Step 1` is successful, 
 then a unit test data coverage object like this is produced:


`Step 1 unit test data coverage object`

| R object | field | element 1 | element 2 | value |
| -------- | ----- | --------- | --------- | ------|
| test_package |pkg_name | | | "test.package.0001" |
| |  pkg_ver | | | "0.1.0" |
| | date_time | | | "2026-03-09 14:27:36.673915" |
| | executor | | | "xxxxxxx" |
| | sysname | | | "Linux" |
| | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec  1 05:36:50 UTC 2025" |
| | release | | | "6.1.158-180.294.amzn2023.x86_64" |
| | machine | | | "x86_64" |
| | r_version | | |"4.4.1" |
| | test_framework_type | | | standard testing framework |
| | total_cov | | | 1 |
| | res_cov | name| |"test.package.0001" |
| | | coverage | filecoverage | R/myscript.R 100 | 
| | | |      totalcoverage | 100 |
| | errors | | |NA |
| | notes  | | |NA |

`Step 2`
 
  - run `setup_covr_env` which 
   
    - loads helper files from `tests/testthat`,
    - loads package namespace objects into `cov_env`,
    - resolves the project root,
    - loads package datasets into `cov_env`,
    - loads Base R datasets and exports into `cov_env`,
    - optionally loads `nycflights13` datasets if available, 
    - loads dependencies into `cov_env`, and
    - unloads the package namespace afterward to reduce conflicts.
 
  - run `create fail reporter` to find failing tests
  - run `create test reporter` to create a reporter for all tests
  - map `exported functions` to test files
  - create a filter for test status i.e. tests that have `"FAILED", "ERROR", "SKIPPED"`
  - identify tests to be skipped based on status
  - comment out tests to be skipped
  - run `covr::environment_coverage` on remaining test files.

  - If `Step 2` is successful, 
 then a unit test data coverage object like this is produced:



`Step 2 unit test data coverage object`

| R object | field | element 1 | element 2 | value |
| -------- | ----- | --------- | --------- | ------|
| test_package |pkg_name | | | "test.package.0003" |
| |  pkg_ver | | | "0.1.0" |
| | date_time | | | "2026-04-09 10:17:27.71564" |
| | executor | | | "xxxxxxx" |
| | sysname | | | "Linux" |
| | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec  1 05:36:50 UTC 2025" |
| | release | | | "6.1.158-180.294.amzn2023.x86_64" |
| | machine | | | "x86_64" |
| | r_version | | |"4.4.1" |
| | test_framework_type | | | standard testing framework |
| | total_cov | | | 0 |
| | res_cov | name| |"test.package.0003" |
| | | coverage | filecoverage | num [1(1d)] 0 
| | | | | attr(*, "dimnames")=List of 1 chr "myscript.R" | 
| | | |       totalcoverage | 0 |
| | errors | | |NA |
| | notes  | | |NA |
| long_summary        |  file | | |      "test-myscript.R#L2_L2" |
| | code_script | | | "myscript" |
| | context     | | | "this works" |
| |  status     | | | "FAILED" |
| | n           | | | 1 |
| | time        | | | 0.291 |
|   test_skip         |  file | | |  "test-myscript.R" |
| | code_script  | | | "myscript" |
| | context   | | |  "this works" |
| | status     | | | "FAILED" |
| | n          | | |  1 |
| | time       | | | 0.291 |
| | expectation: | | |  "expect_equal" |
| | line1      | | | 2 |
| | line2      | | | 2 |

This data object differs from `Step 1 data object` in that it gives information 
about which tests/test blocks were skipped and the status tells the user why 
they were skipped.

`Step 3`
 
  - run `withr::with_dir` which calls `covr::package_coverage`. 
 
  - `covr::package_coverage` has the parameters `type = "tests", quiet = TRUE`

  - If `Step 3` is successful, 
 then a unit test data coverage object like this is produced:


`Step 3 unit test data coverage object`

| R object | field | element 1 | element 2 | value |
| -------- | ----- | --------- | --------- | ------|
| test_package |pkg_name | | | "dplyr" |
| |  pkg_ver | | | "1.1.4" |
| | date_time | | | "2026-04-09 10:17:27.71564" |
| | executor | | | "xxxxxxx" |
| | sysname | | | "Linux" |
| | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec  1 05:36:50 UTC 2025" |
| | release | | | "6.1.158-180.294.amzn2023.x86_64" |
| | machine | | | "x86_64" |
| | r_version | | |"4.4.1" |
| | test_framework_type | | | standard testing framework |
| | total_cov | | | 0 |
| | res_cov | name| |"dplyr" |
| | | coverage | filecoverage | num [1:109(1d)] 98.3 76.7 97.5 100 100 ...
| | | | | attr(*, "dimnames")=List of 1 [1:109] "across.R" "all-equal.R" "arrange.R" "bind-cols.R" ... | 
| | | |       totalcoverage | 0.907 |
| | errors | | |NA |
| | notes  | | |NA |
| long_summary        |  file | | |      [1:3432] "test-across.R#L7_L7" "test-across.R#L14_L14" "test-across.R#L20_L20" "test-across.R#L24_L24" ... |
| | code_script | | | [1:3432] "across" "across" "across" "across" ... |
| | context     | | | [1:3432] "across() works on one column data.frame" "across() does not select grouping variables" "across() corre" |
| |  status     | | | [1:3432] "PASS" "PASS" "PASS" "PASS" ... |
| | n           | | | [1:3432] 1 1 1 1 1 1 1 1 1 1 ... |
| | time        | | | [1:3432] 0.041 0.014 0.077 0.077 0.077 ...|
|   test_skip         |  file | | |  [1:448] "test-across.R" "test-across.R" "test-across.R" "test-across.R" ... |
| | code_script  | | | [1:448] "across" "across" "across" "across" ... |
| | context   | | |  "across() gives meaningful messages" "across() gives meaningful messages" "across() gives meaningful mes" |
| | status     | | | [1:448] "FAILED" "FAILED" "FAILED" "FAILED" ... |
| | n          | | | [1:448] 1 1 1 1 1 1 1 1 1 1 ... |
| | time       | | | [1:448] 2.03 2.03 2.03 2.03 2.03 ... |
| | expectation| | |  [1:448] "expect_snapshot" "expect_error" "expect_error" "expect_error" ... |
| | line1      | | | [1:448] 209 212 216 222 226 230 234 238 243 244 ... |
| | line2      | | | [1:448] 280 215 219 225 229 233 237 241 243 244 ... |

Both `Step 2 unit test data coverage object` and `Step 3 unit test data coverage object` give the user
details about the status of the tests `FAILED`, `PASSED` and `SKIPPED`.

They also contain the test block start line (i.e. `line1` and `line2`) 
and the expectation type in the `expectation` column (e.g. `expect_snapshot` `expect_error`).