---
title: "Building custom requests"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Building custom requests}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  markdown: 
    wrap: sentence
---

```{r}
#| label: setup
#| include: false

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = TRUE,
  error = FALSE,
  fig.path = "static"
)
options(max.print = 50)

vcr::setup_knitr(prefix = "cr-")
nettskjemar:::mock_if_no_auth()
warning(nettskjemar::ns_has_auth())
```

This package focuses on retrieving information from the Nettskjema API, primarily regarding collected data and meta-data for forms.
There are, however, several endpoints (functionalities) of the API that are currently not covered here.
These most often pertain to sending data _to_ the API, either for creating new forms, updating meta-data or even submitting submissions.

In order to facilitate users potential wish to connect to these endpoints too, the base function `ns_req()` is available to build the query you are after.
This vignette will in short show how you can combine this function, with functionality from the [httr2](https://httr2.r-lib.org/index.html) package to create custom requests to the Nettskjema API.

## Understanding the basics of APIs

APIs are a way for computers to talk to each other.
They are often used to allow different software systems to interact and share data.
APIs are built around the concept of "endpoints", which are specific URLs that correspond to different functionalities or resources.
Each endpoint is associated with a specific HTTP method, which defines the type of action to be performed.

The most common HTTP methods are:
- `GET`: Retrieve data from the server.
- `POST`: Send data to the server to create a new resource.
- `PUT`: Update an existing resource on the server.
- `PATCH`: Partially update an existing resource on the server.
- `DELETE`: Remove a resource from the server.

When making a request to an API, you typically need to provide:
- The endpoint URL.
- The HTTP method.
- Any required headers (e.g., for authentication).
- Any required parameters (e.g., query parameters or body data).
- The format of the response (e.g., JSON, XML).

Before you you start setting up a custom request, you should familiarize yourself with the [Nettskjema API documentation](https://nettskjema.no/apidoc).
In particular, you should try the endpoint you are interested in, by authenticating in the API docs and using the "Try it out" functionality.
This will give you a good idea of how the endpoint works, what parameters it requires, and what kind of response you can expect.

To authenticate, you will need to have an access token.
The access tokens from Nettskjema only last for 24 hours, so if you are doing a lot of testing, you will need to generate new tokens often.
You can get a token by running the `ns_get_token()` function.

```{r}
#| label: get-token
#| eval: false

library(nettskjemar)
ns_get_token()
```

## Setting up a custom GET request

Let's assume you want to update the settings for a form.
To do this, we can first retrieve the settings as they are stored, alter the setting we want to alter, then send that back to Nettskjema.

In the API docs, the URL for getting Settings for a form is shown as `/api/v3/form/{formId}/settings`, which is the URL we need to build to access this information.
The `ns_req()` function, will build the URL up to including `/api/v3` and we need to add the rest.

The `ns_req()` function looks like this

```r
ns_req <- function(...) {
  httr2::request("https://nettskjema.no/api/v3/") |>
    ns_req_auth(...)
}
```

where `ns_auth()` is sets up the authentication client as described in the [Authentication vignette](authentication.html). 
Meaning if we use it, the important parts that make sure you connect to the correct API with the correct credentials.

```{r}
#| label: req

library(nettskjemar)
ns_req()
```

As you can see, the output here shows the URL until the `v3` section, and also that there is Authorization added, which the function does in the background for you with the authentication information you have set up as shown in the [Authentication vignette](authentication.html).

From there, we need to use functions from {httr2} to build the remaining url.

```{r}
#| label: req-2

library(httr2)
formid <- 123823

ns_req() |>
  req_url_path_append("form", formid, "settings")
```

Now, you can see that the path is as requested in the API documentation.
But the request has not been submitted yet!
So far, we have just been creating the command we want to send to the API.
To send the request, we need to use the `req_perform` function from httr2.

```{r}
#| label: settings
#| cassette: settings

response <- ns_req() |>
  req_url_path_append("form", formid, "settings") |>
  req_perform()

response
```

In this case, we also saved the output from the request to a variable in our environment, so we can work with what we received, without having to send the request more times. 

The `response` object here shows stats `200` which is the standard response when a request has been fulfilled without error.
If you receive errors, httr2 will show this clearly, and you will need to debug what is wrong.

Now that we have the response, we need to "unpack" it, so we can get the data that was sent with the response, which are all the settings for the form.
We use httr2 functions to do this.
In general, most of the Nettskjema API responses come as `json` in the "body" of the response (this is API lingo, just go with it.).

```{r}
#| label: req-body

settings_data <- resp_body_json(response)
settings_data
```

Using the `resp_body_json()` function from httr2, will automatically unpack the `json` into a R list-format, so we can more easily work with it.

There are endpoints in the Nettskjema API that will not send `json`, for instance the ones where you retrieve _files_ from the server (excel, SPSS, PDF), these come in _binary_ form.
While, this package contains functions to retrieve those specific files for you, we mention this because you will need to pay attention to the format of the response to know how to unpack the data correctly.

## Setting up a custom POST request

So far, we have focused on `GET` requests, which retrieve information from the API.
A `POST` request sends information _to_ the API for storing, and is a little more work.
This is because `POST` (or `PUT` or `PATCH`) also send information in the "body" of the request, with the information you want stored.
You will need to know what format this information should have, and how to set it up to make it work correctly.
This can require a lot of trial and error to get working correctly.

Since we now have the full settings for a form, let us change a setting so we can see how to achieve this.
There is an element that controls if the user can postpone the survey as they are answering it.

```{r}
#| label: req-postpone

settings_data$postponable
```

For this form, it's set to `FALSE` because its just a simple test form I didn't turn this feature on. 
We will turn it on by setting the value to `TRUE`.

There are several things we need to pay particular attention to in the documentation when we send these types of requests:

1. The METHOD (POST, PUT, PATCH)
1. The URL endpoint
1. The information we need to send


For altering settings, these are:

1. PATCH
1. /api/v3/form/{formId}/settings
1. `additionalProperty = string`

The first two bits were easy enough to figure out, but the last one there might require some trial and error.
I'm not entirely sure what it's telling me the body contents should be, so I'll be doing some assumptions.
Since its a `PATCH` method, and the docs is not explicitly saying we need to send all settings, I'm hoping I can send just the property I want to alter in a list (which httr2 will help me turn into a json that API expects).
Since the URL endpoints is the same as when getting the settings, what we need to do it use the same information, but change the method with httr2.

```{r}
#| label: req-3

ns_req() |>
  req_url_path_append("form", formid, "settings") |>
  req_method("PATCH")
```

Ok, we can see now that the method is correct, the URL is correct, but the body is empty. 

```{r}
#| label: req-4

ns_req() |>
  req_url_path_append("form", formid, "settings") |>
  req_method("PATCH") |>
  req_body_json(
    list(
      postponable = TRUE
    )
  )
```

Things look kind of correct now, we're still not sure about the body setup, but we will send the request and see what happens.

```{r}
#| label: patch-err
#| error: true
#| cassette: true

ns_req() |>
  req_url_path_append("form", formid, "settings") |>
  req_method("PATCH") |>
  req_body_json(
    list(
      postponable = TRUE
    )
  ) |>
  req_perform()
```

That didn't work, but why?
We've not got any further error message, but there might be more information in the body of the response, even if it erred.
Erring API calls like these are actually valid proper responses, so we can explore them just like normal responses.
If you didn't store the response, like I did, you can still explore it without doing the call again.
httr2 has functionality to capture the last response even after the fact!

```{r}
#| label: last-response
#| eval: false

last_response() |>
  resp_body_json()
```
 
```
 #> $message
#> [1] "Valideringsfeil"
#> 
#> $errors
#> $errors$postponable
#> $errors$postponable[[1]]
#> [1] "Mellomlagring er bare tillatt for skjemaer med FEIDE-respondenter, er satt til å samle inn personopplysninger og som leverer til Nettskjema"
#> 
#> 
#> 
#> $confirmForceCleanupErrors
#> named list()
```

As we can see, we get a nice clear message about what has erred.
The form I'm trying to change doesn't meet the pre-requisites for allowing postponing, so we cannot alter that setting!

We can try changing another setting in stead, but we have actually already verified that our calls to the API is working as expected.
We can try altering two settings at once, and see what happens.


```{r}
#| label: settings-change
#| cassette: true

# Show old settings
settings_data$personalDataPurposeDescription
settings_data$shouldHideProgressBar

# Set new settings
ns_req() |>
  req_url_path_append("form", formid, "settings") |>
  req_method("PATCH") |>
  req_body_json(
    list(
      personalDataPurposeDescription = "Testing the API",
      shouldHideProgressBar = TRUE
    )
  ) |>
  req_perform()
```

Now we are cooking, we are getting a 204 back, which is a positive [status code](https://www.webcron.org/en/http-codes) from an API.
The response also says there is no content, and nothing in the body, so there is nothing more to inspect here.

Lets retrieve the settings again and see if we can verify our selves that the settings have indeed changed

```{r}
#| label: settings-check
#| cassette: true

settings_new <- ns_req() |>
  req_url_path_append("form", formid, "settings") |>
  req_perform() |>
  resp_body_json()

settings_new$personalDataPurposeDescription
settings_new$shouldHideProgressBar

```


We've now confirmed we have altered some settings in the test form.
Hopefully, this gives an idea of how you can build more custom requests to the API.

```{r}
#| label: settings-reset
#| include: false
#| cassette: true

# Set new settings
ns_req() |>
  req_url_path_append("form", formid, "settings") |>
  req_method("PATCH") |>
  req_body_json(
    list(
      personalDataPurposeDescription = "Form for testing the API",
      shouldHideProgressBar = FALSE
    )
  ) |>
  req_perform()
```
