% Generated by roxygen2: do not edit by hand
% Please edit documentation in R/misc_llm_verify.R
\name{llm_verify}
\alias{llm_verify}
\title{Have LLM check the result of a prompt (LLM-in-the-loop)}
\usage{
llm_verify(
  prompt,
  question = "Is the answer satisfactory?",
  llm_provider = NULL,
  max_words_feedback = 50
)
}
\arguments{
\item{prompt}{A single string or a \linkS4class{tidyprompt} object}

\item{question}{The question to ask the LLM to verify the result of the prompt.
The LLM will be presented the original prompt, its result, and this question.
The LLM will be asked to provide a boolean answer to this question. If TRUE,
the result of the prompt will be accepted; if FALSE, the result will be declined}

\item{llm_provider}{A \linkS4class{llm_provider} object
which will be used to verify the evaluation led to a satisfactory result.
If not provided, the same LLM provider as the prompt was originally
evaluated with will be used}

\item{max_words_feedback}{The maximum number of words allowed in the summary of
why the result was declined.
This summary is sent back to the LLM originally asked to evaluate the prompt}
}
\value{
A \linkS4class{tidyprompt} with an added \code{\link[=prompt_wrap]{prompt_wrap()}} which
will add a check for a LLM to accept or decline the result of the prompt,
providing feedback if the result is declined
}
\description{
This function will wrap a prompt with a check for a LLM to accept or decline
the result of the prompt, providing feedback if the result is declined.
The evaluating LLM will be presented with the original prompt and the result
of the prompt, and will be asked to verify if the answer is satisfactory
(using chain of thought reasoning to arrive at a boolean decision). If
the result is declined, the chain of thought responsible for the decision
will be summarized and sent back to the original LLM that was asked to
evaluate the prompt, so that it may retry the prompt.

Note that this function is experimental and, because it relies on chain of
thought reasoning by a LLM about the answer of another LLM, it may not
always provide accurate results and can increase the token cost of
evaluating a prompt.
}
\details{
The original prompt text shown to the LLM is built from
the base prompt as well as all prompt wraps that have a modify function
but do not have an extraction or validation function. This is to ensure
that no redundant validation is performed by the evaluating LLM on
instructions which have already been validated by functions in those
prompt wraps.
}
\examples{
\dontrun{
  "What is 'Enschede'?!" |>
    answer_as_text(max_words = 50) |>
    llm_verify() |>
    send_prompt()
#   --- Sending request to LLM provider (gpt-4o-mini): ---
#   What is 'Enschede'?!
#
#   You must provide a text response. The response must be at most 50 words.
#   --- Receiving response from LLM provider: ---
#   Enschede is a city in the Netherlands, located in the eastern part near the German border.
#   It is known for its vibrant culture, history, and universities, particularly the
#   University of Twente, as well as its textiles industry and beautiful parks.
#   --- Sending request to LLM provider (gpt-4o-mini): ---
#   You are given a user's prompt.
#   To answer the user's prompt, you need to think step by step to arrive at a final answer.
#
#   ----- START OF USER'S PROMPT -----
#   >>> An assistant was asked:
#
#   What is 'Enschede'?!
#
#   >>> The assistant answered:
#
#   [1] "Enschede is a city in the Netherlands, located in the eastern part near
#   the German border. It is known for its vibrant culture, history, and universities,
#   particularly the University of Twente, as well as its textiles industry and
#   beautiful parks."
#
#   >>> Is the answer satisfactory?
#
#   You must answer with only TRUE or FALSE (use no other characters).
#   ----- END OF USER'S PROMPT -----
#
#   What are the steps you would take to answer the user's prompt?
#   Describe your thought process in the following format:
#     >> step 1: <step 1 description>
#     >> step 2: <step 2 description>
#     (etc.)
#
#   When you are done, you must type:
#     FINISH[<put here your final answer to the user's prompt>]
#
#   Make sure your final answer follows the logical conclusion of your thought process.
#   --- Receiving response from LLM provider: ---
#   >> step 1: Read the user's prompt carefully to understand what is being asked
#   regarding the assistant's answer about Enschede.
#   >> step 2: Analyze the assistant's provided answer for accuracy and completeness,
#   ensuring it conveys relevant information about Enschede.
#   >> step 3: Assess whether the answer includes key aspects, such as the city's
#   location, cultural significance, educational institutions, and any notable
#   historical references or industries tied to the city.
#   >> step 4: Determine if the assistant's answer sufficiently addresses the
#   question "What is Enschede?" and provides a clear and informative overview of the city.
#   >> step 5: Conclude whether the assistant's answer meets the standards for being
#   satisfactory in terms of detail and correctness.
#
#   FINISH[TRUE]
#   [1] "Enschede is a city in the Netherlands, located in the eastern part near the
#   German border. It is known for its vibrant culture, history, and universities,
#   particularly the University of Twente, as well as its textiles industry and beautiful parks."
}
}
