LLMs & Prompt Engineering test (Beginner)

Sample questions

5 sample questions from the LLMs & Prompt Engineering test, with the answers. Samples are taken out of the live test, so you will not meet these when you sit it.

1. Which of the following statements accurately describes chain-of-thought prompting?

   1. Chain-of-thought prompting decreases the model's accuracy in logical reasoning tasks.
   2. Chain-of-thought prompting is only used for simple classification tasks.
   3. Chain-of-thought prompting improves multi-step reasoning by asking the model to reason step-by-step before answering.
   4. Chain-of-thought prompting is an alternative to structured output formatting.

   Answer: C. Chain-of-thought prompting improves multi-step reasoning by asking the model to reason step-by-step before answering.

   Chain-of-thought prompting improves multi-step reasoning by asking the model to reason step-by-step before answering. This technique helps the model break down complex problems into manageable steps, improving accuracy in logic and math tasks. The other options either describe unrelated concepts or misrepresent the purpose of chain-of-thought prompting.

2. Which of the following statements accurately describes zero-shot classification?

   1. Zero-shot classification requires the model to be trained on a large number of examples from all possible categories.
   2. Zero-shot classification is a method where the model is trained on a single example per category.
   3. Zero-shot classification only works when the categories are explicitly mentioned in the model's training data.
   4. Zero-shot classification allows the model to classify inputs into categories it has not been explicitly trained on by leveraging its general understanding of language.

   Answer: D. Zero-shot classification allows the model to classify inputs into categories it has not been explicitly trained on by leveraging its general understanding of language.

   Zero-shot classification is a technique where the model is asked to classify inputs into categories it has not been explicitly trained on, relying on its general language understanding. This is different from few-shot classification, which uses a few examples for training. The other options either describe different techniques or are incorrect in their definitions.

3. A developer receives an output from an LLM that is cut mid-sentence, and the stop/finish reason indicates that the output was truncated. What is the most likely cause of this issue?

   1. The prompt was not formatted properly, causing the model to misinterpret the task.
   2. The model's output was cut mid-sentence due to the max_tokens limit being reached.
   3. The API authentication key was invalid, leading to an error in the response.
   4. The tokenizer failed to process the input correctly.

   Answer: B. The model's output was cut mid-sentence due to the max_tokens limit being reached.

   The model's output was cut mid-sentence due to the max_tokens limit being reached. This is a common issue when the output length exceeds the max_tokens parameter, which caps the number of tokens the model can generate. The stop/finish reason in the API response would indicate that the output was truncated. Other options are incorrect because they refer to unrelated issues such as tokenization errors, prompt formatting, or API authentication, which do not directly relate to the output being cut due to the max_tokens limit.

4. Which of the following best describes BLEU?

   1. BLEU is a type of tokenization used in subword tokenization.
   2. BLEU is a technique used to improve model performance by adding more training data.
   3. BLEU (Bilingual Evaluation Understudy) is a metric used to evaluate the quality of machine-generated text by comparing it to one or more reference texts.
   4. BLEU is a method for generating text using a probabilistic model with temperature sampling.

   Answer: C. BLEU (Bilingual Evaluation Understudy) is a metric used to evaluate the quality of machine-generated text by comparing it to one or more reference texts.

   BLEU is a widely used metric for evaluating the performance of natural language generation systems, such as those used in machine translation and chatbots. It measures the overlap between the generated text and reference texts by considering n-gram precision. Other options are incorrect because they describe different concepts or are not accurate descriptions of BLEU.

5. In a project scenario where you are using sampling with temperature to generate responses from a language model, which of the following statements is correct?

   1. Increasing the temperature value makes the model's outputs more deterministic and focused.
   2. Temperature has no impact on the model's output diversity.
   3. Increasing the temperature value reduces the model's ability to generate novel responses.
   4. Increasing the temperature value makes the model's outputs more random and diverse.

   Answer: D. Increasing the temperature value makes the model's outputs more random and diverse.

   When using sampling with temperature in a realistic project scenario, increasing the temperature value makes the model's outputs more random and diverse. This is because a higher temperature introduces more randomness in the sampling process, allowing the model to explore a wider range of possible responses. On the other hand, decreasing the temperature makes the outputs more deterministic and focused on the most likely responses.