# Long Form Factuality

- Page: https://shared.btbytes.com/p/long-form-factuality
- Date: 2024-03-28
- Tags: llm

Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics.

This paper by Google Deepmind -- ([Wei et al. 2024](https://arxiv.org/abs/2403.18802)) -- proposes a method which they call **Search-Augmented Factuality Evaluator (SAFE)**. SAFE utilizes an LLM to break down a long-form response into a set of individual facts and to evaluate the accuracy of each fact using a multi-step reasoning process comprising sending search queries to Google Search and determining whether a fact is supported by the search results. Furthermore, they propose extending F1 score as an aggregated metric for long-form factuality. To do so, they balance the percentage of supported facts in a response (precision) with the percentage of provided facts relative to a hyper-parameter representing a user's preferred response length (recall).

---

See also: [TruLens for LLMs](https://www.trulens.org/)

> Create credible and powerful LLM apps, faster. TruLens is a software tool that helps you to objectively **measure the quality and effectiveness of your LLM-based applications using feedback functions**. Feedback functions help to programmatically evaluate the quality of inputs, outputs, and intermediate results, so that you can expedite and scale up experiment evaluation. Use it for a wide variety of use cases including question answering, summarization, retrieval-augmented generation, and agent-based applications.

## References

- Wei, Yang, Song, Lu, Hu, Tran, Peng, Liu, Huang, Du, Le (2024). [Long-Form Factuality in Large Language Models](https://arxiv.org/abs/2403.18802).