HackerPapers

arXiv:2307.09702cs.CL

Efficient Guided Generation for Large Language Models

Brandon T. Willard, Rémi Louf

Abstract

In this article we show how the problem of neural text generation can be constructively reformulated in terms of transitions between the states of a finite-state machine. This framework leads to an efficient approach to guiding text generation with regular expressions and context-free grammars by allowing the construction of an index over a language model's vocabulary. The approach is model agnostic, allows one to enforce domain-specific knowledge and constraints, and enables the construction of reliable interfaces by guaranteeing the structure of the generated text. It adds little overhead to the token sequence generation process and significantly outperforms existing solutions. An implementation is provided in the open source Python library Outlines

arXiv comments
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 4 identifiable commenters in 5 comments across 4 threads, from Jul 2023 to Jan 2026. Found in 1 HN submission from Aug 2023 to Aug 2023. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments4 threads

  1. LLM Structured Outputs Handbook 1 comment

    • This is a fantastic guide! I did a lot of work on structured generation for my PhD. Here are a few other pointers for people who might be interested: Some libraries: - Outlines, a nice library for structured generation - https://github.com/dottxt-ai/outlines - Guidance (already…

      mcycView on HNParent context

  2. Mistral Fine-Tune 1 comment

    • Looks like it actually changes how you do token generation to conform to a given context-free grammar. It's a way to structure how you sample from the model rather than a tweak to the prompt, so it's more efficient and guarantees that the output matches the formal grammar.…

      tikhonjView on HNParent context

  3. Show HN: LLMs can generate valid JSON 100% of the time 2 comments

    • Figure 2 in our paper (https://arxiv.org/abs/2307.09702) shows the difference for a single regex.

      remiloufView on HNParent context

    • Figure 2 in our paper (https://arxiv.org/abs/2307.09702) shows the difference between guidance and outlines to generate a sequence that is valid to a regex. Jsonformer uses the same technique as guidance. Extrapolate this to several fields. Note that we still need to manage the…

      remiloufView on HNParent context

  4. Llama: Add grammar-based sampling 1 comment

Submitted to Hacker News

  • Show HN: LLMs can generate valid JSON 100% of the time854 points303 comments

    Outlines is a Python library that focuses on text generation with large language models. Brandon and I are not LLM experts and started the project a few months ago because we wanted to understand better how the generation process works. Our original background is probabilistic, relational and symbolic programming. Recently we came up with a fast way to generate text that matches a regex (https://blog.normalcomputing.ai/posts/2023-07-27-regex-guide...). The basic idea is simple: regular…

Connected through shared HN threads, not inferred similarity or a recommendation.