HackerPapers

arXiv:2306.10763cs.CL

Guiding Language Models of Code with Global Context using Monitors

Lakshya A Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, Sriram K. Rajamani

Abstract

Language models of code (LMs) work well when the surrounding code provides sufficient context. This is not true when it becomes necessary to use types, functionality or APIs defined elsewhere in the repository or a linked library, especially those not seen during training. LMs suffer from limited awareness of such global context and end up hallucinating. Integrated development environments (IDEs) assist developers in understanding repository context using static analysis. We extend this assistance, enjoyed by developers, to LMs. We propose monitor-guided decoding (MGD) where a monitor uses static analysis to guide the decoding. We construct a repository-level dataset PragmaticCode for method-completion in Java and evaluate MGD on it. On models of varying parameter scale, by monitoring for type-consistent object dereferences, MGD consistently improves compilation rates and agreement with ground truth. Further, LMs with fewer parameters, when augmented with MGD, can outperform larger LMs. With MGD, SantaCoder-1.1B achieves better compilation rate and next-identifier match than the much larger text-davinci-003 model. We also conduct a generalizability study to evaluate the ability of MGD to generalize to multiple programming languages (Java, C# and Rust), coding scenarios (e.g., correct number of arguments to method calls), and to enforce richer semantic constraints (e.g., stateful API protocols). Our data and implementation are available at https://github.com/microsoft/monitors4codegen .

arXiv comments
Accepted to NeurIPS 2023 and to appear as "Monitor-Guided Decoding of Code LMs with Static Analysis of Repository Context" at https://neurips.cc/virtual/2023/poster/70362 . Contents: 11 pages, 15 additional pages of appendix, 13 figures, 3 tables
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.AIArtificial Intelligence cs.LGMachine Learning cs.PLProgramming Languages cs.SESoftware Engineering

On Hacker News

Linked by 5 identifiable commenters in 6 comments across 5 threads, from Aug 2023 to May 2025.

Comments linking this paper, per year.

Discussed in comments5 threads

  1. Type-constrained code generation with language models 1 comment

    • Also worth checking out MultiLSPy, effectively a python wrapper around multiple LSPs: https://github.com/microsoft/multilspy Used in multiple similar publications, including "Guiding Language Models of Code with Global Context using Monitors" (https://arxiv.org/abs/2306.10763)…

      _jayhack_View on HNParent context

  2. Show HN: Prompts as WASM Programs 1 comment

    • Each of those is great, and more sophisticated strategies are being developed all the time: incorporating static analysis at inference time to ensure code is more than just syntactically correct [0]; to multi-LLM call agent-based frameworks [1]; decomposing hard tasks into many…

      emrek1View on HNParent context

  3. How to tackle unreliability of coding assistants 1 comment

    • it's been done, in a limited way: https://arxiv.org/abs/2306.10763; we're working on extended this, but yeah, there are a lot of details to work out

      disconcisionView on HNParent context

  4. Hacking Google Bard – From Prompt Injection to Data Exfiltration 1 comment

    • Yeah. E.g. GPT-4-turbo's JSON-mode seems to forcibly block non-JSON-compliant outputs, at least in some way. They document that forgetting to instruct it to emit JSON may lead to producing whitespace until the output length limit is reached. In related info, there is "Guiding…

      namibjView on HNParent context

  5. Show HN: LLMs can generate valid JSON 100% of the time 2 comments

    • Hi, the paper at https://arxiv.org/abs/2306.10763 titled "Guiding Language Models of Code with Global Context using Monitors" shows how to have the language models generate code without hallucinated dereferences.

      LakshyAAAgrawalView on HNParent context

    • In our experience, at least for code generation, the experience has been that base models can be improved significantly by guiding token level generation. In our paper titled "Guiding Language Models of Code with Global Context using Monitors" (https://arxiv.org/abs/2306.10763)…

      LakshyAAAgrawalView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.