HackerPapers

arXiv:1902.04094cs.CL

BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

Alex Wang, Kyunghyun Cho

Abstract

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

arXiv comments
NeuralGen 2019; https://colab.research.google.com/drive/1MxKZGtQ9SSBjTK5ArsZ5LKhkztzg52RV
Journal reference
DOI
Subjects
cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 2 threads, from Oct 2025 to Sep 2026. Found in 1 HN submission from Feb 2019 to Feb 2019. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments2 threads

  1. I turned Jev into a (lousy) chatbot 1 comment
  2. BERT is just a single text diffusion step 2 comments

    • Also relevent - https://arxiv.org/pdf/1902.04094

      loubbradView on HNParent context

    • Back when BERT came out, everyone was trying to get it to generate text. These attempts generally didn't work, here's one for reference though: https://arxiv.org/abs/1902.04094 This doesn't have an explicit diffusion tie in, but Savinov et al. at DeepMind figured out that doing…

      thatguysaguyView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.