HackerPapers

arXiv:1609.04938cs.CV

Image-to-Markup Generation with Coarse-to-Fine Attention

Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, Alexander M. Rush

Abstract

We present a neural encoder-decoder model to convert images into presentational markup based on a scalable coarse-to-fine attention mechanism. Our method is evaluated in the context of image-to-LaTeX generation, and we introduce a new dataset of real-world rendered mathematical expressions paired with LaTeX markup. We show that unlike neural OCR techniques using CTC-based models, attention-based approaches can tackle this non-standard OCR task. Our approach outperforms classical mathematical OCR systems by a large margin on in-domain rendered data, and, with pretraining, also performs well on out-of-domain handwritten data. To reduce the inference complexity associated with the attention-based approaches, we introduce a new coarse-to-fine attention layer that selects a support region before applying attention.

arXiv comments
Accepted by ICML 2017
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.CLComputation and Language cs.LGMachine Learning cs.NENeural and Evolutionary Computing

On Hacker News

Linked by 3 identifiable commenters in 3 comments across 3 threads, from May 2017 to Nov 2023.

Comments linking this paper, per year.

Discussed in comments3 threads

  1. Pix2tex: Using a ViT to convert images of equations into LaTeX code 1 comment

    • Want to give proper credit to my former student for starting this: Yuntian Deng et al., 2016 (https://arxiv.org/abs/1609.04938). I believe this repo uses the dataset from that paper. Some recent cool work he's been doing: https://www.youtube.com/watch?v=lx1XcTdhalU.

      srushView on HNParent context

  2. Accessibility Through Semantic HTML 1 comment

    • Are you sure that extracting document structure from a screenshot is a solved task? I'm aware of "What You Get Is What You See" [1], but they are nowhere close to 100% accuracy even for reconstructing the original markup, which does not even require making inferences about the…

      yorwbaView on HNParent context

  3. Pix2code: Generating Code from a GUI Screenshot 1 comment

    • If you somewhat familiar with DL literature, you will see this paper, while having a very interesting angle, the underlying architecture is a standard, enc-dec network, with encoder being CNN and decoding being LSTM. Such application, has been studied before…

      tanilamaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.