HackerPapers

arXiv:1308.0850cs.NE

Generating Sequences With Recurrent Neural Networks

Alex Graves

Abstract

This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.

arXiv comments
Thanks to Peng Liu and Sergey Zyrianov for various corrections
Journal reference
DOI
Subjects
cs.NENeural and Evolutionary Computing cs.CLComputation and Language

On Hacker News

Linked by 12 identifiable commenters in 16 comments across 12 threads, from Jun 2015 to Feb 2025.

Comments linking this paper, per year.

Discussed in comments12 threads

  1. Ghostwriter – use the reMarkable2 as an interface to vision-LLMs 1 comment

    • Script fonts don’t really look like handwriting - too regular. But one of the early deep learning papers from Alex Graves does this really well with LSTMs - https://arxiv.org/abs/1308.0850 Implementation - https://www.calligrapher.ai/

      dharma1View on HNParent context

  2. Realistic computer-generated handwriting 2 comments

    • Looks like it's based on this paper [1]. It's a recurrent neural network trained on online handwriting data. It means you record how the position of the tip of the pen changes as you write. The training data comes from the IAM On-Line Handwriting Database which has only…

      duckmysickView on HNParent context

    • A large number of handwriting synthesis demos are derivations of/inspired by Alex Graves's paper "Generating Sequences With Recurrent Neural Networks". Alex's code is available under GPL-3.0 here, https://sourceforge.net/projects/rnnl/. One could also port the techniques…

      hm64View on HNParent context

  3. Show HN: Handwritten.js – Convert typed text to realistic handwriting 1 comment
  4. An attempt to make a font look more handwritten 1 comment

    • This problem was more or less solved with neural networks in 2013, see https://arxiv.org/abs/1308.0850. I've implemented a similar model in javascript that you can interact with: https://seanvasquez.com/handwriting-generation/

      sjvView on HNParent context

  5. Varying Speaking Styles with Neural Text-to-Speech 1 comment

    • This type of concatenation I first saw in Alex Graves' work on "Generating Sequences With Recurrent Neural Networks", including his unpublished TTS demo [1]. Biasing with part of another sentence (as in handwriting) can possibly improve style in TTS as well. We followed this…

      kastnerkyleView on HNParent context

  6. A step-by-step guide to the “World Models” AI paper 1 comment

    • Hi, thanks for the feedback! Honestly we didn't intend to over-hype the results. We took the terms from existing works that we knew: 1) Alex Graves on Hallucination with Recurrent Neural Networks, a 2015 lecture at the University of Oxford from a course by Nando de Freitas…

      hardmaruView on HNParent context

  7. Lyrebird – An API to copy the voice of anyone 1 comment

    • There is an older paper [0] and demo from [1] Alex Graves that inspired a ton of work around handwriting, and then speech. Previous work from Jose Sotelo et. al. (including me) called char2wav [2] is a close neighbor to Graves' approach, though he (Graves) never published the…

      kastnerkyleView on HNParent context

  8. Neural Networks Are Impressively Good at Compression 3 comments

    • It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the 1.33 bpc value was calculated on a held-out validation set and doesn't include the network weights. [1]…

      walrusView on HNParent context

    • Man, the RNN-generated Wikipedia data around page 14 of http://arxiv.org/pdf/1308.0850v5.pdf is fascinating. Some things you'd expect from a modern language model--like, your phone keyboard's predictor already generates some grammatical-looking phrases, and there's only room to…

      twotwotwoView on HNParent context

    • As far as I can tell LSTM's are still using a combination of logistic and tanh transfer functions, at least through 2014 and some of the most impressive deep learning results have been obtained with these. For example: http://arxiv.org/abs/1308.0850

      tgflynnView on HNParent context

  9. Dynamic Memory Networks for Visual and Textual Question Answering 1 comment

    • It is a dense read, but you might have a look at [1]. This is how attention is implemented in Theano. Basically the key is going "3D" per timestep (where 2D per timestep is the norm when doing minibatch training), then taking a weighted sum over the correct axis to get the right…

      kastnerkyleView on HNParent context

  10. Ask HN: Papers you read in 2015? 2 comments

    • Generating Sequences With Recurrent Neural Networks http://arxiv.org/abs/1308.0850 Generating Text with Recurrent Neural Networks http://www.cs.utoronto.ca/~ilya/pubs/2011/LANG-RNN.pdf Bitcoin whitepaper https://bitcoin.org/bitcoin.pdf Ethereum paper http://gavwood.com/Paper.pdf

      yankoffView on HNParent context

    • I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then: Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850 Older one, but important to understand deeply since other recent ideas…

      nrmnView on HNParent context

  11. A Beginner’s Guide to Restricted Boltzmann Machines 1 comment

    • The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs…

      kastnerkyleView on HNParent context

  12. Extending “Let It Go” with recurrent neural network 1 comment

    • If you are into this kind of stuff (time series density/probability modeling), there are several places to go beyond this. The simplest example is the RNN-RBM (restricted boltzmann machine) from the Theano deep learning tutorials [1]. There are follow-up papers by Kratarth Goel…

      kastnerkyleView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.