Generating Sequences With Recurrent Neural Networks · HackerPapers discussions

Generating Sequences With Recurrent Neural Networks — mentioned by dharma1

Sat, 08 Feb 2025 11:21:22 GMT

Script fonts don’t really look like handwriting - too regular. But one of the early deep learning papers from Alex Graves does this really well with LSTMs - https://arxiv.org/abs/1308.0850 Implementation - https://www.calligrapher.ai/

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by hm64

Thu, 26 Jan 2023 17:58:22 GMT

A large number of handwriting synthesis demos are derivations of/inspired by Alex Graves's paper "Generating Sequences With Recurrent Neural Networks". Alex's code is available under GPL-3.0 here, https://sourceforge.net/projects/rnnl/. One could also port the techniques described in the paper, https://arxiv.org/abs/1308.0850 using any modern machine learning framework.

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by duckmysick

Thu, 26 Jan 2023 15:23:27 GMT

Looks like it's based on this paper [1]. It's a recurrent neural network trained on online handwriting data. It means you record how the position of the tip of the pen changes as you write. The training data comes from the IAM On-Line Handwriting Database which has only handwritten English text. If you want to support a complete Latin script, you would have to generate a lot of training data yourself, preferably in multiple languages. Quite the effort indeed. 1 - https://arxiv.org/pdf/1308.0850.pdf

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by gfxgirl

Thu, 20 Aug 2020 02:29:46 GMT

From the github project https://arxiv.org/abs/1308.0850 https://github.com/sjvasquez/handwriting-synthesis

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by sjv

Sat, 08 Aug 2020 21:52:25 GMT

This problem was more or less solved with neural networks in 2013, see https://arxiv.org/abs/1308.0850. I've implemented a similar model in javascript that you can interact with: https://seanvasquez.com/handwriting-generation/

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by kastnerkyle

Mon, 19 Nov 2018 20:41:36 GMT

This type of concatenation I first saw in Alex Graves' work on "Generating Sequences With Recurrent Neural Networks", including his unpublished TTS demo [1]. Biasing with part of another sentence (as in handwriting) can possibly improve style in TTS as well. We followed this approach in char2wav [2], but "voice cloning" has come much farther in my opinion [3][4][5]. There's a lot of relevant research on techniques for this beyond concatenating indicators or embeddings, if people are interested in the research side of this technology. [0] https://arxiv.org/abs/1308.0850 [1]…

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by hardmaru

Tue, 17 Apr 2018 21:54:38 GMT

Hi, thanks for the feedback! Honestly we didn't intend to over-hype the results. We took the terms from existing works that we knew: 1) Alex Graves on Hallucination with Recurrent Neural Networks, a 2015 lecture at the University of Oxford from a course by Nando de Freitas (highly recommended). http://www.creativeai.net/posts/kp4bTG993JTQcqy2d/alex-grave... 2) Generating Sequences With Recurrent Neural Networks https://arxiv.org/abs/1308.0850 "Assuming the predictions are probabilistic, novel sequences can be generated from a trained network by iteratively sampling from the network’s output…

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by kastnerkyle

Mon, 24 Apr 2017 18:41:42 GMT

There is an older paper [0] and demo from [1] Alex Graves that inspired a ton of work around handwriting, and then speech. Previous work from Jose Sotelo et. al. (including me) called char2wav [2] is a close neighbor to Graves' approach, though he (Graves) never published the approach for speech so we don't really know. Google's recent Tacotron paper [3] is also a relative to these approaches. WaveNet certainly changed the game in many ways, but approaches to TTS using RNNs have different roots. WaveNet and friends (incl. DeepVoice and NPSS linked elsewhere in this thread) are largely focused…

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by tgflynn

Wed, 04 May 2016 00:54:08 GMT

As far as I can tell LSTM's are still using a combination of logistic and tanh transfer functions, at least through 2014 and some of the most impressive deep learning results have been obtained with these. For example: http://arxiv.org/abs/1308.0850

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by twotwotwo

Sun, 01 May 2016 01:32:49 GMT

Man, the RNN-generated Wikipedia data around page 14 of http://arxiv.org/pdf/1308.0850v5.pdf is fascinating. Some things you'd expect from a modern language model--like, your phone keyboard's predictor already generates some grammatical-looking phrases, and there's only room to do so much in a phone. I'm more impressed that (as the author notes) it gets syntax right, including matched pairs over long distances, occasionally makes words up using plausible components, suffixes, etc. ("quanting"), and learns other quirks (e.g. there's Cyrillic after the [[ru:]] link and some Hangul after the…

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by walrus

Sat, 30 Apr 2016 18:26:00 GMT

It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the 1.33 bpc value was calculated on a held-out validation set and doesn't include the network weights. [1] http://arxiv.org/abs/1308.0850

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by kastnerkyle

Mon, 07 Mar 2016 14:35:17 GMT

It is a dense read, but you might have a look at [1]. This is how attention is implemented in Theano. Basically the key is going "3D" per timestep (where 2D per timestep is the norm when doing minibatch training), then taking a weighted sum over the correct axis to get the right size to combine with the RNN state. Short summary: I: input length M: minibatch size (same for input and output) H_in: input hidden size (arbitrary/user selected) H_out: output hidden size (arbitrary/user selected) C: attention feature size (arbitrary/user selected) Looking at the decode/generator RNN, "context" comes…

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by nrmn

Wed, 23 Dec 2015 19:44:54 GMT

I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then: Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850 Older one, but important to understand deeply since other recent ideas have come from this! Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks - http://arxiv.org/abs/1511.06434 Unitary Evolution Recurrent Neural Networks - http://arxiv.org/abs/1511.06464 State of the Art Control of Atari Games Using Shallow Reinforcement Learning…

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by yankoff

Wed, 23 Dec 2015 17:45:32 GMT

Generating Sequences With Recurrent Neural Networks http://arxiv.org/abs/1308.0850 Generating Text with Recurrent Neural Networks http://www.cs.utoronto.ca/~ilya/pubs/2011/LANG-RNN.pdf Bitcoin whitepaper https://bitcoin.org/bitcoin.pdf Ethereum paper http://gavwood.com/Paper.pdf

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by kastnerkyle

Thu, 06 Aug 2015 21:31:02 GMT

The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3]. There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11]…

Paper and discussion history. A mention is not an endorsement.

Generating Sequences With Recurrent Neural Networks — mentioned by kastnerkyle

Thu, 11 Jun 2015 11:48:00 GMT

If you are into this kind of stuff (time series density/probability modeling), there are several places to go beyond this. The simplest example is the RNN-RBM (restricted boltzmann machine) from the Theano deep learning tutorials [1]. There are follow-up papers by Kratarth Goel extending this to LSTM-DBN, and RNN-NADE (neural autoregressive density estimator) has also been explored in this space. The results for MIDI are pretty good. Ales Graves [2] has done a huge amount of work in sequence modeling for classification and generation. His work with LSTM-GMM for online handwriting modeling can…

Paper and discussion history. A mention is not an endorsement.