HackerPapers

arXiv:1512.08512cs.CV

Visually Indicated Sounds

Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, William T. Freeman

Abstract

Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.

arXiv comments
Journal reference
DOI
Subjects
cs.CVComputer Vision and Pattern Recognition cs.LGMachine Learning cs.SDSound

On Hacker News

Linked by 2 identifiable commenters in 2 comments across 1 thread, from Jun 2016 to Jun 2016. Found in 1 HN submission from Nov 2016 to Nov 2016. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments1 thread

  1. AI produces realistic sounds that fool humans 2 comments

    • Paper is here: http://arxiv.org/abs/1512.08512

      jderickView on HNParent context

    • > But the title makes it seem like the algorithm is synthesizing the sounds from scratch! It does. If you read the paper, they say that first they went with matching sounds from a database, but later turned on to full synthesis. Reference: look for "parametric synthesis" in the…

      visargaView on HNParent context

Submitted to Hacker News

Connected through shared HN threads, not inferred similarity or a recommendation.