arXiv:1706.03762cs.CL2017-06-12

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

arXiv comments: 15 pages, 5 figures
Journal reference
DOI
Subjects: cs.CLComputation and Language cs.LGMachine Learning

On Hacker News

Linked by 101 identifiable commenters in 108 comments across 105 threads, from Jun 2017 to Sep 2026. Found in 21 HN submissions from Jun 2017 to Jul 2025. Open submission discussions.

Comments linking this paper, per year.

Discussed in comments105 threads

The AI Race Just Got Awkward Sep 20261 comment
- >The new game in town is adopting Chinese labs’ advances. Note how I call this adoption instead of the more vitriol-infused “stealing” that Anthropic tends to use. I mean, there's a pretty big difference between labs publishing their research openly and a competitor utilizing it…
  
  nater5000Sep 30, 2026View on HN Parent context
Attention is all you have Sep 20261 comment
- Same, and I thought it's a clever play on words on the classic "Attention is all you need" - https://arxiv.org/abs/1706.03762 - which led us to what we have now
  
  zsergeSep 22, 2026View on HN Parent context
Gemini 3.8 Flash and 3.8 Flash Cyber Sep 20261 comment
- Correct, that's not how LLMs work [0]. [0] - https://arxiv.org/abs/1706.03762
  
  snet0Sep 2, 2026View on HN Parent context
Apple sues OpenAI, accuses ex-employees of stealing trade secrets Jul 20261 comment
- I'm sorry, but I'm not sure what works best for a beginner. I started my PhD when the original Transformers paper [1] had just been released. I had no background in NLP whatsoever and used the original paper to write a Transformer and the full training pipeline from scratch…
  
  nsagentJul 11, 2026View on HN Parent context
30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format Jul 20262 comments
- https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122…
  
  gooobJul 7, 2026View on HN Parent context
- No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks…
  
  cute_boiJul 7, 2026View on HN Parent context
ArXiv's Next Chapter Jul 20261 comment
- https://www.alphaxiv.org/ is a nice place to browse, search for, and read ArXiv papers which have optional AI summaries and chat. If you like one paper, you can get a list of similar papers. To view a specific paper, just take original link and change "arxiv" --> "alphaxiv". For…
  
  KludgeShySirJul 1, 2026View on HN Parent context
Ask HN: What happened to the guy that invented LLMs? Jun 20261 comment
- There's actually eight authors of that paper, if ChatGPT is accurate: - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones and Lukasz Kaiser stayed on at Google - Aidan N. Gomez became a researcher - Illia Polosukhin co-founded https://near.ai…
  
  benoauJun 7, 2026View on HN Parent context
Ask HN: A Brief History of LLMs Jun 20261 comment
- Below is the "Attention is all you need" paper. Transformers and their attention mechanism was the major breakthrough for modern LLMs. ML has been around for a long time, I'd suggest joining kaggle or something and learn by doing. You'll retain more and realize how broad the…
  
  lyfeninjaJun 3, 2026View on HN Parent context
Show HN: udoc. Dependency-free document extraction in Rust May 20261 comment
- I built udoc because most document extraction tools I've used require significant dependencies, only handle one format, or have restrictive licenses. I wanted a single binary that reads PDFs, Office docs (including legacy .doc/.xls/.ppt), ODF, and RTF — with no external parsers…
  
  newelhMay 20, 2026View on HN Parent context
VS Code inserting 'Co-Authored-by Copilot' into commits regardless of usage May 20261 comment
- Maybe read up on how transformers, their encoders and decoders, and the attention matrix works? https://arxiv.org/abs/1706.03762
  
  cookiengineerMay 3, 2026View on HN Parent context
DeepSeek V4 – almost on the frontier May 20261 comment
- And the credit of OpenAI is to Google? https://arxiv.org/abs/1706.03762
  
  johnbarronMay 2, 2026View on HN Parent context
Claude for Creative Work Apr 20261 comment
- The transformer paper was 9 years ago. 9 years between barely translating alright between two very closely related languages (English and French, huge fraction of shared words because of William the conqueror and cultural proximity etc etc) and what we have now. The thing is…
  
  make3Apr 29, 2026View on HN Parent context
The Moat or the Commons Apr 20261 comment
- > Superior architectures will leak pretty quickly via engineers. I agree with the outcome of your premise (i.e., openness), but for different reasons: First, isn't it the case that these bleeding edge 'newfangled' LLMs are basically variations on the same core ideas from…
  
  ninjagooApr 28, 2026View on HN Parent context
Five years of running a systems reading group at Microsoft Mar 20261 comment
- Interestingly, the person at Microsoft states in a reply that even most of them have to pick this up in their spare time. Judging from other replies, it seems that there are quite some differences in how companies approach this. What I meant with "menial coding" is those jobs…
  
  smokelMar 23, 2026View on HN Parent context
Ask HN: What are the greatest discoveries in the last few years? Mar 20261 comment
- No particular order but; 1. Transformers I think they need to be looked into more. They just work too well. I feel like they have a secret within that is yet to be known. [1] https://arxiv.org/abs/1706.03762 2. The 2016 Noble Prize in Medicine Autophagy has such interesting…
  
  max_Mar 21, 2026View on HN Parent context
Don't post generated/AI-edited comments. HN is for conversation between humans. Mar 20261 comment
- LLMs were -in part- designed as translation tools. It's one thing they do really really well. https://arxiv.org/html/1706.03762v7 (Attention is all you need) "Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable…
  
  Kim_BruningMar 11, 2026View on HN Parent context
How will OpenAI compete? Feb 20261 comment
- > i'm just so surprised they'd use chatgpt to do this, when it's quite as easily (and perhaps faster) to use google translate. https://arxiv.org/abs/1706.03762
  
  rrr_oh_manFeb 26, 2026View on HN Parent context
T5Gemma 2: The next generation of encoder-decoder models Dec 20251 comment
- Hi, I'm not on the t5 Gemma team but work on gemma in general. Encoder Decoder comes from the original transformers implementation way back in 2017. If you look at figure 1 you'll see what the first transformer ever looked like. Since that time different implementations of…
  
  canyon289Dec 18, 2025View on HN Parent context
LLMs can get "brain rot" Oct 20251 comment
- In case anyone missed the reference: https://arxiv.org/abs/1706.03762 > (...) We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
  
  dormentoOct 21, 2025View on HN Parent context
The great software quality collapse or, how we normalized catastrophe Oct 20251 comment
- > You're missing the point. In the past bad writing was just bad writing, and it was typically easy to detect. If AI generated text were well written, would it matter to you? Is it bad to use Grammarly? I don't see anything inherently wrong with using AI tools to write, as long…
  
  hodgesrmOct 9, 2025View on HN Parent context
Why do we remember some life moments but not others? Sep 20251 comment
- For those who don't get the reference. It's googles breakthrough paper on LLM's. https://arxiv.org/abs/1706.03762
  
  nadermxSep 27, 2025View on HN Parent context
Learn Your Way: Reimagining Textbooks with Generative AI Sep 20251 comment
- This looks great. Just fyi this is also possible via alphaxiv's assistant. Navigate to a paper, click tools -> assistant. [0] https://www.alphaxiv.org/abs/1706.03762
  
  meander_waterSep 18, 2025View on HN Parent context
Claude Code is all you need Aug 20251 comment
- It's a play on the name of the paper that jump-started ChatGPT: "Attention Is All You Need:" https://arxiv.org/abs/1706.03762
  
  OldfieldFundAug 11, 2025View on HN Parent context
AI capex is so big that it's affecting economic statistics Jul 20251 comment
- > do you remember "natural language processing"? What, ehm, happened to it There’s this paper[1] you should read, is sparked an entire new AI dawn, it might answer your question 1. https://arxiv.org/abs/1706.03762
  
  jpc0Jul 18, 2025View on HN Parent context
Graphical Linear Algebra Jul 20251 comment
- Generalized Transformers from Applicative Functors >Transformers are a machine-learning model at the foundation of many state-of-the-art systems in modern AI, originally proposed in [arXiv:1706.03762]. In this post, we are going to build a generalization of Transformer models…
  
  Xmd5aJul 10, 2025View on HN Parent context
Reinforcement Pre-Training Jun 20251 comment
- So true, just like this one: https://arxiv.org/abs/1706.03762
  
  NotAnOtterJun 10, 2025View on HN Parent context
TransMLA: Multi-head latent attention is all you need May 20251 comment
- The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in…
  
  tankenmateMay 13, 2025View on HN Parent context
Markov Chains Explained Visually (2014) Feb 20251 comment
- Attention. It's really all you need [1] 1: https://arxiv.org/abs/1706.03762
  
  aGHzFeb 28, 2025View on HN Parent context
S1: The $6 R1 Competitor? Feb 20251 comment
- https://arxiv.org/abs/1706.03762
  
  menaerusFeb 5, 2025View on HN Parent context
ArXiv LaTeX Cleaner: Clean the LaTeX code of your paper to submit to ArXiv Feb 20251 comment
- It's really a pity that they do this now. Some of their older papers had actually quite some valuable information, comments, discussions, thoughts, even commented out sections, figures, tables in it. It gave a much better view on how the paper was written over time, or how even…
  
  albertzeyerFeb 4, 2025View on HN Parent context
SmolGPT: A minimal PyTorch implementation for training a small LLM from scratch Jan 20251 comment
- This is a faithful reproduction of the original Transformer paper [1]. Except, these days we use trainable parameters for positional embedding. The paper used a static calculation for positional embedding using sine and cosine. Figure 1 in the paper can be seen implemented in…
  
  leopoldjJan 30, 2025View on HN Parent context
Tensor Product Attention Is All You Need Jan 20251 comment
- The abstract https://arxiv.org/abs/1706.03762 explains it well: "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through…
  
  yorwbaJan 22, 2025View on HN Parent context
Trump sides with Musk on support for H-1B visas for foreign tech workers Dec 20241 comment
- Here are the folks who contributed to the "Attention is all you need" paper [1] which changed the world as we know it, and the countries they came from. I rest my case. Ashish Vaswani: India Noam Shazeer: USA Niki Parmar: India Jakob Uszkoreit: Germany Llion Jones: USA Aidan…
  
  breadwinnerDec 29, 2024View on HN Parent context
How AlphaChip transformed computer chip design Sep 20241 comment
- Attention is all you need is 2017... https://arxiv.org/abs/1706.03762
  
  bombitaSep 28, 2024View on HN Parent context
DOJ accuses Visa of monopoly that affects price of 'nearly everything’ Sep 20241 comment
- US is an economic powerhouse due to land/population sizes, but if you look at attention is all you need paper (for example): https://arxiv.org/abs/1706.03762 majority of it is imported talent. It's like saying that because Apple is paying a Chinese factory to build phones that…
  
  fennecfoxySep 25, 2024View on HN Parent context
Darwin Machines Jul 20241 comment
- The short answer as to why capsule networks have "fallen out of fashion" is... Transformers. Transformers came out at roughly the same time[a] and have proven to be great at... pretty much everything. They just work. Since then, most AI research money, effort, and compute has…
  
  cs702Jul 17, 2024View on HN Parent context
New AI Training Technique Is Drastically Faster, Says Google Jul 20241 comment
- Deepmind did not invent transformers... https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1810.04805
  
  hiddencostJul 6, 2024View on HN Parent context
The Illustrated Transformer (2018) Jul 20241 comment
- I gave a talk on using Google BERT for financial services problems at a machine learning conference in early 2019. During my preparation, this was the only resource on transformers I could find that was even remotely understandable to me. I had a lot of trouble understand what…
  
  ryan-duveJul 2, 2024View on HN Parent context
If you replace the "x" with a "5" in an ArXiv URL, you get an HTML version May 20241 comment
- Nothing will work on all PDFs. A PDF could order its contents randomly letter by letter and then just position them right. They're both using LaTeXML v0.8.8, so both will output the same HTML. Example: https://ar5iv.org/abs/1706.03762 https://arxiv.org/html/1706.03762v7
  
  yencabulatorMay 30, 2024View on HN Parent context
Ilya Sutskever: “If you learn all of these, you’ll know 90% of what matters” May 20241 comment
- a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf…
  
  is_trueMay 18, 2024View on HN Parent context

Showing 40 of 105 threads. Download the full data to see all.

Submitted to Hacker News

Show HN: Talk to any ArXiv paper just by changing the URL194 points73 commentsDec 2023
Hello HN, Talk2Arxiv is a small open-source RAG application I've been building for a few weeks. To use it just prepend any arxiv.org link with 'talk2' to load the paper into a responsive RAG chat application (e.g. www.arxiv.org/abs/1706.03762 -> www.talk2arxiv.org/abs/1706.03762). All implementation details are in the GitHub. Currently, because I've opted to extract text from the PDF of the paper rather than reading the LaTeX source code (since I wanted to build a more generic PDF RAG in the…
The paper that made ChatGPT possible173 points55 commentsFeb 2023
Show HN: ArXiv-txt, LLM-friendly ArXiv papers22 points11 commentsFeb 2025
Just change arxiv.org to arxiv-txt.org in the URL to get the paper info in markdown Example: Original URL: https://arxiv.org/abs/1706.03762 Change to: https://arxiv-txt.org/abs/1706.03762 To fetch the raw text directly, use https://arxiv-txt.org/raw/abs/1706.03762, this will be particularly useful for APIs and agents
Show HN: Explain any ArXiv paper just add 'explain'14 points6 commentsFeb 2025
Deep Neural Networks – Attention is all you need (No recurrence or convolutions)12 points0 commentsJun 2017
Attention Is All You Need (Neural Networks)8 points3 commentsJun 2017
Attention Is All You Need6 points2 commentsJun 2025
Show HN: Turn Any ArXiv Paper into a 200-Page Prerequisite Reading Book6 points3 commentsSep 2024
I created this tool over the weekend because, as someone interested in AI and technology, I find many research papers on arXiv fascinating but often incredibly dense and difficult to understand due to the heavy jargon and technical language. I wanted to make these complex topics more accessible to laypeople like myself by converting the topics described in these papers into full-length books that break down key concepts, making cutting-edge research easier to grasp. You can think of it as…
Attention Is All You Need4 points0 commentsNov 2022
Attention is all you need (2017)3 points0 commentsApr 2025
Attention is all you need paper is offline due to invalid certificate3 points0 commentsMay 2024
Attention Is All You Need3 points0 commentsDec 2020
Attention Is All You Need3 points0 commentsJun 2017
Attention Is All You Need3 points0 commentsJun 2017
Attention Is All You Need2 points0 commentsMay 2025
Attention Is All You Need [pdf]2 points0 commentsFeb 2019
Attention Is All You Need2 points0 commentsJun 2017
Attention is All You Need by Google Brain2 points1 commentJun 2017
Attention is all you need (2017)1 point0 commentsJul 2025
Only journalistic or scholarly works are permitted for Attention paper1 point0 commentsSep 2023
Attention Is All You Need paper got an updated submission with this red note "Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or scholarly works." https://arxiv.org/abs/1706.03762v7
Attention Is All You Need (2017)1 point0 commentsNov 2022

Connected through shared HN threads, not inferred similarity or a recommendation.

Abstract

On Hacker News

Discussed in comments105 threads

Submitted to Hacker News

Also discussed alongside