Attention Is All You Need · HackerPapers discussions

Attention Is All You Need — mentioned by nater5000

Wed, 30 Sep 2026 16:46:28 GMT

>The new game in town is adopting Chinese labs’ advances. Note how I call this adoption instead of the more vitriol-infused “stealing” that Anthropic tends to use. I mean, there's a pretty big difference between labs publishing their research openly and a competitor utilizing it versus a lab breaking TOS to... hmmm, what's the word? steal data from a competitor? >That’s because, unlike the Western companies, the Chinese are pretty much giving away their recipes. Yeah, Western AI companies have never published their research. It's crazy how the Chinese had to independently develop the…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by zserge

Tue, 22 Sep 2026 11:47:37 GMT

Same, and I thought it's a clever play on words on the classic "Attention is all you need" - https://arxiv.org/abs/1706.03762 - which led us to what we have now

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by snet0

Wed, 02 Sep 2026 17:59:47 GMT

Correct, that's not how LLMs work [0]. [0] - https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by nsagent

Sat, 11 Jul 2026 16:40:31 GMT

I'm sorry, but I'm not sure what works best for a beginner. I started my PhD when the original Transformers paper [1] had just been released. I had no background in NLP whatsoever and used the original paper to write a Transformer and the full training pipeline from scratch during the first couple months of my PhD without referencing any existing code (only reading the paper and it's references). So I'd say, if you're motivated you could do the same. That said, I've always been a self-starter and I started my PhD after working for a decade. I'm sure there are other resources out there, but…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by cute_boi

Tue, 07 Jul 2026 19:03:23 GMT

No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/ ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385 Multi-Scale Context Aggregation by Dilated Convolutions…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by gooob

Tue, 07 Jul 2026 18:54:32 GMT

https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... https://papers.nips.cc/paper_files/paper/2012/file/c399862d3... https://arxiv.org/pdf/1512.03385 https://arxiv.org/pdf/1511.07122 https://arxiv.org/pdf/1603.05027 https://arxiv.org/pdf/1409.2329 https://arxiv.org/pdf/1512.02595 https://arxiv.org/pdf/1409.0473 https://arxiv.org/pdf/1506.03134 https://arxiv.org/pdf/1706.03762 https://nlp.seas.harvard.edu/annotated-transformer/ https://arxiv.org/pdf/1410.5401 https://arxiv.org/pdf/1706.01427 https://arxiv.org/pdf/1806.01822…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by KludgeShySir

Wed, 01 Jul 2026 17:50:37 GMT

https://www.alphaxiv.org/ is a nice place to browse, search for, and read ArXiv papers which have optional AI summaries and chat. If you like one paper, you can get a list of similar papers. To view a specific paper, just take original link and change "arxiv" --> "alphaxiv". For example: https://www.alphaxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by benoau

Sun, 07 Jun 2026 17:11:23 GMT

There's actually eight authors of that paper, if ChatGPT is accurate: - Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones and Lukasz Kaiser stayed on at Google - Aidan N. Gomez became a researcher - Illia Polosukhin co-founded https://near.ai https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by lyfeninja

Wed, 03 Jun 2026 10:45:20 GMT

Below is the "Attention is all you need" paper. Transformers and their attention mechanism was the major breakthrough for modern LLMs. ML has been around for a long time, I'd suggest joining kaggle or something and learn by doing. You'll retain more and realize how broad the category is anymore. https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by newelh

Wed, 20 May 2026 23:14:28 GMT

I built udoc because most document extraction tools I've used require significant dependencies, only handle one format, or have restrictive licenses. I wanted a single binary that reads PDFs, Office docs (including legacy .doc/.xls/.ppt), ODF, and RTF — with no external parsers, no system packages, nothing to install. It's written in pure Rust with Python bindings via PyO3. If you have uv, you can try it right now without installing anything: `curl -sL https://arxiv.org/pdf/1706.03762 \ | uvx udoc - | grep -A 18 '^Abstract'` Highlights: A CLI: e.g. udoc -J ingest.pdf | duckdb -c "COPY (SELECT…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by cookiengineer

Sun, 03 May 2026 01:10:51 GMT

Maybe read up on how transformers, their encoders and decoders, and the attention matrix works? https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by johnbarron

Sat, 02 May 2026 15:10:40 GMT

And the credit of OpenAI is to Google? https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by make3

Wed, 29 Apr 2026 01:58:54 GMT

The transformer paper was 9 years ago. 9 years between barely translating alright between two very closely related languages (English and French, huge fraction of shared words because of William the conqueror and cultural proximity etc etc) and what we have now. The thing is able to code up full pretty competent thousand lines projects in an hour. Even hardcore engineers use it now, as of this year. My senior front end friends already can't find jobs. You're crazy if you think things won't change dramatically, at the scale of all of society. https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by ninjagoo

Tue, 28 Apr 2026 04:19:17 GMT

> Superior architectures will leak pretty quickly via engineers. I agree with the outcome of your premise (i.e., openness), but for different reasons: First, isn't it the case that these bleeding edge 'newfangled' LLMs are basically variations on the same core ideas from "Attention Is All You Need" from 2017? [1]. Different scale, but still the same basic architecture. Even the "MoE" innovation keeps the Transformer attention stack while replacing or augmenting the dense feed-forward/MLP part with routed expert blocks. And, I would argue that Engineers aren't working on new architectures.…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by smokel

Mon, 23 Mar 2026 09:30:20 GMT

Interestingly, the person at Microsoft states in a reply that even most of them have to pick this up in their spare time. Judging from other replies, it seems that there are quite some differences in how companies approach this. What I meant with "menial coding" is those jobs where people have to submit TPS reports on how much hours they spend for each customer. Reading a paper such as this one [1] is typically not directly necessary for being a good frontend developer, but it might stimulate someone to develop into a more fruitful employee in the long run. Managers would have to explain to…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by max_

Sat, 21 Mar 2026 00:09:49 GMT

No particular order but; 1. Transformers I think they need to be looked into more. They just work too well. I feel like they have a secret within that is yet to be known. [1] https://arxiv.org/abs/1706.03762 2. The 2016 Noble Prize in Medicine Autophagy has such interesting properties. It can probably do alot more for us now than any new pill or gene editing tool [2] https://www.nobelprize.org/prizes/medicine/2016/press-releas... 3. zk-STARKS I can prove to you than X * Y = Z, without ever revealing what, X, Y or Z are. And I can do this for any function, or math operator.…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by Kim_Bruning

Wed, 11 Mar 2026 20:36:06 GMT

LLMs were -in part- designed as translation tools. It's one thing they do really really well. https://arxiv.org/html/1706.03762v7 (Attention is all you need) "Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train." Ok, looking that up, that was quite literally one of the main design goals. And they're really quite good at translating between the languages I use. They're the best tool for the job.

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by rrr_oh_man

Thu, 26 Feb 2026 20:03:30 GMT

> i'm just so surprised they'd use chatgpt to do this, when it's quite as easily (and perhaps faster) to use google translate. https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by canyon289

Thu, 18 Dec 2025 21:19:46 GMT

Hi, I'm not on the t5 Gemma team but work on gemma in general. Encoder Decoder comes from the original transformers implementation way back in 2017. If you look at figure 1 you'll see what the first transformer ever looked like. Since that time different implementations of transformers use either just the encoder portion, or the decoder portion, or both. Its a deep topic so hard to summarize here, but Gemini explains it really well! Hope this gets you started on some prompting to learn more https://arxiv.org/pdf/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by dormento

Tue, 21 Oct 2025 17:01:28 GMT

In case anyone missed the reference: https://arxiv.org/abs/1706.03762 > (...) We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by hodgesrm

Thu, 09 Oct 2025 19:49:45 GMT

> You're missing the point. In the past bad writing was just bad writing, and it was typically easy to detect. If AI generated text were well written, would it matter to you? Is it bad to use Grammarly? I don't see anything inherently wrong with using AI tools to write, as long as writers take the responsibility to ensure the final result is good. Fighting against use of LLMs seems like a fool's errand at this point. Personally I've been using Google Translate for years to help with writing in German, little knowing at the time that it was using transformers under the covers. [0] I'm pretty…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by nadermx

Sat, 27 Sep 2025 00:07:03 GMT

For those who don't get the reference. It's googles breakthrough paper on LLM's. https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by meander_water

Thu, 18 Sep 2025 22:06:51 GMT

This looks great. Just fyi this is also possible via alphaxiv's assistant. Navigate to a paper, click tools -> assistant. [0] https://www.alphaxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by OldfieldFund

Mon, 11 Aug 2025 18:07:37 GMT

It's a play on the name of the paper that jump-started ChatGPT: "Attention Is All You Need:" https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by jpc0

Fri, 18 Jul 2025 23:31:28 GMT

> do you remember "natural language processing"? What, ehm, happened to it There’s this paper[1] you should read, is sparked an entire new AI dawn, it might answer your question 1. https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by Xmd5a

Thu, 10 Jul 2025 17:58:34 GMT

Generalized Transformers from Applicative Functors >Transformers are a machine-learning model at the foundation of many state-of-the-art systems in modern AI, originally proposed in [arXiv:1706.03762]. In this post, we are going to build a generalization of Transformer models that can operate on (almost) arbitrary structures such as functions, graphs, probability distributions, not just matrices and vectors. >[...] >This work is part of a series of similar ideas exploring machine learning through abstract diagrammatical means. https://cybercat.institute/2025/02/12/transformers-applicati...

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — HN submission

Fri, 04 Jul 2025 07:06:51 GMT

Attention is all you need (2017)

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — HN submission

Sun, 29 Jun 2025 18:43:24 GMT

Attention Is All You Need

6 points; 2 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — mentioned by NotAnOtter

Tue, 10 Jun 2025 19:25:18 GMT

So true, just like this one: https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — HN submission

Sat, 31 May 2025 02:58:48 GMT

Attention Is All You Need

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — mentioned by tankenmate

Tue, 13 May 2025 08:38:08 GMT

The title of this paper is a reference to a previous paper titled "Attention Is All You Need"[0][1]. This seminal work described the transformer model that is the basis for almost all LLMs, and is almost certainly the most cited paper on AI even though it was only published in 2017. [0] https://arxiv.org/abs/1706.03762 [1] https://en.wikipedia.org/wiki/Attention_Is_All_You_Need

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — HN submission

Fri, 18 Apr 2025 16:17:30 GMT

Attention is all you need (2017)

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — mentioned by aGHz

Fri, 28 Feb 2025 02:01:49 GMT

Attention. It's really all you need [1] 1: https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — HN submission

Thu, 20 Feb 2025 04:23:54 GMT

Show HN: ArXiv-txt, LLM-friendly ArXiv papers

Just change arxiv.org to arxiv-txt.org in the URL to get the paper info in markdown Example: Original URL: https://arxiv.org/abs/1706.03762 Change to: https://arxiv-txt.org/abs/1706.03762 To fetch the raw text directly, use https://arxiv-txt.org/raw/abs/1706.03762, this will be particularly useful for APIs and agents

22 points; 11 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — mentioned by menaerus

Wed, 05 Feb 2025 15:54:52 GMT

https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by albertzeyer

Tue, 04 Feb 2025 08:58:52 GMT

It's really a pity that they do this now. Some of their older papers had actually quite some valuable information, comments, discussions, thoughts, even commented out sections, figures, tables in it. It gave a much better view on how the paper was written over time, or how even the work processed over time. Sometimes you also see some alternative titles being discussed, which can be quite funny. E.g. from https://arxiv.org/abs/1804.09849: %\title{Sequence-to-Sequence Tricks and Hybrids\\for Improved Neural Machine Translation} % \title{Mixing and Matching Sequence-to-Sequence Modeling…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — HN submission

Sat, 01 Feb 2025 07:29:56 GMT

Show HN: Explain any ArXiv paper just add 'explain'

14 points; 6 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — mentioned by leopoldj

Thu, 30 Jan 2025 14:17:50 GMT

This is a faithful reproduction of the original Transformer paper [1]. Except, these days we use trainable parameters for positional embedding. The paper used a static calculation for positional embedding using sine and cosine. Figure 1 in the paper can be seen implemented in the forward() method of the GPT class in model.py. Here are the rough steps: 1. Tokens are embedded using a nn.Embedding layer. 2. Tokens are positionally embedded using a nn.Embedding layer. 3. The two embedding values are added to make the input x. 4. A sequence of N number of transformer blocks are then executed. This…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by yorwba

Wed, 22 Jan 2025 09:11:51 GMT

The abstract https://arxiv.org/abs/1706.03762 explains it well: "The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely." They did not invent attention, but while previous language models had used attention as an auxiliary mechanism, they removed everything but…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by breadwinner

Sun, 29 Dec 2024 00:48:26 GMT

Here are the folks who contributed to the "Attention is all you need" paper [1] which changed the world as we know it, and the countries they came from. I rest my case. Ashish Vaswani: India Noam Shazeer: USA Niki Parmar: India Jakob Uszkoreit: Germany Llion Jones: USA Aidan Gomez: British Canadian Lukasz Kaiser: Poland Illia Polosukhin: Ukraine [1] https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by bombita

Sat, 28 Sep 2024 05:30:32 GMT

Attention is all you need is 2017... https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by fennecfoxy

Wed, 25 Sep 2024 15:14:25 GMT

US is an economic powerhouse due to land/population sizes, but if you look at attention is all you need paper (for example): https://arxiv.org/abs/1706.03762 majority of it is imported talent. It's like saying that because Apple is paying a Chinese factory to build phones that Apple "builds phones".

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — HN submission

Mon, 09 Sep 2024 18:23:51 GMT

Show HN: Turn Any ArXiv Paper into a 200-Page Prerequisite Reading Book

I created this tool over the weekend because, as someone interested in AI and technology, I find many research papers on arXiv fascinating but often incredibly dense and difficult to understand due to the heavy jargon and technical language. I wanted to make these complex topics more accessible to laypeople like myself by converting the topics described in these papers into full-length books that break down key concepts, making cutting-edge research easier to grasp. You can think of it as generating prerequisite reading materials before being able to read the actual paper. Here are some…

6 points; 3 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — mentioned by cs702

Wed, 17 Jul 2024 16:55:19 GMT

The short answer as to why capsule networks have "fallen out of fashion" is... Transformers. Transformers came out at roughly the same time[a] and have proven to be great at... pretty much everything. They just work. Since then, most AI research money, effort, and compute has been invested to study and improve Transformers and related models, at the expense of almost everything else. Many promising ideas, including routing, won't be seriously re-explored until and unless progress towards AGI seems to stall. --- [a] https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by hiddencost

Sat, 06 Jul 2024 22:30:09 GMT

Deepmind did not invent transformers... https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1810.04805

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by ryan-duve

Tue, 02 Jul 2024 23:28:25 GMT

I gave a talk on using Google BERT for financial services problems at a machine learning conference in early 2019. During my preparation, this was the only resource on transformers I could find that was even remotely understandable to me. I had a lot of trouble understand what was going on from just the original publication[0]. [0] https://arxiv.org/abs/1706.03762

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by yencabulator

Thu, 30 May 2024 19:26:18 GMT

Nothing will work on all PDFs. A PDF could order its contents randomly letter by letter and then just position them right. They're both using LaTeXML v0.8.8, so both will output the same HTML. Example: https://ar5iv.org/abs/1706.03762 https://arxiv.org/html/1706.03762v7

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — HN submission

Wed, 22 May 2024 11:43:45 GMT

Attention is all you need paper is offline due to invalid certificate

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Attention Is All You Need — mentioned by is_true

Sat, 18 May 2024 12:26:29 GMT

a backup just in case https://nlp.seas.harvard.edu/annotated-transformer/ https://scottaaronson.blog/?p=762 https://karpathy.github.io/2015/05/21/rnn-effectiveness/ https://colah.github.io/posts/2015-08-Understanding-LSTMs/ https://arxiv.org/pdf/1409.2329.pdf https://www.cs.toronto.edu/~hinton/absps/colt93.pdf https://arxiv.org/pdf/1506.03134.pdf https://proceedings.neurips.cc/paper_files/paper/2012/file/c... https://arxiv.org/pdf/1511.06391.pdf https://arxiv.org/pdf/1811.06965.pdf https://arxiv.org/pdf/1512.03385.pdf https://arxiv.org/pdf/1511.07122.pdf https://arxiv.org/pdf/1704.01212.pdf…

Paper and discussion history. A mention is not an endorsement.

Attention Is All You Need — mentioned by nabla9

Thu, 16 May 2024 06:59:56 GMT

Huge loss for for sure. The man, the legend, the Harrison Kinsley: Attention Is All You Need Harrison Kinsley, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin https://arxiv.org/abs/1706.03762 Deep Learning. Harrison Kinsley, LeCun, Y., Bengio, Y. and Hinton, G. E. Deep Learning. Nature, Vol. 521, pp 436-444. ImageNet Classification with Deep Convolutional Neural Networks Harrison Kinsley, Ilya Sutskever, Geoffrey E. Hinton Adam: A Method For Stochastic Optimization Harrison Kinsley, Diederik P. Kingma

Paper and discussion history. A mention is not an endorsement.