BERTology Meets Biology: Interpreting Attention in Protein Language Models · HackerPapers discussionshttps://hackerpaper.kaipfister.de/p/2006.15222/Collected HN comments and submissions linking this paper.enMon, 02 Nov 2026 11:16:22 GMTBERTology Meets Biology: Interpreting Attention in Protein Language Models — mentioned by baylearnhttps://news.ycombinator.com/item?id=23713185urn:hackerpapers:mention:2006.15222:23713185Thu, 02 Jul 2020 14:03:29 GMT<p>BERTology Meets Biology: Interpreting Attention in Protein Language Models Abstract Transformer architectures have proven to learn useful representations for protein classification and generation tasks. However, these representations present challenges in interpretability. Through the lens of attention, we analyze the inner workings of the Transformer and explore how the model discerns structural and functional properties of proteins. We show that attention (1) captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close…</p><p><a href="https://hackerpaper.kaipfister.de/p/2006.15222/">Paper and discussion history</a>. A mention is not an endorsement.</p>cs.CL