A Logic for Expressing Log-Precision Transformers — HN submission

Fri, 26 May 2023 18:43:55 GMT

A Logic for Expressing Transformers

2 points; 6 discussion replies on HN. Paper and source history. A submission is not an endorsement.

A Logic for Expressing Log-Precision Transformers — mentioned by bmc7505

Tue, 25 Apr 2023 17:58:32 GMT

See Merrill & Sabharwal (2023) [1]: A transformer N is a specific kind of a computation graph family over vectors of floating-point numbers with p precision. We work with fixed-precision transformers, i.e., we take p to be constant with respect to the input sequence length n. In practice, transformers typically use p = 32, and, several newer, larger transformer language models use p = 16 (Brown et al., 2020; Zhang et al., 2022), even while allowing larger context lengths n than their predecessors. [1]: https://arxiv.org/pdf/2210.02671.pdf#page=7

Paper and discussion history. A mention is not an endorsement.

A Logic for Expressing Log-Precision Transformers · HackerPapers discussions

A Logic for Expressing Log-Precision Transformers — HN submission

A Logic for Expressing Log-Precision Transformers — mentioned by bmc7505