Fine-Tuning Pre-trained Transformers into Decaying Fast Weights — mentioned by lucidrains

Thu, 30 Mar 2023 15:24:05 GMT

while this topic has visibility, there is also another relatively unexplored research direction, fine-tuning pretrained transformers into RNNs https://arxiv.org/abs/2210.04243 https://aclanthology.org/2021.emnlp-main.830.pdf

Paper and discussion history. A mention is not an endorsement.

Fine-Tuning Pre-trained Transformers into Decaying Fast Weights · HackerPapers discussions

Fine-Tuning Pre-trained Transformers into Decaying Fast Weights — mentioned by lucidrains