Function Vectors in Large Language Models — mentioned by famouswaffles

Thu, 26 Oct 2023 01:05:20 GMT

This is speculation but i don't think this information (how it came to be) is actually irrelevant information for the LLM. Your state, which transformer LLMs notably have no way of transferring is "just" the computation that preceded the prediction. I think LLMs use previous tokens to compute the likely state vector that preceded the tokens and make predictions that continue this computed state. So the 2 board positions at time x may be identical but the state preceding it or "strategy" was not and so the next move is not identical. This way also, you can make a decision based on the…

Paper and discussion history. A mention is not an endorsement.

Function Vectors in Large Language Models · HackerPapers discussions

Function Vectors in Large Language Models — mentioned by famouswaffles