Boosting Search Engines with Interactive Agents — mentioned by gamegoblin

Thu, 06 Jul 2023 14:06:05 GMT

The benefit of "traditional" O(N^2) transformer attention is you correlate every token to every other token. So, in the limit, your network won't "miss" much. When you abandon O(N^2) attention, you are forced to start adding heuristics to choose what to correlate. Any time you see one of those giant context window LLMs, you need to be asking what heuristics they added, what is getting correlated, and what is not getting correlated. This paper chooses an exponential heuristic where tokens further in the past get exponentially less attention. This heuristic is fine for certain tasks like…

Paper and discussion history. A mention is not an endorsement.

Boosting Search Engines with Interactive Agents · HackerPapers discussions

Boosting Search Engines with Interactive Agents — mentioned by gamegoblin