Pure Transformers are Powerful Graph Learners — mentioned by coffee_am

Thu, 21 Sep 2023 14:35:23 GMT

A few considerations come to mind: 1. The O(N^2*d) computation cost of the attention layers. For large graphs (millions of nodes) it's quickly too costly. And in some of the social network problems, the more data you feed the better is the inference on average (on a ~log scale). 2. As the paper suggests, in some cases the graph structure has important information. Flattening out everything, or fully connecting the nodes, a more accurate description of what goes on in an attention layer in this scenario, the structure is lost. The structure information can be introduced as a positional…

Paper and discussion history. A mention is not an endorsement.

Pure Transformers are Powerful Graph Learners · HackerPapers discussions

Pure Transformers are Powerful Graph Learners — mentioned by coffee_am