Skip to main content

Graph Attention Networks (GAT)

Adaptive attention-based aggregation for graph neural networks - multi-head attention, learned weights, and interpretable graph learning

How much should each neighbor contribute to node 2? A node carries a feature vector; an edge permits a message. Original GAT first applies the same matrix W to every node, then scores each permitted pair with an additive vector a and normalizes over that node’s neighbors.

The worked graph has six nodes with authored four-dimensional features. Here W is identity, a is displayed, self loops and dropout are omitted, and every weight and weighted output is calculated. Inspect node 2: its five neighbor weights must sum to one. Compare the multi-head and matrix tabs only after following that input → score → softmax → weighted sum. The guide uses a second explicitly labeled toy graph and the same operator.

Graph attention

Neighbors of node 2: transform, score, normalize, aggregate

Calculated attention for node 2

Authored features and parameters, no training, no self loops, no dropout. W = identity; a = [0.25, 0.25, 0.25, 0.25, 0.25, 0.5, 0.75, 1]. Softmax is over actual neighbors only. Weight sum = 1.000000. Output = [0.5226, 0.4589, 0.5390, 0.6326].

Scroll the table horizontally to inspect all columns.

NeighborInput = WhLogitWeight
0[0.8, 0.2, 0.5, 0.9]2.07500.249615
1[0.3, 0.9, 0.1, 0.4]1.50000.140459
3[0.2, 0.7, 0.3, 0.6]1.72500.175900
4[0.9, 0.1, 0.6, 0.3]1.52500.144015
5[0.4, 0.5, 0.9, 0.7]2.22500.290011

Overview

Graph Attention Networks (GAT) introduce attention mechanisms to graph neural networks, allowing nodes to adaptively weight their neighbors' contributions based on learned attention coefficients. Unlike GCNs, which use fixed aggregation weights based solely on graph structure with degree-dependent neighbor weights, GATs learn which neighbors are most relevant for each node.

Key Concepts

Attention Mechanism

  • Shared transform: Apply W to every feature vector; score concatenated transformed pairs with a and LeakyReLU
  • Attention Scores: Compute compatibility between node pairs; nodes compute attention scores for all neighbors
  • Softmax Normalization: Convert scores to probabilities
  • Weighted Aggregation: Messages flow through the graph weighted by learned attention coefficients. Each node's new features are computed as a weighted sum of neighbor features

Multi-Head Attention

  • Parallel Attention: Multiple attention heads learn different relationships
  • Feature Diversity: Each head focuses on different aspects
  • Concatenation/Average: Combine outputs from all heads
  • Improved Stability: More robust learning through ensemble

Advantages Over GCN

  1. Adaptive Weights: Learn the importance of each edge dynamically, adapting to node features
  2. Inspectable coefficients: Visualize attention weights without treating them as causal explanations
  3. Inductive Learning: Generalize to unseen graphs
  4. Parallelizable: Efficient computation across edges

Applications

  • Social network analysis with varying relationship strengths
  • Molecular property prediction with chemical bond attention
  • Knowledge graph reasoning with relation-aware attention
  • Traffic prediction with dynamic road importance

Implementation Tips

  • Use LeakyReLU for attention coefficient computation
  • Apply dropout to attention weights for regularization
  • Initialize attention parameters carefully
  • Monitor attention entropy to detect collapse

What comes next

Continue to graph pooling to turn updated node representations into a smaller graph or a graph-level vector.

If you found this explanation helpful, consider sharing it with others.

Mastodon