How much should each neighbor contribute to node 2? A node carries a feature vector; an edge permits a message. Original GAT first applies the same matrix W to every node, then scores each permitted pair with an additive vector a and normalizes over that node’s neighbors.
The worked graph has six nodes with authored four-dimensional features. Here W is identity, a is displayed, self loops and dropout are omitted, and every weight and weighted output is calculated. Inspect node 2: its five neighbor weights must sum to one. Compare the multi-head and matrix tabs only after following that input → score → softmax → weighted sum. The guide uses a second explicitly labeled toy graph and the same operator.
Neighbors of node 2: transform, score, normalize, aggregate
Calculated attention for node 2
Authored features and parameters, no training, no self loops, no dropout. W = identity; a = [0.25, 0.25, 0.25, 0.25, 0.25, 0.5, 0.75, 1]. Softmax is over actual neighbors only. Weight sum = 1.000000. Output = [0.5226, 0.4589, 0.5390, 0.6326].
Scroll the table horizontally to inspect all columns.
| Neighbor | Input = Wh | Logit | Weight |
|---|---|---|---|
| 0 | [0.8, 0.2, 0.5, 0.9] | 2.0750 | 0.249615 |
| 1 | [0.3, 0.9, 0.1, 0.4] | 1.5000 | 0.140459 |
| 3 | [0.2, 0.7, 0.3, 0.6] | 1.7250 | 0.175900 |
| 4 | [0.9, 0.1, 0.6, 0.3] | 1.5250 | 0.144015 |
| 5 | [0.4, 0.5, 0.9, 0.7] | 2.2250 | 0.290011 |
Overview
Graph Attention Networks (GAT) introduce attention mechanisms to graph neural networks, allowing nodes to adaptively weight their neighbors' contributions based on learned attention coefficients. Unlike GCNs, which use fixed aggregation weights based solely on graph structure with degree-dependent neighbor weights, GATs learn which neighbors are most relevant for each node.
Key Concepts
Attention Mechanism
- Shared transform: Apply W to every feature vector; score concatenated transformed pairs with a and LeakyReLU
- Attention Scores: Compute compatibility between node pairs; nodes compute attention scores for all neighbors
- Softmax Normalization: Convert scores to probabilities
- Weighted Aggregation: Messages flow through the graph weighted by learned attention coefficients. Each node's new features are computed as a weighted sum of neighbor features
Multi-Head Attention
- Parallel Attention: Multiple attention heads learn different relationships
- Feature Diversity: Each head focuses on different aspects
- Concatenation/Average: Combine outputs from all heads
- Improved Stability: More robust learning through ensemble
Advantages Over GCN
- Adaptive Weights: Learn the importance of each edge dynamically, adapting to node features
- Inspectable coefficients: Visualize attention weights without treating them as causal explanations
- Inductive Learning: Generalize to unseen graphs
- Parallelizable: Efficient computation across edges
Applications
- Social network analysis with varying relationship strengths
- Molecular property prediction with chemical bond attention
- Knowledge graph reasoning with relation-aware attention
- Traffic prediction with dynamic road importance
Implementation Tips
- Use LeakyReLU for attention coefficient computation
- Apply dropout to attention weights for regularization
- Initialize attention parameters carefully
- Monitor attention entropy to detect collapse
What comes next
Continue to graph pooling to turn updated node representations into a smaller graph or a graph-level vector.
Related concepts
Learn Graph Convolutional Networks (GCN) with spectral theory, message passing, and node classification for geometric deep learning.
Understanding end-to-end object detection with transformers, from DETR's object queries to bipartite matching and attention-based localization
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
Learn batch normalization in deep learning: how normalizing layer inputs accelerates training, improves gradient flow, and acts as regularization.
BatchNorm normalizes over the batch and spatial axes; LayerNorm normalizes over the channel and spatial axes for each sample. The choice changes whether your model trains stably with batch=1, depends on batch composition at inference, and behaves consistently across train and eval.
How the Calinski-Harabasz index evaluates clustering quality by measuring the ratio of between-cluster to within-cluster variance — fast, intuitive, and ideal for k-selection with convex clusters.
