An Introduction to Graph Attention Networks

Graph Learning is increasingly becoming more and more relevant as a significant amount of real-world data can be modelled as graphs.

The Graph Attention Network or GAT is a non-spectral learning method which utilizes the spatial information of the node directly for learning. This is in contrast to the spectral approach of the Graph Convolutional Network which mirrors the same basics as the Convolutional Neural Net.

In this article, I will explain how the GAT is constructed.

The basic building block of the GAT is the Graph Attention Layer. To explain the following graph is used as an example.

Here hi is a feature vector of length F.

The first step performed by the Graph Attention Layer is to apply a linear transformation — Weighted matrix W to the feature vectors of the nodes.

Attention Coefficients determine the relative importance of neighbouring features to each other. They are calculated using the formula.

Here a is a function that we determine subject to the following restriction.

i and j are neighbouring nodes.

The following figures further explain this.

As we observe in the series of images above, we first calculate the self-attention coefficient and then compute attention coefficients with all of the neighbours.

Due to the varied structure of graphs, nodes can have a different number of neighbours. To have a common scaling across all neighbourhoods, the Attention coefficients are Normalized.

Here N is the Neighbourhood of node i.

Now we compute the learned features of nodes. σ is a Non-Linear Transformation.

To improve the stability of the learning process, Multi-head attention is employed. We compute multiple different attention maps and finally aggregate all the learned representations.

K denotes the number of independent attention maps used.

This is my first Medium article and I hope you find it informative. I will update on the actual implementation of the Network in Python in the next article.

[1] P. Velickovic, G. Cucurull, A. Casanova, and A. Romero. Graph Attention Networks. In International Conference on Learning Representations, 2018.

Related posts:

The user interface frameworks and libraries of Javascript programming language work in a cycle. After every six months, they change positions and a new one pops up. From past four years I am using…

They say a stitch in time saves nine. This popular proverb’s wisdom extends even to a website! Without proper maintenance, your website’s health can decline over time. So before you get there…

You have a heart for change and want to impact the world with your words. You feel like you’re finally making your way, getting it out there, sharing a message tethered to your dreams. Then it hits…