Towards Foundation Models for Knowledge Graph Reasoning
Abstract
Foundation models in language and vision have the ability to run inference on any textual and visual inputs thanks to the transferable representations such as a vocabulary of tokens in language. Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap. The key challenge of designing foundation models on KGs is to learn such transferable representations that enable inference on any graph with arbitrary entity and relation vocabularies. In this work, we make a step towards such foundation models and present ULTRA, an approach for learning universal and transferable graph representations. ULTRA builds relational representations as a function conditioned on their interactions. Such a conditioning strategy allows a pre-trained ULTRA model to inductively generalize to any unseen KG with any relation vocabulary and to be fine-tuned on any graph. Conducting link prediction experiments on 57 different KGs, we find that the zero-shot inductive inference performance of a single pre-trained ULTRA model on unseen graphs of various sizes is often on par or better than strong baselines trained on specific graphs. Fine-tuning further boosts the performance.
Community
Proposes ULTRA (unified learnable and transferable knowledge graph representations): a foundation model for knowledge graph (KG) reasoning (learn universal and transferable representations over graph data structure); generalizes to KGs of different sizes (number of nodes and edges); inductive link prediction through structural information in the graph. A graph is a tuple of vertices, relations (types), and edges (vertices connected by particular relation types); transductive setup has same graph for training and inference (validation or test), inductive has different graphs, fully-inductive have non-intersecting vertices and relations during train and test; knowledge graph reasoning answers queries like “predict tail given head and relation” (or the inverse); GNN encoders can assign different nodes the same features (making them indistinguishable), labelling trick assigns each node a unique label/feature vector based on structural properties - distance encoding, or even conditional node representations like NBFNet. Generalizes KG reasoning to new entities (nodes) and relations (edges) by using graph of relations (and inverse relations). There are four core/fundamental relation-to-relation interactions (tail or head, to, tail or head - all combinations). Three steps to handle a query (head and relation query for a tail) and a graph: build/lift to graph of relations, get relative relation representation conditioned on query and graph of relations, and run inductive link prediction on the original graph. Conditional representation through the labeling trick and using message passing GNN. Uses (modified) NBFNet for conditional representation and inductive link prediction. Trained with binary cross entropy with triplet loss. Pretrained on three KG datasets: WN18RR, CoDEx-Medium, and FB15k237; zero-shot is better than fine-tuning in most cases, but fine-tuning gives even better results. Has ablations to pre-training mixture of datasets and fine-tuning improvements, also on edge types in relation graph (and effects on different transductive and inductive - edge and relation - settings). Appendix has information on datasets, sparse matrix multiplication for relational graphs (GNNs), hyperparameter and training details, and more results. From Intel, Mila, CIFAR.
Models citing this paper 3
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper