Rethinking Language Models as Symbolic Knowledge Graphs

In the ever-evolving field of artificial intelligence, a recent study titled "Rethinking Language Models as Symbolic Knowledge Graphs" by Vishwas Mruthyunjaya, Pouya Pezeshkpour, Estevam Hruschka, and Nikita Bhutani has sparked a significant discussion. The paper delves into the capabilities of contemporary language models (LMs) and their ability to match up to knowledge graphs (KGs), a critical component in knowledge-centric applications such as search, question answering, and recommendation.

The researchers conducted a comprehensive evaluation of language models of varying sizes and abilities. They developed nine qualitative benchmarks that encompass a spectrum of attributes including symmetry, asymmetry, hierarchy, bidirectionality, compositionality, paths, entity-centricity, bias, and ambiguity. These benchmarks were designed to test the language models' ability to capture intricate topological and semantic traits of knowledge graphs.

The research findings indicate that while language models exhibit considerable potential in recalling factual information, their ability to capture intricate topological and semantic traits of KGs remains significantly constrained. For instance, larger LMs such as GPT-4 were found to be on par with or outperform smaller models across most patterns, barring a few exceptions. However, the grasp of KG attributes was not consistent across different relations/patterns.

Interestingly, the study also challenges the common notion that larger LMs universally outshine their smaller counterparts. For example, GPT-4 is outperformed by BERT on bidirectional, compositional, and ambiguity benchmarks. This suggests that existing metrics which evaluate triples in isolation might overestimate the model performance in capturing nuanced KG attributes.

The implications of this research are significant. While LMs can efficiently retrieve knowledge from KGs, their limitations in capturing the nuanced attributes of KGs, which are crucial for reasoning processes, are evident. This means that while LMs can access specific pieces of information within KGs, they may not perform complex reasoning tasks as effectively as symbolic KGs. This research opens up new avenues for improving the capabilities of language models and their application in various knowledge-centric domains.

Read the whole article here: http://arxiv.org/abs/2308.13676v1