The Framework of Knowledge

Retrieval Augmented Generation provides a solution by clustering the smaller chunks by meaning

What Fortune 500 companies are really spending money on AI for right now:

• format their existing documents

• search through them

• explain them using a language model

This simplistic process goes by the rather clunky moniker of Retrieval Augmented Generation but has become very attractive as the search AI (embeddings) seems to understand context much better than simple keywords.

The main issue with this methodology is the key question of how many documents get fed to the language model to explain. This paper from Stanford provides a solution that beats state of the art, namely just repeatedly summarize small chunks of documents until the main themes are stored in the top layer. Notably, it ends up clustering the smaller chunks by meaning (semantic similarity).