地址

論文地址：https:///abs/2404.16130

時間

2024年4月24日

作者

Microsoft團隊

The use of retrieval-augmented generation (RAG) to retrieve relevant informa-tion from an external knowledge source enables large language models (LLMs) to answer questions over private and/or previously unseen document collections. However, RAG fails on global questions directed at an entire text corpus, such as “What are the main themes in the dataset?”, since this is inherently a query-focused summarization (QFS) task, rather than an explicit retrieval task. Prior QFS methods, meanwhile, fail to scale to the quantities of text indexed by typical RAG systems. To combine the strengths of these contrasting methods, we propose a Graph RAG approach to question answering over private text corpora that scales with both the generality of user questions and the quantity of source text to be in-dexed. Our approach uses an LLM?to build a graph-based text index in two stages: first to derive an entity knowledge graph from the source documents, then to pre-generate community summaries for all groups of closely-related entities. Given a question, each community summary is used to generate a partial response, before all partial responses are again summarized in a final response to the user. For a class of global sensemaking questions over datasets in the 1 million token range, we show that Graph RAG leads to substantial improvements over a 簡單的RAG?baseline for both the comprehensiveness and diversity of generated answers. An open-source, Python-based implementation of both global and local Graph RAG approaches is forthcoming at https:///graphrag.

使用檢索增強生成(retrieve -augmented generation, RAG)從外部知識來源檢索相關(guān)信息，使大型語言模型(LLM)能夠回答私有和/或以前未見過的文檔集合上的問題。然而，RAG在針對整個文本語料庫的全局問題上失敗了，例如“數(shù)據(jù)集中的主題是什么?”，因為這本質(zhì)上是一個以查詢?yōu)橹行牡?strong>查詢聚焦摘要(QFS)任務(wù)，而不是一個明確的檢索任務(wù)。與此同時，先前的QFS方法無法擴展到典型RAG系統(tǒng)索引的文本數(shù)量。為了結(jié)合這些對比方法的優(yōu)勢，我們提出了一種基于私有文本語料庫的Graph RAG方法，該方法可以根據(jù)用戶問題的通用性和要索引的源文本的數(shù)量進行擴展。我們的方法使用LLM分兩個階段構(gòu)建基于圖的文本索引：首先從源文檔中導(dǎo)出實體知識圖，然后為所有密切相關(guān)的實體組預(yù)生成社區(qū)摘要。給定一個問題，每個社區(qū)摘要用于生成部分響應(yīng)，然后將所有部分響應(yīng)再次匯總為對用戶的最終響應(yīng)。對于100萬個令牌范圍內(nèi)的數(shù)據(jù)集上的一類全局語義問題，我們表明Graph RAG在生成答案的全面性和多樣性方面比簡單的RAG基線有了實質(zhì)性的改進。一個開源的、基于python的全局和局部Graph RAG方法的實現(xiàn)即將在https:///graphrag上實現(xiàn)。

Human endeavors across a range of domains rely on our ability to read and reason about large collections of documents, often reaching conclusions that go beyond anything stated in the source texts themselves. With the emergence of large language models (LLMs), we are already witnessing attempts to automate human-like sensemaking in complex domains like scientific discovery (Mi-crosoft, 2023) and intelligence analysis (Ranade and Joshi, 2023), where sensemaking is defined as?“a motivated, continuous effort to understand connections (which can be among people, places, and events) in order to anticipate their trajectories and act effectively” (Klein et al., 2006a). Supporting human-led sensemaking over entire text corpora, however, needs a way for people to both apply and refine their mental model of the data (Klein et al., 2006b) by asking questions of a global nature.

人類在各個領(lǐng)域進行的活動依賴于我們閱讀和推理大量文檔的能力，常常得出超出源文本本身的結(jié)論。隨著大型語言模型（LLMs）的出現(xiàn)，我們已經(jīng)見證了在科學(xué)發(fā)現(xiàn)(Mi-crosoft, 2023)和情報分析(Ranade和Joshi, 2023)等復(fù)雜領(lǐng)域自動化類人語義構(gòu)建的嘗試，其中語義構(gòu)建被定義為“一種有動機的、持續(xù)的努力，以理解聯(lián)系(可以是人、地點和事件之間的聯(lián)系)，以便預(yù)測它們的軌跡并有效地采取行動”(Klein等人，2006a)。然而，支持人類主導(dǎo)的整個文本語料庫的語義構(gòu)建，需要一種方法，讓人們通過提出全局性的問題來應(yīng)用和完善他們對數(shù)據(jù)的心理模型(Klein等人，2006b)。

Retrieval-augmented generation (RAG, Lewis et al., 2020) is an established approach to answering user questions over entire datasets, but it is designed for situations where these answers are contained locally within regions of text whose retrieval provides sufficient grounding for the generation task. Instead, a more appropriate task framing is query-focused summarization (QFS, Dang, 2006), and in particular, query-focused abstractive summarization that generates natural language summaries and not just concatenated excerpts (Baumel et al., 2018; Laskar et al., 2020; Yao et al., 2017) . In recent years, however, such distinctions between summarization tasks that are abstractive versus extractive, generic versus query-focused, and single-document versus multi-document, have become less rele-vant. While early applications of the transformer architecture showed substantial improvements on the state-of-the-art for all such summarization tasks (Goodwin et al., 2020; Laskar et al., 2022; Liu and Lapata, 2019), these tasks are now trivialized by modern LLMs, including the GPT (Achiam et al., 2023; Brown et al., 2020), Llama (Touvron et al., 2023), and Gemini (Anil et al., 2023) series, all of which can use in-context learning to summarize any content provided in their context window.

檢索增強生成(RAG, Lewis等人，2020)是一種針對整個數(shù)據(jù)集回答用戶問題的既定方法，但它是為這些答案局部包含在文本區(qū)域內(nèi)的情況而設(shè)計的，這些文本區(qū)域的檢索為生成任務(wù)提供了足夠的基礎(chǔ)。相反，更合適的任務(wù)框架是以查詢?yōu)橹行牡恼?/strong>(QFS, Dang, 2006)，特別是以查詢?yōu)橹行牡某橄笳?#xff0c;它生成自然語言摘要，而不僅僅是連接的摘錄(Baumel等人，2018;Laskar et al.， 2020;Yao等人，2017)。然而，近年來，抽象與抽取、通用與以查詢?yōu)橹行摹挝臋n與多文檔的摘要任務(wù)之間的區(qū)別已經(jīng)變得不那么重要了。雖然transformer架構(gòu)的早期應(yīng)用在所有此類匯總?cè)蝿?wù)上都顯示出了巨大的進步(Goodwin et al.， 2020;Laskar et al.， 2022;Liu和Lapata, 2019)，這些任務(wù)現(xiàn)在被現(xiàn)代LLMs簡化了，包括GPT (Achiam等人，2023;Brown et al.， 2020)， Llama (Touvron et al.， 2023)和Gemini (Anil et al.， 2023)系列，所有這些都可以使用上下文學(xué)習(xí)來總結(jié)上下文窗口中提供的任何內(nèi)容。

The challenge remains, however, for query-focused abstractive summarization over an entire corpus. Such volumes of text can greatly exceed the limits of LLM?context windows, and the expansion of such windows may not be enough given that information can be “l(fā)ost in the middle” of longer contexts (Kuratov et al., 2024; Liu et al., 2023). In addition, although the direct retrieval of text chunks in 簡單的RAG?is likely inadequate for QFS tasks, it is possible that an alternative form of pre-indexing could support a new RAG approach specifically targeting global summarization.

然而，對于整個語料庫的以查詢?yōu)橹行牡某橄笳獊碚f，挑戰(zhàn)仍然存在。這樣的文本量可以大大超過LLM上下文窗口的限制，并且考慮到信息可能會“丟失在中間”的較長的上下文，這樣的窗口的擴展可能是不夠的(Kuratov等人，2024;Liu et al.， 2023)。此外，盡管在簡單的RAG中直接檢索文本塊可能不適合QFS任務(wù)，但是一種替代形式的預(yù)索引可能支持專門針對全局摘要的新RAG方法。

In this paper, we present a Graph RAG approach based on global summarization of an LLM-derived knowledge graph (Figure 1). In contrast with related work that exploits the structured retrieval and traversal affordances of graph indexes (subsection 4.2), we focus on a previously unexplored quality of graphs in this context: their inherent modularity (Newman, 2006) and the ability of com-munity detection algorithms to partition graphs into modular communities of closely-related nodes (e.g., Louvain, Blondel et al., 2008; Leiden, Traag et al., 2019). LLM-generated summaries of these?community descriptions provide complete coverage of the underlying graph index and the input doc-uments it represents. Query-focused summarization of an entire corpus is then made possible using a map-reduce approach: first using each community summary to answer the query independently and in parallel, then summarizing all relevant partial answers into a final global answer.

在本文中，我們提出了一種基于LLM派生的知識圖的全局總結(jié)的Graph RAG方法(圖1)。與利用圖索引的結(jié)構(gòu)化檢索和遍歷可視性的相關(guān)工作(第4.2節(jié))相比，我們專注于在此背景下以前未探索的圖的質(zhì)量:它們固有的模塊化(Newman, 2006)以及社區(qū)檢測算法將圖劃分為密切相關(guān)節(jié)點的模塊化社區(qū)的能力(例如，Louvain, Blondel等人，2008;萊頓，Traag等人，2019)。LLM生成的這些社區(qū)描述的摘要提供了底層圖形索引及其所代表的輸入文檔的完整覆蓋。然后，可以使用map-reduce方法對整個語料庫進行以查詢?yōu)橹行牡膮R總:首先使用每個社區(qū)匯總來獨立地并行地回答查詢，然后將所有相關(guān)的部分答案匯總為最終的全局答案。

To evaluate this approach, we used an LLM?to generate a diverse set of activity-centered sense-making questions from short descriptions of two representative real-world datasets, containing pod-cast transcripts and news articles respectively. For the target qualities of comprehensiveness, diver-sity, and empowerment (defined in subsection 3.4) that develop understanding of broad issues and themes, we both explore the impact of varying the the hierarchical level of community summaries used to answer queries, as well as compare to 簡單的RAG?and global map-reduce summarization of source texts. We show that all global approaches outperform 簡單的RAG?on comprehensiveness and diversity, and that Graph RAG with intermediate- and low-level community summaries shows favorable performance over source text summarization on these same metrics, at lower token costs.

為了評估這種方法，我們使用LLM從兩個具有代表性的真實世界數(shù)據(jù)集的簡短描述中生成了一組以活動為中心的意義構(gòu)建問題，這些數(shù)據(jù)集分別包含播客文稿和新聞文章。對于發(fā)展對廣泛問題和主題的理解的綜合性、多樣性和賦權(quán)(在第3.4小節(jié)中定義)的目標質(zhì)量，我們都探索了用于回答查詢的不同社區(qū)摘要的層次水平的影響，并與簡單的RAG和源文本的全球地圖減少摘要進行了比較。我們表明，所有全局方法在全面性和多樣性方面都優(yōu)于簡單的RAG，并且具有中級和低級社區(qū)摘要的Graph RAG在這些相同的指標上以更低的令牌成本顯示出比源文本摘要更好的性能。

We now unpack the high-level data flow of the Graph RAG approach (Figure 1) and pipeline, de-scribing key design parameters, techniques, and implementation details for each step.

現(xiàn)在我們解壓縮Graph RAG方法的高級數(shù)據(jù)流(圖1)和管道，描述每個步驟的關(guān)鍵設(shè)計參數(shù)、技術(shù)和實現(xiàn)細節(jié)。

A fundamental design decision is the granularity with which input texts extracted from source doc-uments should be split into text chunks for processing. In the following step, each of these chunks will be passed to a set of LLM?prompts designed to extract the various elements of a graph index. Longer text chunks require fewer LLM?calls for such extraction, but suffer from the recall degrada-tion of longer LLM?context windows (Kuratov et al., 2024; Liu et al., 2023). This behavior can be observed in Figure 2 in the case of a single extraction round (i.e., with zero gleanings): on a sample dataset (HotPotQA, Yang et al., 2018), using a chunk size of 600 token extracted almost twice as many entity references as when using a chunk size of 2400. While more references are generally better, any extraction process needs to balance recall and precision for the target activity.

一個基本的設(shè)計決策是將從源文檔中提取的輸入文本分割成文本塊進行處理的粒度。在接下來的步驟中，每個塊都將傳遞給一組LLM提示符，這些提示符旨在提取圖索引的各種元素。較長的文本塊需要較少的LLM調(diào)用來進行這種提取，但較長的LLM上下文窗口會導(dǎo)致召回率下降(Kuratov等人，2024;Liu et al.， 2023)。在單個提取輪(即零收集)的情況下，可以在圖2中觀察到這種行為:在樣本數(shù)據(jù)集(HotPotQA, Yang等人，2018)上，使用塊大小為600的令牌提取的實體引用幾乎是使用塊大小為2400時的兩倍。雖然引用越多越好，但任何提取過程都需要平衡目標活動的召回率和精度。

The baseline requirement for this step is to identify and extract instances of graph nodes and edges from each chunk of source text. We do this using a multipart LLM?prompt that first identifies all entities in the text, including their name, type, and description, before identifying all relationships between clearly-related entities, including the source and target entities and a description of their relationship. Both kinds of element instance are output in a single list of delimited tuples.

The primary opportunity to tailor this prompt to the domain of the document corpus lies in the choice of few-shot examples provided to the LLM?for in-context learning (Brown et al., 2020).?For example, while our default prompt extracting the broad class of “named entities” like people, places, and organizations is generally applicable, domains with specialized knowledge (e.g., science, medicine, law) will benefit from few-shot examples specialized to those domains. We also support a secondary extraction prompt for any additional covariates we would like to associate with the extracted node instances. Our default covariate prompt aims to extract claims linked to detected entities, including the subject, object, type, description, source text span, and start and end dates.

這一步的基本要求是從每個源文本塊中識別和提取圖節(jié)點和邊的實例。我們使用多部分LLM提示符來完成此操作，該提示符首先識別文本中的所有實體，包括它們的名稱、類型和描述，然后識別明確相關(guān)實體之間的所有關(guān)系，包括源實體和目標實體以及它們之間關(guān)系的描述。這兩種類型的元素實例都輸出在單個分隔元組列表中。

將此提示定制為文檔語料庫領(lǐng)域的主要機會在于選擇提供給LLMs進行上下文學(xué)習(xí)的少量示例(Brown et al.， 2020)。例如，雖然我們的默認提示提取“命名實體”(如人員、地點和組織)的廣泛類別通常是適用的，但具有專門知識的領(lǐng)域(例如，科學(xué)、醫(yī)學(xué)、法律)將受益于專門針對這些領(lǐng)域的少量示例。對于我們希望與提取的節(jié)點實例相關(guān)聯(lián)的任何其他協(xié)變量，我們還支持輔助提取提示符。我們默認的協(xié)變量提示旨在提取與檢測到的實體相關(guān)聯(lián)的聲明，包括主題、對象、類型、描述、源文本跨度以及開始和結(jié)束日期。

To balance the needs of efficiency and quality, we use multiple rounds of “gleanings”, up to a specified maximum, to encourage the LLM?to detect any additional entities it may have missed on prior extraction rounds. This is a multi-stage process in which we first ask the LLM?to assess whether all entities were extracted, using a logit bias of 100 to force a yes/no decision. If the LLM?responds that entities were missed, then a continuation indicating that “MANY entities were missed in the last extraction” encourages the LLM?to glean these missing entities. This approach allows us to use larger chunk sizes without a drop in quality (Figure 2) or the forced introduction of noise.

為了平衡效率和質(zhì)量的需要，我們使用多輪“收集”，直到指定的最大值，以鼓勵LLM檢測之前提取輪次中可能遺漏的任何其他實體。這是一個多階段的過程，我們首先要求LLM評估是否提取了所有實體，使用100的logit偏差來強制做出是/否的決定。如果LLM響應(yīng)實體丟失了，那么指示“在上次提取中丟失了許多實體”的延續(xù)將鼓勵LLM收集這些丟失的實體。這種方法允許我們使用更大的塊大小，而不會降低質(zhì)量(圖2)或強制引入噪聲。

The use of an LLM?to “extract” descriptions of entities, relationships, and claims represented in source texts is already a form of abstractive summarization, relying on the LLM?to create inde-pendently meaningful summaries of concepts that may be implied but not stated by the text itself (e.g., the presence of implied relationships). To convert all such instance-level summaries into sin-gle blocks of descriptive text for each graph element (i.e., entity node, relationship edge, and claim covariate) requires a further round of LLM?summarization over matching groups of instances.

使用LLM來“提取”源文本中表示的實體、關(guān)系和聲明的描述已經(jīng)是一種抽象摘要的形式，依靠LLM來創(chuàng)建可能隱含但未由文本本身說明的概念的獨立有意義的摘要(例如，隱含關(guān)系的存在)。要將所有這樣的實例級摘要轉(zhuǎn)換為每個圖元素(即實體節(jié)點、關(guān)系邊緣和索賠協(xié)變量)的單個描述性文本塊，需要對匹配的實例組進行進一步的LLM摘要。

A potential concern at this stage is that the LLM?may not consistently extract references to the same entity in the same text format, resulting in duplicate entity elements and thus duplicate nodes in the entity graph. However, since all closely-related “communities” of entities will be detected and summarized in the following step, and given that LLMs can understand the common entity behind multiple name variations, our overall approach is resilient to such variations provided there is sufficient connectivity from all variations to a shared set of closely-related entities.

Overall, our use of rich descriptive text for homogeneous nodes in a potentially noisy graph structure is aligned with both the capabilities of LLMs and the needs of global, query-focused summarization. These qualities also differentiate our graph index from typical knowledge graphs, which rely on concise and consistent knowledge triples (subject, predicate, object) for downstream reasoning tasks.

這個階段的一個潛在問題是，LLM可能無法始終如一地以相同的文本格式提取對同一實體的引用，從而導(dǎo)致重復(fù)的實體元素，從而導(dǎo)致實體圖中的重復(fù)節(jié)點。然而，由于所有密切相關(guān)的實體“社區(qū)”將在接下來的步驟中被檢測和總結(jié)，并且考慮到LLM可以理解多個名稱變化背后的共同實體，我們的整體方法對于這些變化是有彈性的，只要所有變化與一組共享的密切相關(guān)的實體有足夠的連接。

The index created in the previous step can be modelled as an homogeneous undirected weighted graph in which entity nodes are connected by relationship edges, with edge weights representing the normalized counts of detected relationship instances. Given such a graph, a variety of community detection algorithms may be used to partition the graph into communities of nodes with stronger connections to one another than to the other nodes in the graph (e.g., see the surveys by Fortu-nato, 2010 and Jin et al., 2021). In our pipeline, we use Leiden (Traag et al., 2019) on account of its ability to recover hierarchical community structure of large-scale graphs efficiently (Figure 3). Each level of this hierarchy provides a community partition that covers the nodes of the graph in a mutually-exclusive, collective-exhaustive way, enabling divide-and-conquer global summarization.

在前一步中創(chuàng)建的索引可以建模為一個同構(gòu)無向加權(quán)圖，其中實體節(jié)點通過關(guān)系邊連接，邊的權(quán)重表示檢測到的關(guān)系實例的規(guī)范化計數(shù)。給定這樣一個圖，可以使用各種社區(qū)檢測算法將圖劃分為節(jié)點社區(qū)，這些節(jié)點之間的連接比圖中其他節(jié)點之間的連接更強(例如，參見fortune -nato, 2010和Jin et al.， 2021的調(diào)查)。在我們的管道中，我們使用Leiden (Traag等人，2019)，因為它能夠有效地恢復(fù)大規(guī)模圖的分層社區(qū)結(jié)構(gòu)(圖3)。該層次結(jié)構(gòu)的每個級別都提供了一個社區(qū)分區(qū)，該分區(qū)以互斥的、集體詳盡的方式覆蓋圖的節(jié)點，從而實現(xiàn)分而治之的全局總結(jié)。

The next step is to create report-like summaries of each community in the Leiden hierarchy, using a method designed to scale to very large datasets. These summaries are independently useful in their own right as a way to understand the global structure and semantics of the dataset, and may themselves be used to make sense of a corpus in the absence of a question. For example, a user may scan through community summaries at one level looking for general themes of interest, then follow links to the reports at the lower level that provide more details for each of the subtopics. Here, however, we focus on their utility as part of a graph-based index used for answering global queries.Community summaries are generated in the following way:

下一步是使用一種旨在擴展到非常大的數(shù)據(jù)集的方法，為Leiden層次結(jié)構(gòu)中的每個社區(qū)創(chuàng)建類似報告的摘要。這些摘要作為理解數(shù)據(jù)集的整體結(jié)構(gòu)和語義的一種方式，它們本身是獨立有用的，并且可以在沒有問題的情況下用于理解語料庫。例如，用戶可以瀏覽某一級別的社區(qū)摘要，尋找感興趣的一般主題，然后點擊指向較低級別報告的鏈接，這些鏈接為每個子主題提供了更多詳細信息。然而，在這里，我們關(guān)注的是它們作為用于回答全局查詢的基于圖的索引的一部分的效用。社區(qū)摘要以以下方式生成:

Given a user query, the community summaries generated in the previous step can be used to generate a final answer in a multi-stage process. The hierarchical nature of the community structure also means that questions can be answered using the community summaries from different levels, raising the question of whether a particular level in the hierarchical community structure offers the best balance of summary detail and scope for general sensemaking questions (evaluated in section 3).

給定一個用戶查詢，在前一步中生成的社區(qū)摘要可用于在多階段流程中生成最終答案。社區(qū)結(jié)構(gòu)的層次性也意味著可以使用來自不同層次的社區(qū)摘要來回答問題，這就提出了這樣一個問題:在層次化社區(qū)結(jié)構(gòu)中，某個特定的層次是否提供了概要細節(jié)和一般性問題范圍的最佳平衡(在第3節(jié)中進行了評估)。

For a given community level, the global answer to any user query is generated as follows:

>> Prepare community summaries. Community summaries are randomly shuffled and divided into chunks of pre-specified token size. This ensures relevant information is distributed across chunks, rather than concentrated (and potentially lost) in a single context window.

>> Map community answers. Generate intermediate answers in parallel, one for each chunk.The LLM?is also asked to generate a score between 0-100 indicating how helpful the gen-erated answer is in answering the target question. Answers with score 0 are filtered out.

>> Reduce to global answer. Intermediate community answers are sorted in descending order of helpfulness score and iteratively added into a new context window until the token limit is reached. This final context is used to generate the global answer returned to the user.

對于給定的社區(qū)級別，生成任何用戶查詢的全局答案如下:

>>準備社區(qū)總結(jié)。社區(qū)摘要被隨機打亂并分成預(yù)先指定的令牌大小的塊。這確保了相關(guān)信息分布在各個塊之間，而不是集中在單個上下文窗口中(并且可能丟失)。

>>地圖社區(qū)答案。并行生成中間答案，每個塊一個。LLMs還被要求生成一個0-100分之間的分數(shù)，表明生成的答案對回答目標問題的幫助程度。得分為0的答案將被過濾掉。

>>減少到全局答案。中間社區(qū)答案按有用性分數(shù)降序排序，并迭代地添加到新的上下文窗口中，直到達到令牌限制。最后一個上下文用于生成返回給用戶的全局答案。

We selected two datasets in the one million token range, each equivalent to about 10 novels of text and representative of the kind of corpora that users may encounter in their real world activities:

>> Podcast transcripts. Compiled transcripts of podcast conversations between Kevin Scott, Microsoft CTO, and other technology leaders (Behind the Tech, Scott, 2024). Size: 1669 × 600-token text chunks, with 100-token overlaps between chunks (～1 million tokens).

>> News articles. Benchmark dataset comprising news articles published from September 2013 to December 2023 in a range of categories, including entertainment, business, sports, technology, health, and science (MultiHop-RAG; Tang and Yang, 2024). Size: 3197 × 600-token text chunks, with 100-token overlaps between chunks (～1.7 million tokens).

我們在100萬個令牌范圍內(nèi)選擇了兩個數(shù)據(jù)集，每個數(shù)據(jù)集相當于大約10本小說的文本，代表了用戶在現(xiàn)實世界活動中可能遇到的語料庫類型:

>>播客文本。匯編了微軟首席技術(shù)官凱文·斯科特與其他技術(shù)領(lǐng)袖之間的播客對話記錄(《科技背后》，斯科特，2024年)。大小:1669 × 600個令牌文本塊，塊之間有100個令牌重疊(約100萬個令牌)。

新聞文章?；鶞蕯?shù)據(jù)集包括從2013年9月到2023年12月在一系列類別中發(fā)布的新聞文章，包括娛樂，商業(yè)，體育，技術(shù)，健康和科學(xué)(MultiHop-RAG;Tang and Yang, 2024)。大小:3197 × 600個令牌文本塊，塊之間有100個令牌重疊(約170萬個令牌)。

Many benchmark datasets for open-domain question answering exist, including HotPotQA (Yang et al., 2018), MultiHop-RAG (Tang and Yang, 2024), and MT-Bench (Zheng et al., 2024). However, the associated question sets target explicit fact retrieval rather than summarization for the purpose of data sensemaking, i.e., the process though which people inspect, engage with, and contextualize data within the broader scope of real-world activities (Koesten et al., 2021). Similarly, methods for extracting latent summarization queries from source texts also exist (Xu and Lapata, 2021), but such extracted questions can target details that betray prior knowledge of the texts.

目前存在許多開放域問答的基準數(shù)據(jù)集，包括HotPotQA (Yang等人，2018)、MultiHop-RAG (Tang和Yang, 2024)和MT-Bench (Zheng等人，2024)。然而，相關(guān)的問題集以明確的事實檢索為目標，而不是以數(shù)據(jù)語義為目的的總結(jié)，即人們在更廣泛的現(xiàn)實世界活動范圍內(nèi)檢查、參與和情境化數(shù)據(jù)的過程(Koesten et al.， 2021)。同樣，從源文本中提取潛在摘要查詢的方法也存在(Xu和Lapata, 2021)，但這些提取的問題可能針對背叛文本先驗知識的細節(jié)。

To evaluate the effectiveness of RAG systems for more global sensemaking tasks, we need questions that convey only a high-level understanding of dataset contents, and not the details of specific texts. We used an activity-centered approach to automate the generation of such questions: given a short description of a dataset, we asked the LLM to identify N potential users and N tasks per user, then for each (user, task) combination, we asked the LLM to generate N questions that require understanding of the entire corpus. For our evaluation, a value of N = 5 resulted in 125 test questions per dataset. Table 1 shows example questions for each of the two evaluation datasets.

為了評估RAG系統(tǒng)在更多全局意義生成任務(wù)中的有效性，我們需要的問題只傳達對數(shù)據(jù)集內(nèi)容的高層次理解，而不是特定文本的細節(jié)。我們使用以活動為中心的方法來自動生成此類問題:給定數(shù)據(jù)集的簡短描述，我們要求LLM識別N個潛在用戶和每個用戶的N個任務(wù)，然后對于每個(用戶，任務(wù))組合，我們要求LLM生成N個需要理解整個語料庫的問題。對于我們的評估，N = 5的值導(dǎo)致每個數(shù)據(jù)集有125個測試問題。表1顯示了兩個評估數(shù)據(jù)集的示例問題。

We compare six different conditions in our analysis, including Graph RAG using four levels of graph communities (C0, C1, C2, C3), a text summarization method applying our map-reduce approach directly to source texts (TS), and a na¨?ve “semantic search” RAG approach (SS):

在我們的分析中，我們比較了六種不同的情況，包括使用四個級別的圖社區(qū)(C0, C1, C2, C3)的Graph RAG，直接應(yīng)用我們的map-reduce方法到源文本的文本摘要方法(TS)，以及na¨?ve“語義搜索”RAG方法(SS)。

LLMs have been shown to be good evaluators of natural language generation, achieving state-of-the-art or competitive results compared against human judgements (Wang et al., 2023a; Zheng et al., 2024). While this approach can generate reference-based metrics when gold standard answers are known, it is also capable of measuring the qualities of generated texts (e.g., fluency) in a reference-free style (Wang et al., 2023a) as well as in head-to-head comparison of competing outputs (LLM-as-a-judge, Zheng et al., 2024). LLMs have also shown promise at evaluating the performance of conventional RAG systems, automatically evaluating qualities like context relevance, faithfulness, and answer relevance (RAGAS, Es et al., 2023).

LLMs已被證明是自然語言生成的良好評估者，與人類判斷相比，取得了最先進或具有競爭力的結(jié)果(Wang等人，2023a;鄭等人，2024)。雖然這種方法可以在黃金標準答案已知的情況下生成基于參考的指標，但它也能夠以無參考的方式(Wang等人，2023a)測量生成文本的質(zhì)量(例如流暢性)，以及對競爭輸出進行正面比較(LLM-as-a-judge, Zheng等人，2024)。LLMs在評估傳統(tǒng)RAG系統(tǒng)的性能方面也表現(xiàn)出了希望，自動評估上下文相關(guān)性、可信度和答案相關(guān)性等質(zhì)量(RAGAS, Es等人，2023)。

The indexing process resulted in a graph consisting of 8564 nodes and 20691 edges for the Podcast dataset, and a larger graph of 15754 nodes and 19520 edges for the News dataset. Table 3 shows the number of community summaries at different levels of each graph community hierarchy.

Global approaches vs. 簡單的RAG. As shown in Figure 4, global approaches consistently out-performed the 簡單的RAG?(SS) approach in both comprehensiveness and diversity metrics across datasets. Specifically, global approaches achieved comprehensiveness win rates between 72-83%for Podcast transcripts and 72-80% for News articles, while diversity win rates ranged from 75-82%and 62-71% respectively. Our use of directness as a validity test also achieved the expected results,??e., that 簡單的RAG produces the most direct responses across all comparisons.

索引過程的結(jié)果是Podcast數(shù)據(jù)集的圖由8564個節(jié)點和20691條邊組成，News數(shù)據(jù)集的圖由15754個節(jié)點和19520條邊組成。表3顯示了每個圖社區(qū)層次結(jié)構(gòu)中不同級別的社區(qū)摘要數(shù)量。

全球方法vs. ?ve RAG。如圖4所示，在數(shù)據(jù)集的全面性和多樣性指標方面，全局方法始終優(yōu)于簡單的RAG?(SS)方法。具體而言，全球方法在播客文本和新聞文章上的綜合勝率分別為72-83%和72-80%，而多樣性勝率分別為75-82%和62-71%。我們使用直接性作為效度測試也達到了預(yù)期的結(jié)果，即簡單的RAG在所有比較中產(chǎn)生最直接的反應(yīng)。

Community summaries vs. source texts. When comparing community summaries to source texts using Graph RAG, community summaries generally provided a small but consistent improvement in answer comprehensiveness and diversity, except for root-level summaries. Intermediate-level summaries in the Podcast dataset and low-level community summaries in the News dataset achieved comprehensiveness win rates of 57% and 64%, respectively. Diversity win rates were 57% for Podcast intermediate-level summaries and 60% for News low-level community summaries. Table 3 also illustrates the scalability advantages of Graph RAG compared to source text summarization: for low-level community summaries (C3), Graph RAG required 26-33% fewer context tokens, while for root-level community summaries (C0), it required over 97% fewer tokens. For a modest drop in performance compared with other global methods, root-level Graph RAG offers a highly efficient method for the iterative question answering that characterizes sensemaking activity, while retaining advantages in comprehensiveness (72% win rate) and diversity (62% win rate) over 簡單的RAG.

社區(qū)摘要vs.源文本。當使用Graph RAG將社區(qū)摘要與源文本進行比較時，除了根級摘要外，社區(qū)摘要通常在答案的全面性和多樣性方面提供了小而一致的改進。Podcast數(shù)據(jù)集中的中級摘要和News數(shù)據(jù)集中的低級社區(qū)摘要的綜合勝率分別為57%和64%。播客中級總結(jié)的多樣性勝率為57%，新聞低級社區(qū)總結(jié)的多樣性勝率為60%。表3還說明了與源文本摘要相比，Graph RAG的可伸縮性優(yōu)勢:對于低級社區(qū)摘要(C3)， Graph RAG需要的上下文令牌減少了26-33%，而對于根級社區(qū)摘要(C0)，它需要的令牌減少了97%以上。與其他全局方法相比，在性能上略有下降的情況下，根級圖RAG提供了一種高效的迭代問題回答方法，該方法具有意義生成活動的特征，同時保留了比簡單的RAG在全面性(72%勝率)和多樣性(62%勝率)方面的優(yōu)勢。

Empowerment. Empowerment comparisons showed mixed results for both global approaches versus 簡單的RAG?(SS) and Graph RAG approaches versus source text summarization (TS). Ad-hoc LLM?use to analyze LLM?reasoning for this measure indicated that the ability to provide specific exam-ples, quotes, and citations was judged to be key to helping users reach an informed understanding. Tuning element extraction prompts may help to retain more of these details in the Graph RAG index.

賦權(quán)。授權(quán)比較顯示，全局方法與簡單的RAG?(SS)和圖形RAG方法與源文本摘要(TS)的結(jié)果不同。專門使用LLMs來分析LLMs對這一度量的推理表明，提供具體示例、引用和引用的能力被認為是幫助用戶獲得知情理解的關(guān)鍵。調(diào)優(yōu)元素提取提示可能有助于在Graph RAG索引中保留更多這些細節(jié)。

When using LLMs, RAG involves first retrieving relevant information from external data sources, then adding this information to the context window of the LLM?along with the original query (Ram et al., 2023). 簡單的RAG?approaches (Gao et al., 2023) do this by converting documents to text, splitting text into chunks, and embedding these chunks into a vector space in which similar positions represent similar semantics. Queries are then embedded into the same vector space, with the text chunks of the nearest k vectors used as context. More advanced variations exist, but all solve the problem of what to do when an external dataset of interest exceeds the LLM’s context window.

當使用LLM時，RAG首先涉及從外部數(shù)據(jù)源檢索相關(guān)信息，然后將此信息與原始查詢一起添加到LLM的上下文窗口(Ram等人，2023)。簡單的RAG方法(Gao et al.， 2023)通過將文檔轉(zhuǎn)換為文本，將文本分割成塊，并將這些塊嵌入到向量空間中，其中相似的位置表示相似的語義來實現(xiàn)這一點。然后將查詢嵌入到相同的向量空間中，使用最近k個向量的文本塊作為上下文。存在更高級的變體，但都解決了當感興趣的外部數(shù)據(jù)集超出LLM的上下文窗口時該怎么辦的問題。

Advanced RAG systems include pre-retrieval, retrieval, post-retrieval strategies designed to over-come the drawbacks of 簡單的RAG, while Modular RAG systems include patterns for iterative and dynamic cycles of interleaved retrieval and generation (Gao et al., 2023). Our implementation of Graph RAG incorporates multiple concepts related to other systems. For example, our community summaries are a kind of self-memory (Selfmem, Cheng et al., 2024) for generation-augmented re-trieval (GAR, Mao et al., 2020) that facilitates future generation cycles, while our parallel generation of community answers from these summaries is a kind of iterative (Iter-RetGen, Shao et al., 2023) or federated (FeB4RAG, Wang et al., 2024) retrieval-generation strategy. Other systems have also combined these concepts for multi-document summarization (CAiRE-COVID, Su et al., 2020) and multi-hop question answering (ITRG, Feng et al., 2023; IR-CoT, Trivedi et al., 2022; DSP, Khattab et al., 2022). Our use of a hierarchical index and summarization also bears resemblance to further approaches, such as generating a hierarchical index of text chunks by clustering the vectors of text embeddings (RAPTOR, Sarthi et al., 2024) or generating a “tree of clarifications” to answer mul-tiple interpretations of ambiguous questions (Kim et al., 2023). However, none of these iterative or hierarchical approaches use the kind of self-generated graph index that enables Graph RAG.

先進的RAG系統(tǒng)包括預(yù)檢索、檢索和后檢索策略，旨在克服簡單的RAG的缺點，而模塊化RAG系統(tǒng)包括交錯檢索和生成的迭代和動態(tài)循環(huán)模式(Gao等人，2023)。我們對Graph RAG的實現(xiàn)包含了與其他系統(tǒng)相關(guān)的多個概念。例如，我們的社區(qū)摘要是一種自我記憶(Selfmem, Cheng等人，2024)，用于世代增強檢索(GAR, Mao等人，2020)，有利于未來的世代循環(huán)，而我們從這些摘要中并行生成社區(qū)答案是一種迭代(ter- retgen, Shao等人，2023)或聯(lián)合(FeB4RAG, Wang等人，2024)檢索生成策略。其他系統(tǒng)也將這些概念結(jié)合起來用于多文檔摘要(cire - covid, Su等人，2020)和多跳問答(ITRG, Feng等人，2023;IR-CoT, Trivedi等，2022;DSP, Khattab et al.， 2022)。我們對層次索引和摘要的使用也與進一步的方法相似，例如通過聚類文本嵌入向量來生成文本塊的層次索引(RAPTOR, Sarthi等人，2024)或生成“澄清樹”來回答對歧義問題的多種解釋(Kim等人，2023)。然而，這些迭代或分層方法都沒有使用支持graph RAG的自生成圖索引。

Use of graphs in connection with LLMs and RAG is a developing research area, with multiple directions already established. These include using LLMs for knowledge graph creation (Tra-janoska et al., 2023) and completion (Yao et al., 2023), as well as for the extraction of causal graphs (Ban et al., 2023; Zhang et al., 2024) from source texts. They also include forms of ad-vanced RAG (Gao et al., 2023) where the index is a knowledge graph (KAPING, Baek et al., 2023), where subsets of the graph structure (G-Retriever, He et al., 2024) or derived graph metrics (Graph-ToolFormer, Zhang, 2023) are the objects of enquiry, where narrative outputs are strongly grounded in the facts of retrieved subgraphs (SURGE, Kang et al., 2023), where retrieved event-plot sub-graphs are serialized using narrative templates (FABULA, Ranade and Joshi, 2023), and where the system supports both creation and traversal of text-relationship graphs for multi-hop question an-swering (Wang et al., 2023b). In terms of open-source software, a variety a graph databases are supported by both the LangChain (LangChain, 2024) and LlamaIndex (LlamaIndex, 2024) libraries, while a more general class of graph-based RAG applications is also emerging, including systems that can create and reason over knowledge graphs in both Neo4J (NaLLM, Neo4J, 2024) and Nebula-Graph (GraphRAG, NebulaGraph, 2024) formats. Unlike our Graph RAG approach, however, none of these systems use the natural modularity of graphs to partition data for global summarization.

在LLMs和RAG中使用圖形是一個發(fā)展中的研究領(lǐng)域，已經(jīng)建立了多個方向。其中包括使用LLMs創(chuàng)建知識圖譜(Tra-janoska等人，2023)和完成知識圖譜(Yao等人，2023)，以及提取因果圖(Ban等人，2023;Zhang et al.， 2024)。它們還包括高級RAG的形式(Gao等人，2023)，其中索引是一個知識圖(KAPING, Baek等人，2023)，其中圖結(jié)構(gòu)的子集(g - retriver, He等人，2024)或派生的圖度量(graph - toolformer, Zhang, 2023)是查詢對象，其中敘事輸出強烈地基于檢索子圖的事實(SURGE, Kang等人，2023)，其中檢索的事件情節(jié)子圖使用敘事模板(FABULA, Ranade和Joshi)序列化。2023)，其中系統(tǒng)支持多跳問答的文本關(guān)系圖的創(chuàng)建和遍歷(Wang et al.， 2023b)。在開源軟件方面，LangChain (LangChain, 2024)和LlamaIndex (LlamaIndex, 2024)庫都支持多種圖形數(shù)據(jù)庫，而更通用的基于圖形的RAG應(yīng)用程序也正在興起，包括可以在Neo4J (NaLLM, Neo4J, 2024)和星云圖(GraphRAG，星云圖，2024)格式下創(chuàng)建和推理知識圖的系統(tǒng)。然而，與我們的Graph RAG方法不同，這些系統(tǒng)都沒有使用圖的自然模塊化來劃分數(shù)據(jù)以進行全局匯總。

Limitations of evaluation approach. Our evaluation to date has only examined a certain class of sensemaking questions for two corpora in the region of 1 million tokens. More work is needed to understand how performance varies across different ranges of question types, data types, and dataset sizes, as well as to validate our sensemaking questions and target metrics with end users. Comparison of fabrication rates, e.g., using approaches like SelfCheckGPT (Manakul et al., 2023), would also improve on the current analysis.

評價方法的局限性。到目前為止，我們的評估僅檢查了100萬個令牌區(qū)域中兩個語料庫的某類語義問題。需要做更多的工作來了解不同問題類型、數(shù)據(jù)類型和數(shù)據(jù)集大小范圍的性能變化，以及與最終用戶驗證我們的語義問題和目標指標。比較偽造率，例如，使用SelfCheckGPT (Manakul等人，2023)等方法，也將改進當前的分析。

Trade-offs of building a graph index. We consistently observed Graph RAG achieve the best head-to-head results against other methods, but in many cases the graph-free approach to global summa-rization of source texts performed competitively. The real-world decision about whether to invest in building a graph index depends on multiple factors, including the compute budget, expected number of lifetime queries per dataset, and value obtained from other aspects of the graph index (including the generic community summaries and the use of other graph-related RAG approaches).

構(gòu)建圖表索引的利弊權(quán)衡。我們一直觀察到，Graph RAG與其他方法相比獲得了最佳的正面結(jié)果，但在許多情況下，無圖方法對源文本進行全局匯總具有競爭性。關(guān)于是否投資構(gòu)建圖索引的實際決策取決于多個因素，包括計算預(yù)算、每個數(shù)據(jù)集的預(yù)期生命周期查詢數(shù)量，以及從圖索引的其他方面獲得的值(包括通用社區(qū)摘要和其他與圖相關(guān)的RAG方法的使用)。

Future work. The graph index, rich text annotations, and hierarchical community structure support-ing the current Graph RAG approach offer many possibilities for refinement and adaptation. This includes RAG approaches that operate in a more local manner, via embedding-based matching of user queries and graph annotations, as well as the possibility of hybrid RAG schemes that combine embedding-based matching against community reports before employing our map-reduce summa-rization mechanisms. This “roll-up” operation could also be extended across more levels of the community hierarchy, as well as implemented as a more exploratory “drill down” mechanism that follows the information scent contained in higher-level community summaries.

未來的工作。支持當前graph RAG方法的圖索引、富文本注釋和分層社區(qū)結(jié)構(gòu)為改進和調(diào)整提供了許多可能性。這包括以更局部的方式操作的RAG方法，通過基于嵌入的用戶查詢匹配和圖形注釋，以及混合RAG方案的可能性，該方案在使用我們的map-reduce匯總機制之前將基于嵌入的匹配與社區(qū)報告結(jié)合起來。這種“上卷”操作還可以擴展到社區(qū)層次結(jié)構(gòu)的更多級別，也可以作為一種更具探索性的“下鉆”機制來實現(xiàn)，該機制遵循更高級別社區(qū)摘要中包含的信息氣味。

We have presented a global approach to Graph RAG, combining knowledge graph generation, retrieval-augmented generation (RAG), and query-focused summarization (QFS) to support human sensemaking over entire text corpora. Initial evaluations show substantial improvements over a 簡單的RAG?baseline for both the comprehensiveness and diversity of answers, as well as favorable comparisons to a global but graph-free approach using map-reduce source text summarization. For situations requiring many global queries over the same dataset, summaries of root-level communi-ties in the entity-based graph index provide a data index that is both superior to 簡單的RAG?and achieves competitive performance to other global methods at a fraction of the token cost.

我們提出了一種全局的Graph RAG方法，將知識圖生成、檢索增強生成(RAG)和以查詢?yōu)橹行牡牟樵兙劢拐?QFS)相結(jié)合，以支持人類對整個文本語料庫的意義理解。初步評估表明，在答案的全面性和多樣性方面，相較于簡單的RAG基線有實質(zhì)性的改進，并且與使用映射-簡化源文本摘要的全局但無圖方法相比也顯示出有利的比較。對于需要針對同一數(shù)據(jù)集進行許多全局查詢的情況，基于實體的圖索引中的根級社區(qū)摘要提供了一個優(yōu)于簡單RAG的數(shù)據(jù)索引，并且以較小的令牌成本實現(xiàn)了與其他全局方法相媲美的性能。

An open-source, Python-based implementation of both global and local Graph RAG approaches is forthcoming at https:///graphrag.

一個開源的、基于python的全局和局部Graph RAG方法的實現(xiàn)即將在https:///graphrag上實現(xiàn)。