Light RAG Experiment Report

#RAG #LightRAG #graphRAG

Overcoming GraphRAG’s Limitations with LightRAG: A Direct Token Usage Comparison

Microsoft’s graphRAG consumes significantly more tokens than traditional RAG for tasks like Knowledge Graph construction and query processing (e.g., 610,000 tokens). To address these limitations, new architectures have emerged, and the one I tested this time is LightRAG.

What caught my attention was a claim in the LightRAG paper: "While GraphRAG consumes 610,000 tokens, LightRAG can perform searches with fewer than 100 tokens." However, the term "search tokens" felt vague to me, so I decided to implement it myself and compare token usage directly. (Source: LightRAG GitHub)

Initial Indexing: Setting the Baseline

This token consumption reflects the initial indexing process and serves as a key baseline for understanding LightRAG’s core behavior.

Token Usage Summary

Document Details

Note: A second try was performed with slightly different results.

Query Result Comparison: Query modes

LightRAG offers two query processing modes: Mix Mode (combining vector search and Knowledge Graph) and Local Mode (entity-centric search). Below is the token usage for the same query in each mode:

Query Token Usage by Mode

input query 15token

Check point

GetKeywords Input Default Input Format Consumes about 405 tokens
Real Query Input, system instruct and full context are mostly used as input, except for input query 15token.
For example, in naive, 2870 was used as input context, and in MIx, 9129 was used as input context

Update Token Usage Comparison (Including Initial Indexing)

Caution: Some data may lack precision due to unaccounted execution order; use it only as a trend reference.

I was particularly interested in how LightRAG handles updates to existing data, as it claims to support incremental updates. However, my tests revealed that adding new documents and updating existing ones behave differently. Here are the results:

Document A: Sequential Update Analysis

Process 1 Input : Instruction + full text used
Process 1 Output: get entity and relationship information of chunk
Process 2 Input: Previous conversation + how to extract and rest data
Process 2 Output: All entity and relationship data

This is repeated until enitity and realationship are extracted from a chunk of all docs.

Regardless of how much content is added, the entire document is embedded, and the conversation repeats until entities and relations are extracted from all tokens.

Document B: New Document Addition (104 tokens)

Process 1 Input : Instruction + full text used
Process 1 Output: get entity and relationship information of chunk
Process 2 Input: Previous conversation + how to extract and rest data
Process 2 Output: All entity and relationship data

This is repeated until enitity and realationship are extracted from a chunk of all docs.

When adding a new document, only the tokens from the newly added document are considered, independent of existing documents.

Key Findings from Single Document Update:

• Embedding Input Token Analysis: LightRAG appears to re-embed the entire file during updates, not just the added tokens
• ChatCompletion Input Token Analysis: During incremental updates, input tokens scale efficiently with added tokens and generally, instructions or context are included in the input as a prefix.
• ChatCompletion Output Token Analysis: Output remains stable or decreases regardless of input size, possibly due to simplification or deduplication of existing relationships

Conclusion

LightRAG drastically reduces token usage compared to GraphRAG, enhancing efficiency with incremental updates and caching. However, its high token consumption during initial indexing and full re-embedding during updates remain limitations. If it evolves to support partial embedding or maximizes cache usage, LightRAG could become an even lighter, more optimized model for small-scale updates.