Chunking Strategies for Agents: Structuring Data for Machine Consumption // Bharath Decodes

In traditional information retrieval, text is chunked for human eyes. In Agentic AI, retrieved chunks are not consumed by humans; they are consumed as intermediate computational context by an LLM reasoning engine.

If a chunk is too small, the agent misses critical causal context (e.g., retrieving a single sentence without understanding the conditional caveats above it). If a chunk is too large, it floods the context window with irrelevant noise, triggers attention dilution, and inflates inference costs.

In this first post of Module 2: The Retrieval & Context Layer, we examine Chunking Strategies for Agents. We benchmark the four dominant chunking architectures and demonstrate how to preserve structural metadata for downstream agentic tool execution.

1. The Core Trade-off: Precision vs. Contextual Grounding

When an agent searches an internal knowledge base, it balances two competing vectors:

┌───────────────────────────────┐               ┌───────────────────────────────┐
│     TINY CHUNKS (50 tokens)   │               │   LARGE CHUNKS (1000 tokens)  │
├───────────────────────────────┤               ├───────────────────────────────┤
│ • High embedding specificity  │               │ • High semantic completeness  │
│ • Loss of contextual grounding│               │ • Dilutes vector relevance    │
│ • "Blind needle" problem      │               │ • Burns agent context window  │
└───────────────────────────────┘               └───────────────────────────────┘
                                ▲               ▲
                                └───────┬───────┘
                                        │
                         Optimal Machine Consumption Zone
                                 (250 - 500 tokens)

2. The Four Chunking Architectures

1. Fixed-Size Sliding Window

Slices text into rigid $N$-token intervals with an overlap buffer (e.g., 500 tokens with 50 tokens overlap).
* Pros: Simple, fast, deterministic compute costs.
* Cons: Splits sentences, code functions, and markdown tables down the middle, destroying semantic coherence.

2. Recursive Character Text Splitting

Splits hierarchically along natural syntactic boundaries: paragraph breaks (\n\n), then sentence breaks (\n), then spaces ().
* Pros: Preserves paragraphs and logical thoughts intact.
* Cons: Still blind to semantic topic transitions within paragraphs.

3. Semantic Chunking (Embedding Distance Clustering)

Computes the cosine distance between consecutive sentences. When semantic distance spikes above a threshold $\tau$, a new chunk boundary is dynamically created.
* Pros: Chunks reflect true conceptual shifts rather than arbitrary character counts.
* Cons: Computationally expensive during ingestion (requires embedding every sentence).

4. Hierarchical Parent-Document Retrieval (The Agentic Gold Standard)

Stores small, highly specific chunks (e.g., 100 tokens) in the vector index for high-precision retrieval, but maps each child chunk to a larger parent document (e.g., 800 tokens) stored in a key-value store. When the agent retrieves the child, the parent context is injected into the LLM!

[Document: Enterprise Cloud Security Guidelines]
  │
  ├── Parent Chunk 1 (800 tokens) ──────────────────────────┐
  │     ├── Child A (100 tokens: IAM policies)   ─> Index   │
  │     └── Child B (100 tokens: MFA enforcement)─> Index   │
  │                                                         ▼
[Agent searches "MFA rules"] ──> Matches Child B ──> Injects Parent Chunk 1!

3. Python Implementation: Parent-Document Retreiver

Here is a production implementation using Python:

import uuid
from typing import List, Dict

class ParentDocumentChunker:
    def __init__(self, child_size: int = 200, parent_size: int = 800):
        self.child_size = child_size
        self.parent_size = parent_size
        self.parent_store: Dict[str, str] = {}
        self.child_store: List[dict] = []

    def process_document(self, doc_text: str, metadata: dict) -> List[dict]:
        paragraphs = [p.strip() for p in doc_text.split("\n\n") if p.strip()]

        # 1. Generate Parent Chunks
        parents = []
        current_parent = []
        current_len = 0

        for p in paragraphs:
            p_len = len(p)
            if current_len + p_len > self.parent_size and current_parent:
                parents.append("\n\n".join(current_parent))
                current_parent = [p]
                current_len = p_len
            else:
                current_parent.append(p)
                current_len += p_len
        if current_parent:
            parents.append("\n\n".join(current_parent))

        # 2. Slice Parents into Child Vectors linked by parent_id
        indexed_children = []
        for parent_text in parents:
            parent_id = str(uuid.uuid4())
            self.parent_store[parent_id] = parent_text

            # Create granular children (sentence level)
            sentences = parent_text.split(". ")
            for sentence in sentences:
                if len(sentence.strip()) > 30:
                    child_record = {
                        "child_id": str(uuid.uuid4()),
                        "parent_id": parent_id,
                        "text": sentence.strip(),
                        "metadata": {**metadata, "parent_id": parent_id}
                    }
                    indexed_children.append(child_record)
                    self.child_store.append(child_record)

        return indexed_children

    def resolve_context(self, retrieved_child: dict) -> str:
        """Hydrates the small retrieved vector match into its complete parent context."""
        parent_id = retrieved_child["metadata"]["parent_id"]
        return self.parent_store.get(parent_id, retrieved_child["text"])

4. Chunking Markdown & Technical Code for Agents

When chunking documentation, code snippets and markdown tables must never be severed.

Key Rules for Technical Data Ingestion:

Fenced Code Blocks: Treat ```python ... ``` as atomic blocks. If a code block exceeds chunk size, split by logical function definitions rather than line counts.
Markdown Tables: Convert markdown tables into self-contained JSON records before vectorizing, or duplicate the table header row on each split chunk to preserve column semantic references.
Breadcrumb Metadata: Always inject document heading hierarchy into the chunk metadata:
yaml metadata: document: "system_architecture.md" hierarchy: "Infrastructure > Networking > VPC Peering"

Summary & Next Steps

Chunking is the foundational data engineering step that determines whether your agent receives surgical signal or noisy distraction. For agentic workloads, Hierarchical Parent-Document Chunking offers the optimal compromise: pinpoint vector matching with complete reasoning context.

In [Post 06: Hybrid Search & Re-ranking], we explore the retrieval engine itself: fusing dense neural vectors with sparse lexical search and cross-encoder precision re-rankers.