LLM Orchestration Frameworks

People often ask if they should use LangChain or LlamaIndex. What about all of them?

The power of those frameworks is not only in what they can do but also in what they can teach us!

I used to enjoy Langchain quite a bit, mostly because the documentation was structured to educate the users about LLM pipeline orchestration and showcased how they approached building a solution for implementing those pipelines. To some extent, all the existing frameworks took their own opinionated approach to provide solutions to the complexities around LLM pipeline orchestration.

Getting a wide overview of the different capabilities provided by those frameworks is a real learning experience in terms of what it means to build LLM applications, what the typical difficulties are, and how to address those.

There are many overlaps in the capabilities of the different frameworks, but I tend to separate those by their specialties:

• Micro-orchestration: I refer to Micro-orchestration as the fine-grained coordination and management of individual LLM interactions and related processes. It is more about the granular details of how data flows into, through, and out of LLM within a single task or a small set of related tasks. It involves things like:

• Prompt engineering

• Input preprocessing and output postprocessing

• Handling of model-specific parameters and configurations

• Chaining of multiple LLM calls within a single logical operation

• Integration of external tools or APIs at a task-specific level

The best examples of that are LangChain, LlamaIndex, Haystack, and AdalFlow.

• Macro-orchestration: it is more about the high-level design, coordination, and management of complex workflows that may incorporate multiple LLM interactions, as well as other AI and non-AI components. It focuses on the overall structure and flow of larger systems or applications.

This is a newer type of orchestration system, and LangGraph, Haystack, LlamaIndex Workflows, and Burr seem to be leading the charge.

• Agentic Design Frameworks: These frameworks focus on creating and managing autonomous or semi-autonomous AI agents that can perform complex tasks, often involving multiple steps, decision-making, and interaction with other agents or systems:

Most frameworks have their own approach to agentic design, but Autogen and CrewAI tend to separate themselves by having a unique angle to the problem.

• Optimizer frameworks: These frameworks use algorithmic approaches, often inspired by techniques like backpropagation, to optimize prompts, outputs, and overall system performance in LLM applications. The optimization process is typically guided by specific performance metrics or objectives.

This is a newer category of orchestrators, and it has been led by frameworks like DSPY and TextGrad. To my knowledge, AdalFlow is currently the most mature framework in the domain.

Author

AiUTOMATING PEOPLE, ABN ASIA was founded by people with deep roots in academia, with work experience in the US, Holland, Hungary, Japan, South Korea, Singapore, and Vietnam. ABN Asia is where academia and technology meet opportunity. With our cutting-edge solutions and competent software development services, we're helping businesses level up and take on the global scene. Our commitment: Faster. Better. More reliable. In most cases: Cheaper as well.

Feel free to reach out to us whenever you require IT services, digital consulting, off-the-shelf software solutions, or if you'd like to send us requests for proposals (RFPs). You can contact us at [email protected]. We're ready to assist you with all your technology needs.

© ABN ASIA