haystack

Haystack is deepset's open-source AI orchestration framework for building production LLM applications in Python. Its model is modular pipelines and agent workflows with explicit control over how information is retrieved, ranked, filtered, combined, routed and generated. You assemble components for retrieval, indexing, tool calling, memory and evaluation, or write your own, and add loops, branches and conditional logic.

On the agent side it offers lifecycle hooks (before_llm, before_tool, on_exit) for guardrails, out-of-the-box tracking of step_count, token_usage and tool calls, ready-made agents in Agent Pack, and progressive skill discovery via SkillToolset so skill descriptions only enter context when needed. Pipelines run synchronously or asynchronously and stream token by token, and agents can run concurrent tool calls.

It is model- and vendor-agnostic, integrating OpenAI, Anthropic, Mistral, Cohere, Hugging Face, Google, Azure and Bedrock plus local models and many vector stores. The companion Hayhooks project serves pipelines as REST APIs or MCP servers with OpenAI-compatible chat endpoints. Haystack is widely adopted in production, with a paid Enterprise platform available on top of the Apache-2.0 core.