Dynamic AI Agent Ecosystems: Creating & Managing Subagents | Humanitarians AI Fellows Program
This lecture builds a system where a parent AI agent dynamically creates subagents for new problems and reuses existing ones for similar problems, backed by ChromaDB and Neo4j.
If an AI system can spin up a new specialized agent for every problem it encounters, how does it avoid creating a thousand redundant agents that all do roughly the same thing? This lecture works through that exact question, building an architecture where a parent agent dynamically creates subagents when it needs to, checks a registry before doing so, and reuses an existing agent whenever a similar problem has already been solved.
Two modules that scale an agent ecosystem
AI agents are described here as autonomous systems that perceive their environment, process information, and take actions to achieve specific goals, whether they're rule-based, driven by machine learning models, or capable of creating and interacting with other agents. To scale that idea into a genuine agent ecosystem rather than a single agent, the architecture relies on two modules working together: dynamic subagent creation and role assignment, and an agent registry with a metadata database. The first module decides when a new agent is actually needed; the second keeps track of every agent that already exists so the first module doesn't have to guess.
How a subagent gets created
The creation flow starts when an external agent, described as a "super parent agent," sends in a problem statement or query. The module's job is to produce a list of newly created subagents, if any are needed, each with an assigned role, a role description, and a task prompt. That output is generated as structured JSON from the underlying language model specifically to keep agent creation consistent and machine-readable rather than free-form text.
Before creating anything new, the system consults the agent registry to avoid duplicating an agent that already exists. Every agent created carries a defined set of fields: an agent ID, an assigned role, a role description, a task prompt, the parent agent ID that created it, any dependencies it has, and metadata such as a creation timestamp and which language model was used to generate it.
Why two different databases
The registry module maintains a database for every agent in the ecosystem, and it deliberately splits that job across two different kinds of database. ChromaDB stores agent metadata and is used for similarity search, comparing a new problem statement against the profiles of existing agents. Neo4j stores agent-to-agent relationships, specifically parent-child dependencies, as a graph. That split matters because the two questions being asked are genuinely different: "does an agent like this already exist" is a similarity question, well suited to a vector database, while "how do these agents relate to each other" is a relationship question, well suited to a graph database.
The decision loop: reuse or create
The workflow runs in a consistent sequence. An external agent sends in a problem statement. The system consults the registry to check for existing agents that could handle it. If an existing agent is found that's a sufficiently close match, that agent gets reused, and the new parent is simply mapped to it. If no existing agent is found, the module creates a new one, assigns it a role, a description, and a task prompt using structured JSON output, and may go through multiple passes to account for the context of agents that already exist. Once a new agent is finalized, its metadata and relationships get sent to the registry, stored in ChromaDB and Neo4j respectively, so the next problem statement that comes in can potentially reuse it instead of triggering another new creation.
Watching the decision happen live
The demonstration runs this system using LLaMA 3.1 (LLaMA 3.2 is mentioned as an alternative), served through a FastAPI backend, with a codebase organized around a main file, a ChromaDB database, a Neo4j database, and a dedicated agent creation file. In the first demo run, a parent ID and a problem statement, "create a data analysis report," are submitted for the first time. The logs report that no sufficiently similar agent was found, so a new agent is created, assigned the role of data analyst, and mapped to its parent in Neo4j, which can be queried directly to confirm the relationship.
The second run submits a new but very similar problem statement under a different parent ID. This time, the similarity search against ChromaDB, using a defined similarity threshold, finds a match: the log reports an existing agent was found, and no new agent gets created. The new parent is simply mapped to the agent that already exists, confirmed again by checking the graph in Neo4j.
A third run submits a genuinely different problem statement, and the system correctly creates a new agent for it, since nothing sufficiently similar exists in the registry yet, which is confirmed the same way, through the logs and a Neo4j query showing the new parent-child relationship.
Key takeaways
- Scaling an agent ecosystem requires two coordinated modules: dynamic subagent creation and role assignment, and an agent registry with metadata storage.
- New agents are created with structured JSON output, always carrying an agent ID, role, description, task prompt, parent ID, dependencies, and metadata.
- ChromaDB handles similarity search to detect whether an existing agent can already solve a new problem; Neo4j stores the resulting parent-child relationship graph.
- A similarity threshold determines whether a new problem statement reuses an existing agent or triggers the creation of a new one.
- The live demo confirms the loop in both directions: a genuinely new problem creates a new agent, while a similar problem reuses one that already exists.
Who this is for
This lecture is aimed at developers and students building agentic AI systems who need a practical pattern for avoiding redundant agent creation at scale. It's part of the technical curriculum in the Humanitarians AI Fellows Program.
Full transcript(auto-generated, with timestamps)
[0:01]Hi. Uh so today we will be uh studying about dynamic sub agent creation and role assignment and we will also be going through agent registry and metadata database. So AI agents are autonomous systems designed to perceive their environment, process the information and take actions to achieve specific goals. So uh they can either be rule based or can be driven by various machine learning models or they can often create and interact with different agents as well. So in in an agentic uh architecture AI agents can you know dynamically create configure and deploy other agents based on the task requirements. So this is uh basically done using a
[0:45]Self-replicating task allocating or or a hierarchal based system. So to scale a super AI agents infrastructure we need uh two different modules. The first one is dynamic sub agent creation and role assignment and the second thing is agent registry and a meta data database. So uh now coming to the key requirements the key features and requirements of an AI agentic universe. So the first thing that is dynamic sub aent creation and role assignment module. So in this basically a problem statement or query is given from an external agent probably a super parent agent. Output is a list of newly created sub aents if needed with their rows their
[1:31]Descriptions and their different task prompts. So uh basically uh dyn uh a parent agent is there and a problem statement is given. Now the now different roles are assigned uh role descriptions and task prompts are assigned to each sub agent. Now we will be using JSON output from LLMs to ensure structured and aligned agent creation. Then we will consult the agent registry which is uh we will be using uh chromb for the agent metadata storage and neoforj for graph based agent agent relationship management. So to consult the agent registry and metadata database to avoid uh duplicate agents. Now uh we will be using the following
[2:16]Fields for agent creation. So every agent should have an agent ID, a role, um a certain role which has been assigned to an agent and it is supposed to perform that, a role description, a task prompt, a parent agent ID which is the ID of the parent agent which has created a subsequent agent, various dependencies and obviously some metadata like timestamp or the LLM used to create that agent. Our second thing is agent registry and metadata database module. So basically we maintain a database for all the agents in the ecosystem. We use chromod to store metadata agent metadata and we use neoforj to store agent agent
[2:56]Relationships you know parent child dependencies and all. Now we check for existing agents. If if there's a problem that is given and if there is an existing agent that that can solve that problem then we will be using that agent to solve the problem. Otherwise we will create a new agent. So these newly created agents will be registered with the meta meta data in the chromb and the relationships in the neo fallj. So this is uh the designed workflow. So basically a problem statement input is given. So an external agent sends a problem statement or query with dynamic sub aent creation and role assignment module. Now the second thing that
[3:39]Happens is we consult the agent registry. Now the model consils the agent registry to check if existing agents are there or not. Now comes the agent creation. So if there are no existing agents found that can handle the task, the module will dynamically create new sub aents. Assign roles, descriptions and task forms to each sub aent and this uses JSON output from the LLMs to structure agent creation. Third thing is the module may require multiple terms to create agents. considering the context of existing agents. Now if we need new agents to be created. So once all the new agents are created, the module sends the final list
[4:21]Of the metadata and the relationship to the agent registry and then the list of agents is stored in meta. Uh so yeah agent registry ensures that the agent dependencies which is like the parent child relationships are correctly stored in Neo4j and all the metadata of the newly created agents is stored in chrom. uh so I will be showing a demo as well. So basically this is the code base that uh I have designed. So the llm that we are using for creating the agent is uh llama 3.1. You can use llama 3.2 as well. And I'm using fast API to give the input. So this codebase basically has a main py
[5:12]File. Then we have a chromb database, new 4j database and also agent creation file. Uh okay. So I will just give a demo. Yeah. So the fast API is started. Now I have a I have created a validation file since I was trying this earlier. So um I will give parent A as the first parent ID. So this is the master agent that is being created. parent is the first parent ID and I'm giving this problem statement as the
[6:23]First problem statement. So ideally this is the first agent that is being created in the entire agent agent AI universe with a parent ID of parent A and a certain problem statement. Now this is the first time we are giving this problem statement. So this means there's no existing agent that uh there's no existing agent which can solve this problem. So ideally it should create a new agent. So let's see what happens. We can check the logs over here. The lock says no sufficiently similar
[7:35]Agent found which means a new agent will be cured. Yeah, it usually takes a bit time to create a new for the first time. Yeah, here. So, a new agent has been created and mapped and we can check for the same in Neo4j.
[8:50]Yeah. So, this is a query which we use in Neo 4j to get the graphs. Yeah. So this is a parent A that is the dummy agent that was there with the parent with the agent ID as parent A and this depends on a newly created agent which can solve the problem. The problem statement is create a data analysis report. Basically the problem statement that we gave and the role assigned to this agent is data analyst. Now we will check for some new ones. Now if we give a similar problem. So if I give a new problem statement. So this is a new problem statement and I
[9:55]Will change the pairing title. So now ideally what should happen? So since this is a new problem statement but it is very similar to the previous one. So the previous created agent should be able to solve this problem. So ideally the new parent should be mapped to the already previously created agent. Now coming to how we are deciding if new agent is needed or not. So we are using the similarity search uh which is there in vector databases since we using chrom. So we are using the similarity search and we have defined a certain threshold. Oh let's see what happens. Yeah. So this says existing agent is
[10:38]Found in map. So no new agent was created. The already existing agent could solve this problem. Let's check this in your project. Yeah. So the new agent is mapped to the already existing agent. I will show one more example. So okay let's give one more new problem statement. So ideally there should be a new agent created for this problem statement. So the parent agent for this problem statement would be parent C
[11:58]If you check the lot, it says that it is creating a new agent since no sufficiently similar agent found. Yeah. So it has created a new agent. We can check the same in your 4G. Yeah. So for this parent a new agent has been created. So uh that was all from my site and this was the demo of uh about how agents AI agents can be used to create new agents. Thank you.
More videos
2:08Bridging the Pixel Gap in Browser Automation.
2:23How One Narrow Safety Rule Can Make an AI Less Safe Everywhere Else.
2:04Why splitting a chunk from its document makes it retrieve for the wrong question
4:20Three You Can Take Back. One You Can't.
2:21Why a 50-turn agent pays for the same screenshot 35 times unless it caches the pixels
1:53