self-engineering agent
an autonomous ai system that creates its own tools on demand. rather than relying on pre-built tool libraries that developers must manually create and maintain, this system enables ai agents to synthesize new tools automatically when needed using test-driven development methodology.
---
the problem: static tool libraries
every major ai agent framework (langchain, llamaindex, autogen, crewai) shares a fundamental limitation: they depend on static tool libraries. when a user needs functionality that doesn't exist, development stops until a human developer manually creates the new tool.
this process typically takes hours to days per new capability. for organisations needing dozens of specialised tools, this becomes unsustainable.
---
the solution: self-engineering agents
when the agent encounters a request it cannot fulfill with existing tools, it automatically synthesizes a complete solution: specification, test suite, implementation, security verification and registration.
the framework uses test-driven development for a critical reason: tests serve as unambiguous specifications. by generating tests before implementation, the system ensures clear requirements, automatic verification, edge case coverage and quality assurance.
---
system architecture
orchestration layer: the agent orchestrator serves as the central brain, coordinating all subsystems. it receives user requests, manages session context, routes to appropriate handlers, tracks workflow execution and synthesizes final responses.
intelligence layer: query planner analyzes requests to determine complexity and optimal execution strategy. semantic search finds conceptually similar tools using vector embeddings. memory manager maintains conversational context. reflection engine analyzes failures and generates automatic fixes.
synthesis layer: specification generator transforms natural language into formal function specifications. test generator creates comprehensive pytest test suites. implementation generator writes production code to satisfy all tests. sandbox verifier executes tests in isolated docker containers.
---
the synthesis pipeline
stage 1 - specification: transforms natural language into formal function specification including function name, typed parameter definitions, return type and comprehensive docstring.
stage 2 - test generation: creates pytest tests before implementation covering normal operation with typical inputs, edge cases with boundary values, error conditions with invalid inputs and data quality issues.
stage 3 - implementation: produces python functions with proper type hints, handles all edge cases identified in tests, provides meaningful error messages and follows production coding standards.
stage 4 - sandbox verification: fresh docker container created, implementation and tests copied in, pytest executes with timeout limit, results captured, container destroyed regardless of outcome.
stage 5 - registration: generate semantic embedding of docstring, save implementation to tools directory, insert metadata into database with embedding, tool immediately available for future requests.
---
security architecture
ai-generated code presents unique security challenges. the framework implements defense in depth:
container isolation: every tool execution runs in a docker container completely separate from the host. containers use minimal python image with only essential dependencies. container destruction after each run prevents state persistence.
network isolation: containers created with network disabled entirely. no dns resolution, no outbound connections, no listening ports. prevents data exfiltration and external communication.
resource limits: cpu quota limits to 50% of single core, memory limit of 256mb prevents memory bombs, 30-second timeout catches infinite loops, process limits prevent fork bombs.
---
semantic intelligence
traditional tool discovery relies on exact keyword matching. "analyze csv" finds tools with "csv" in the name, but "examine spreadsheet" finds nothing despite identical intent.
the semantic system converts text into 1536-dimensional vectors using openai embeddings that capture meaning. cosine similarity measures conceptual relatedness. multi-factor re-ranking combines semantic similarity (70%), historical success rate (20%) and usage frequency (10%).
---
self-learning mechanisms
workflow pattern recognition: when tools are used in consistent sequences (a followed by b followed by c), patterns are recorded. patterns gain confidence through repetition.
composite tool promotion: frequently-used patterns meeting promotion criteria (minimum frequency, high success rate) become candidates for composite tool generation. the synthesis engine creates a single tool encapsulating the multi-tool workflow.
reflection engine: when tools fail in production, the engine analyzes error messages and execution context to identify root causes, produces corrected implementations, tests fixes in sandbox before applying, maintains version history for rollback.
---
conversational memory
practical workflows require remembering context. without memory, "now filter that data" fails because the system doesn't know what "that data" refers to. with memory, users can reference previous results, build on earlier computations and develop multi-step analyses conversationally.
session management: each conversation tracked as a session with unique identifier. sessions group related messages and persist across browser refreshes.
data reference tracking: pattern recognition identifies dataframes, lists or results mentioned in responses. reference resolution maps "use that data" to the correct data object. availability verification confirms data is accessible in current session.
context window management: sliding window includes most recent messages. relevance filtering selects semantically relevant historical messages. summarization condenses older context to preserve key information while reducing tokens.
---
what i learned
- llms are surprisingly good at writing focused, single-purpose functions
- the hard part is the test harness, not the generation
- docker overhead (~2 seconds) is acceptable for the isolation guarantee
- tool descriptions matter more than implementations for retrieval
- tdd works even better for ai than for humans because tests provide unambiguous success criteria
---
stack: python 3.10+, flask, flask-socketio, openai api (gpt-4, text-embedding-3-small), supabase (postgresql + pgvector), docker sdk