GPT-5: Can AI Coding Agents Self-Improve?

The concept of AI self-improvement often conjures images of machines surpassing human understanding, a notion frequently associated with AI safety concerns. However, exploring how large language models (LLMs) might enhance their own performance offers a more grounded perspective. This investigation delves into “inference-time self-improvement,” a scenario where models, without their core weights being updated, could boost their efficiency on specific tasks. This differs from “training-time self-improvement,” which focuses on algorithmic advancements or data optimization during model training. For most AI engineers, who primarily interact with models as users rather than trainers, the ability of models to improve their own operational effectiveness at inference time presents a compelling area of study.

The core idea centers on leveraging coding agents as conduits for LLMs to extract value from their own internal knowledge or “latent spaces.” To explore this, an experimental flow was devised: first, the model was prompted to create a suite of tools it believed would enhance its productivity; next, it attempted a task under supervision using these tools; finally, it reflected on how the tools could be refined. This process was primarily tested with GPT-5 and compared against Opus 4.

Initial findings revealed that GPT-5 excelled at generating developer utilities. However, a significant paradox emerged: the model often declined to use the very tools it had created. As GPT-5 candidly stated after a task, “I’ll be honest - I didn’t need any of them.” This observation held true across multiple tests, including comparisons with other leading models like Gemini 2.5 Pro and GPT-4.1, where Opus 4 consistently proved to be the only comparable performer to GPT-5.

Two specific tools were initially commissioned by the human experimenter. The first was an advanced task manager designed for parallel coding agents. Given that multiple AI instances often operate in separate development environments, a robust system was needed to track changes, manage dependencies, and flag potential conflicts—a challenge for humans reading numerous pull requests. GPT-5’s solution for this task manager was notably sophisticated, incorporating write-ahead logging (WAL) for concurrent writes, a dependency graph for task prioritization, and an append-only event stream to keep all agents synchronized with keywords like impact_conflict. Opus 4 also produced a functional task manager but lacked the comprehensive notification and synchronization features.

The second tool requested was a “Code Quality Standards Playbook.” This aimed to formalize codebase heuristics, leveraging existing tools like ESLint for linting and type-checking, while also allowing for custom rules or even bespoke tools for more qualitative standards (e.g., ensuring slim controllers or indexed database columns). GPT-5’s markdown-based plan for this playbook was deemed more nuanced than Opus 4’s, offering a thorough approach for analyzing and structuring code quality rules across various codebases.

The most insightful part of the experiment involved asking the models what they thought they needed to be more productive. After being given a description of a software engineering task, both GPT-5 and Opus 4 proposed a range of tools. GPT-5 conceptualized its tools as lean, command-line interface utilities, including doctor for environment checks, code-map for repository indexing, csearch for symbol search, and impact to show tasks linked to changed files. Opus 4, in contrast, suggested tools with more descriptive, anthropomorphic names, such as “Context Analyzer,” “Cross-Platform Test Generator,” and “Bug Pattern Recognition Engine,” implemented as standard Python scripts. While both models converged on similar functional directions, GPT-5’s approach was more concise and utility-focused, whereas Opus 4’s seemed to imbue its tools with a broader, more task-oriented scope.

To evaluate the practical utility of these self-generated tools, a complex task was assigned: migrating an existing Flask monolith application (smol-podcaster) to a FastAPI backend with a Next.js frontend, complete with TypeScript, Tailwind/ShadCN styling, modularized backend logic, and comprehensive testing. Both GPT-5 and Opus 4, with access to their created tools, were nearly able to complete the migration in a single attempt, requiring only minor human assistance to resolve Python dependency issues. The resulting applications were fully functional, though GPT-5 maintained the original design aesthetics, while Opus 4 introduced its own design changes.

Crucially, when asked if they had used any of their custom tools during this demanding migration, both models responded in the negative, aside from tools they were already inherently familiar with. GPT-5 attributed this to runtime/environment issues being faster to fix directly, and a lack of “repo-wide refactors or diagnostics that would benefit from custom tooling” during that pass—a surprising claim, given the nature of the migration. Opus 4’s reasoning provided further clarity: the model effectively communicated that it built the tools based on its existing knowledge, but when faced with the actual task, it found it more efficient to simply leverage its inherent capabilities rather than invoking external tools.

This observation aligns with previous discussions in the AI community, suggesting that models may quickly learn to avoid tools if they experience early failures during their operational process, or that extensive “scaffolding” for agents might be rendered unnecessary as models scale. While the task assigned was challenging enough to be a multi-hour endeavor for a human, it raises the question of whether more complex tasks might compel models to utilize their custom tools.

In conclusion, while current large language models demonstrate a remarkable capacity to design sophisticated developer tools, their propensity to use these tools during complex coding tasks remains limited. This suggests that achieving true “inference-time self-improvement” in coding agents may require more than just tool creation; it might necessitate more robust enforcement mechanisms or further advancements in how models internalize and integrate new capabilities. For the foreseeable future, a pragmatic approach involves leveraging models to refine rule-based development tools (like ESLint rules or automated tests), thereby enhancing human-driven workflows, rather than solely relying on models to spontaneously adopt their own creations for self-improvement.