2026 was a big year for AI, with agents taking the main stage and tools like Claude Code going mainstream. We put together a summary of what we've researched so far. The story of H1 2026 is Exploding Spend, Diverging Returns. You can read the first half of our report below — and if you want the full version, you can download it here.

Introduction

In March 2026, we saw the first user spend more on tokens in a month than a junior developer earns. Power users are bending what is possible in the new AI-SDLC, pulling away from the median engineer. Spend is exploding, returns are diverging, and most teams cannot yet tell which side of that divergence they are on.

Whether AI helps engineers write code is no longer the interesting question. The sharper one is about outcomes: can an agent accomplish a goal when it is set up for success, and what does that setup actually cost? Two more sit behind it. Why is the bill rising faster than the sticker price, and why do a few teams convert tokens into throughput while most do not?

This release concentrates our H1 2026 research from Jellyfish to answer those questions.

Exploding spend, diverging returns is the story of H1 ’26. This report highlights a major shift in how AI is used in software development:

1. The cost shock. Spend has gone from a rounding error to a real constraint, rising 16x across the board with a tail that now rivals a junior engineer’s salary.

2. The economics of tokens. What AI actually costs, why effective cost keeps rising even when sticker prices hold still, and where more tokens stop buying more output.

3. The harness advantage. What the best teams invest in instead of raw spend: context, harnesses, and agent-friendly codebases.

Leading teams are chasing what we call the “AI pop”: a clear jump in productivity and throughput. Standing in their way is what our research calls the agentic barrier: the hurdles in human attention, infrastructure, and cost that teams hit as they push agents further. There is no single way through it, but the industry is pressing on two fronts:

• Running agents on more complex, longer tasks with less supervision: Autonomy

• Running more agents in parallel: Concurrency

Both patterns drive an explosion of token consumption. At the same time, new models and agentic features keep raising the effective cost of running the same workload, so bills pile up and become unsustainable for users who are not converting tokens into productivity. A minority is converting, and converting hard. That is the divergence we are witnessing.

AI is collapsing old processes and creating new ones at an extremely fast pace. We hope this research helps you navigate what comes next in 2026.

PART 1

The cost shock - explosion of intelligence

Cost is rising across the board, with a runaway tail.

For most of 2025, AI spend in software engineering was a rounding error. It was a predictable utility: you spent a little more on tokens, you got a little more code. It scaled linearly, cleanly, and predictably.

In the first half of 2026, that relationship broke.

The shift to autonomous, long-running agentic workflows didn’t just increase usage; it fundamentally changed the unit economics of software development. As teams moved from ‘AI as a chat assistant’ to ‘AI as an engineer,’ token consumption exploded—not by percentage points, but by orders of magnitude. By March 2026, we saw the first users spending more on tokens in a month than a junior developer earns. We call it the cost shock: a structural decoupling of spend from output that has caught the industry off guard.

Cost per user has risen across the board: at a typical company the median user went from about $5 a month to $81 in ten months, a 16x climb, and even mid-tier (P75) companies went from $10 to $170. This is a broad-based shift in what software development costs in the new AI-SDLC.

Cost in AI is rising broadly for all companies, a typical company’s cost per user grew 16x in ten months

Figure 1. Per-company cost per user by company percentile, Aug 2025 to May 2026. The median company’s typical user went from $5 to $81 a month, a 16x rise, increase is broad-based.

Spend is rising across the board, and the extremes are shocking

On top of that broad rise sits an extremely skewed tail. In May 2026, more than half of users still spent under $100 a month, yet the mean ($248) ran 3.5x the median and the 99th percentile ($2,452) ran 35x it. A small number of power users spend enormously, and at the very end of that tail the numbers start to rival human salaries.

Charting the single heaviest user’s monthly spend against a US junior developer’s salary (about $12,500 a month) shows how far the tail reaches: that user touched and at times exceeded the line in spring 2026.

At the very tail, a single user’s bill has touched a junior engineer’s salary

Figure 2. Highest single-user monthly spend vs a US junior developer salary (~$12,500/mo). The tail touches salary territory but does not hold there.

Although this behavior is concentrated only on the top few percent of users at the top few percent of companies, it might be a light of the future we might expect on AI usage in software development.

Dollars follow tokens, so the next question is how fast token consumption is climbing, and then why the bill rises even when prices do not.

Token spend tripled since October, and power users are decoupling from the median

Tokens are how LLMs process text, and AI labs’ business models are based on how tokens are priced and charged per million.

We have been tracking token usage, and in January something changed. November 2025 brought the release of Opus 4.5, Anthropic’s main model, and usage patterns shifted sharply afterward.

Figure 3. Weekly Claude Code tokens per user, by percentile. Median usage is flat; the top decile separates and accelerates.

Token consumption climbed steeply from the start of 2026, tripling for power users (P90) from 50M to 170M.

We can also see a divergence between power users (P90) and the median (P50). The first group completely decoupled from the rest of the group, benefiting from the increase in throughput and widening the gap from a 50M token difference to 150M tokens.

Who is justifying the bill, and why most cannot follow

If a thin tail is pulling away, the obvious question is what that tail does differently. Three frontier behaviors separate them: they put more work through agents, they let agents run longer with less supervision, and they push on running more agents at once. Each one helps explain who sits on the paying-off side of the divergence, and why the rest are locked out for now.

Autonomous agents are spreading, leading teams now run 1 in 3 PRs through agents

Figure 4. Share of pull requests opened by autonomous agents, by company percentile. The median holds near 2% while the 90th percentile bends steadily upward.

The clearest picture of the widening gap is the share of pull requests opened by autonomous agents, tracked at the company level. At the median, the line barely lifts off 2%. At the 90th percentile it rises and keeps rising: 10% in January, 14.5% by the end of February, 35% by April, against roughly 2% a year earlier. The top of the market is not adopting agents so much as compounding on them, and the distance to the median grows month over month rather than closing.

What are those leading teams doing differently while running autonomous agents? Sorting Claude Code agents by session length and by how often a human interrupted them, supervision barely matters in short and medium sessions.

In the longest sessions, the top 1% by duration, the pattern turns stark: runs left largely alone (under 30% human intervention) produce about eleven times the net lines of code of heavily supervised runs of the same length. Low supervision drives the most lines of code, but the biggest impact lands in the power-user segment. What duration measures here is whether the agent was set up to succeed without rescue, and the output gap suggests power users are engineering that setup differently from their peers.

Long-running agents with light supervision produce 11x more code

Figure 5. Average net lines per turn by session length and supervision level. Teams with low supervision and long running turns generate 11x more net code than turns with high supervision

The longest sessions in the dataset, the top 0.1%, ran past 94 minutes, and 83% of them involved little human contact start to finish. According to METR, the length of tasks AI can do is doubling every 7 months (at 50% success rate). At this pace, by the end of 2026 we will have agents that will be able to run for as long as 4 hours on a single task.

Setting agents up to complete long tasks autonomously is becoming a huge unlock, but only P99 users are seeing the gains so far. If net lines of added code is a signal of productivity of long workflows - power users understand how to run them and have increased productivity.

Autonomy is one of the challenges that teams are still trying to overcome. We asked ourselves: how many of these agents are power users able to run concurrently? To do so, we studied agent concurrency.

Concurrency hits a ceiling: 84% run just one or two agents at once

If running agents for long periods of time in a loop with a goal until it completes is what unlocks output, the obvious next step is to run more agents at once. But we discovered that high concurrency levels are rare.

Measuring peak concurrent agent use across customers’ turns (each session is composed of multiple turns) hit a ceiling: 84% of active users top out at one or two agents at a time, and even the small group launching four or more spends over 80% of session time attending to a single agent.

Figure 6. Peak concurrent agents per active user. Almost everyone runs one or two; the agentic barrier is human attention, not tooling.

This is the agentic barrier at its clearest. Human attention is the blocker on multi-agent workflows, and attention does not parallelize. Scaling infrastructure to orchestrate more than a pair of agents is not trivial either, and the industry is still building the tooling for users to scale agents. As Karpathy put it, we will need a bigger IDE.

While agent concurrency currently faces a functional ceiling, the market is introducing solutions to address these limitations. This includes Anthropic’s release of agent teams and platforms like Conductor that facilitate parallel agent execution. For organizations aiming to scale productivity, prioritizing new infrastructure and tools alongside strategies to manage human attention challenges has become essential. We think that the 16% of users running 3+ agents in multi-agentic workflows are highly correlated to teams spreading autonomous agents and getting the productivity gains we see in PRs from Figure 4.

Concentrated gains among elite teams are driven by these frontier behaviors. Because human attention remains a non-parallel resource, most users face rising costs with diminishing returns, while only a few outliers successfully convert high spend into significant output. Cost shock is affecting companies broadly, but these frontier capabilities are creating a divergence in productivity that median users aren’t seeing.

These three behaviors are why the gains stay concentrated. Attention does not parallelize, so only outliers turn heavy spend into output while everyone else pays a rising bill for less. The next part looks at why the bill climbs even when sticker prices hold still, and why more tokens eventually stop buying more output.

PART 2

The economics of tokens

Where the cost comes from, and why more stops working.

At GTC 2026, Jensen Huang told the room that a $500,000 engineer who isn’t spending $250,000 a year on tokens is a wasted seat. Tokens, in his framing, now belong in the comp package. Sam Altman went further, floating universal basic compute for everyone. The industry pitch is that more tokens are always worth buying, and this part is where that pitch meets the invoice. We looked into whether it is always true, and whether AI labs genuinely mean it or are selling a narrative.

To deep dive into this, we decided to study the price per token: how is the effective cost affected by the release of a new model, how are new agentic features pushing token consumption, and whether tokenmaxxing is the right strategy for companies to build.

If you want to read the full report you can do it here

Subscribe now

The most important value that it brings to us, however, is how usage patterns and customer interactions with it inform our thinking about the future of our product. As more and more SaaS companies move towards a “headless” product vision, we’ve been thinking a lot about where Jellyfish fits into that world, and the MCP has provided a perfect testing ground for our hypotheses and experiments.

For a product like ours, MCP isn’t just a new piece of UI. Its promise is deeper: a flexible, conversational access point to the underlying data. A user can point an LLM at our server and interrogate their Jellyfish data in essentially any way they want, including questions we never built a view for in the web app. The data stops being trapped behind the views we happened to build. The user brings their own client, asks whatever they want, and gets it back in whatever shape is useful that day: a table, a summary, a chart, without us having to anticipate the question in advance.

So when Anthropic launched MCP Apps earlier this year, we were thrilled by the thought of being able to offer our customers a new, highly flexible way to interact with their data. We thought this might be the thing we had been waiting for, to finally bridge the gap between our MCP and the main primary product surface. We had to try it out, and we found what we thought was a perfect use case.

The Experiment

A customer of ours, a Jellyfish MCP power user, had been running a Monday-morning ritual on top of our MCP server for months. He’d paste a carefully refined prompt into Claude Desktop and it would pull a curated subset of his initiatives from Jellyfish, format them into a card grid, and generate a week-over-week prose summary. He’d built the workflow himself, refined it over time, and while a few of the steps were manual and a little annoying, for the most part it worked.

We thought an MCP App could be a great way to convert his improvised solution into something more polished, and in many ways it was. Just not in all the ways we wanted it to be. We spoke to him about it, identified his pain points, and set out to build him a better solution. The gap we set out to close was the lack of a persistent, refreshable, and sharable home for his workflow: he was regenerating his summary from scratch every week, with no reliable record of old data and no way to refresh without retyping his long, carefully crafted prompt. So we built him the real version: a highly configurable interactive dashboard, served from our MCP server, rendered inline in Claude Desktop.

It had a card grid of major active initiatives, FTE investment, progress bars, projected dates, week-over-week deltas, and per-card drill-downs into weekly trajectory data for each one. There were status and team filters at the top, plus a saved combination of fields, statuses, and team scope that the user could configure to their liking. Mechanically, it was five MCP tools, an HTML resource for the iframe, and local JSON persistence for configuration preferences in the user’s home directory. The whole thing ran as part of our existing MCP server, with no separate deployment, no hosted infrastructure, and no new auth flow. We used the standard patterns for MCP Apps: iframe-as-resource, companion tools backing every interactive control, and the ability to fetch additional data on demand without an additional LLM round-trip. It worked exactly as promised.

The realization

Ultimately, an MCP App doesn’t extend the flexible-data-access promise. In some ways it even cuts against it. The tool layer gives the user open-ended ways to interrogate the data. An MCP App takes a slice of that and freezes it back into a fixed, pre-built view. That’s genuinely useful, but it is still a view, with the same essential shape and constraints as any other UI we’d built.

So the offering ends up being a third category of thing, sitting alongside (1) our main web app and (2) the flexible tool layer. It doesn’t have the infinite flexibility of the tool layer, and it doesn’t have the depth of the full app. It’s its own surface, in between.

That’s not a failure mode– it’s just what an App is, and it’s worth being clear-eyed about that before you build one. For a user who already lives in our main app, a thinner version of that, embedded in chat, isn’t actually much of an upgrade. The chat container, the iframe sandbox, and the available tool surface all cap how much depth you can push through it.

The chat surface is for glanceable previews and quick actions. The full product is for deep work. The tool layer is for everything in between that we hadn’t anticipated and built a view for.

Where we’re going next

While we do still see genuine value in MCP apps, and we’re planning to release the one we have, we’re largely keeping our efforts focused for now on expanding and reinforcing the underlying infrastructure on top of which our MCP is built. This means designing better, more efficient, MCP-friendly endpoints, implementing a more flexible permissions structure for improved shareability, and better instrumentation.

What we want our MCP to offer people right now is more of the flexibility: the open-ended, interrogate-it-any-way-you-want access that drew us to the protocol in the first place. That’s the real value of MCP to us, and this project helped to clarify that.

We’ve been doing our homework and learning about some really interesting, creative solutions that others are implementing when faced with these same challenges, and we’re testing out some of them now. Stay tuned for more about how we’re approaching these challenges and what we’ve been learning along the way!

The Jellyfish MCP just got a little smarter about Jellyfish itself. With our latest update, your AI assistant can now reach into the Jellyfish help center — searching our documentation and reading full articles on demand — so the answers it gives are grounded in how the product actually works.

If you’ve used the Jellyfish MCP to ask about your team’s effort allocation, delivery metrics, or AI impact, you already know how much faster it is to get insights in a single conversation. But until now, questions about how Jellyfish works — how to configure a feature, where a metric comes from, how to set something up — were answered from the model’s general knowledge alone. That left room for vague or out-of-date guidance.

What’s new

We added two new tools, backed by two new endpoints, that connect the MCP to the Jellyfish help center. When you ask a “how do I…” or “how does this work” question, the assistant first searches our documentation to find the most relevant articles, then pulls in their full content as context before answering.

What this means for you

• Fewer generic answers. Guidance reflects the current product, not a best guess.

• The source is right there. Responses are backed by the actual help articles, so you can dig deeper when you want to.

• One conversation, end to end. Ask about your data and how to act on it without leaving the assistant or hunting through docs in another tab.

How to use it

You don’t need to do anything special — just ask your questions naturally, and the assistant decides when to consult the help center on its own. Two patterns where this is especially powerful:

• Ask questions: in plain language via you agent. “How do I set up SSO in Jellyfish?” or “How does AI Impact measure adoption?” The assistant searches the documentation, pulls the most relevant article, and answers with steps grounded in how the product actually works — no tab-switching, no guesswork

• Build context for agentic coding loops: If you’re using an AI agent to write code against the Jellyfish API, the MCP can now feed it the right documentation as it works. Instead of the agent guessing at endpoints or parameters, it can search the help center, read the relevant articles, and ground its implementation in real, current docs. The result is more accurate code, fewer wrong turns, and a tighter loop — the agent pulls its own context exactly when it needs it rather than relying on what it happened to know.

  

Image: A documentation example showing how the MCP returns reliable information about our API.

Setup:

Nothing changes about how you use the MCP. If you already have the Claude Desktop extension installed, you will need to update the MCP config with the latest release 1.1.1

• For MCP Extension Users: Update your MCP config with new release. Install the extension by downloading the .mcpb file from the Jellyfish MCP GitHub repository, double-clicking to install, and providing your Jellyfish API token when prompted.

• For NPM users: Update will be pushed automatically, so no setup is required for existing users. New users can follow instructions from the Jellyfish MCP GitHub repository

Grounding AI in a trustworthy, current context is one of the hardest parts of making it genuinely useful — and connecting the MCP to our own documentation is a meaningful step in that direction.

It’s one more way we’re working to make sure that when you ask Jellyfish a question, you get an answer you can rely on and trust

Context

When Openclaw came out it changed the AI industry in a similar way ChatGPT did and rapidly became one of the fastest growing repos in history to have more stars.

Even major AI players named it:

As the most important software release in history, calling it the “new operating system” for agentic AI (1)

During launch in February our journey started with major roadblocks - it seemed to be a great tool for consumers to experiment with but major security concerns were still unsolved leading to a wave of articles naming it “Unusable” or “Giving the keys to the kingdom”.

This slowed us down to verify if concerns were genuine or not, but eventually good ideas win out and we decided to start simple and take the necessary precautions.

From our research, autonomous agents are becoming more present and power users are using agents to run on a single task for more than 160 minutes per our last research. Although this was based on usage of Claude Code, we realized that this same behaviour is becoming present in other agents like Openclaw and we decided to research them.

Build it simple and show it works

The first time a big idea is released it generates a lot of excitement and suspicion, fear can affect reasoning and bypass it entirely. This happened throughout history with major inventions, my favourite one is when bridges were first built and citizens didn’t believe they were safe to use. To prove it worked, it needed a “show me, don’t tell me” moment, in 1884 when the Brooklyn Bridge was inaugurated 20 elephants at the same time crossed the bridge to show it was safe - show me don’t tell me.

We decided to start simple and move forward with more complex use cases as we go. If we could at least demonstrate in a simple environment that Openclaw is safe, then we could move faster to understand Openclaws capabilities.

Security

Jellyfish’s security program heavily prioritizes understanding risk and ensuring such risk is reasonable for the business and our customers before pursuing any initiatives. We do this using standard security engineering practices (threat modeling, etc.), but uniquely incorporate actuarial-style concepts to ensure there is statistical rigor in our methods.

OpenClaw deployments are a unique challenge. OpenClaw is highly configurable, and its own security model is highly dependent on such configuration (their security documentation is about 41 pages long)! Further, the attack surface is quite large and changes are made quickly. At the time of this writing, there have been 165 security advisories (48 CVEs) since its release which is unusually high.

A few things became quickly apparent during the security review:

1. There is a lot of uncertainty in the product, use-case, and non-determinism of LLMs.

2. The primary way to reliably manage risk is to ensure the impact of any issues is small.

3. The secondary way to manage risk is to constrain the capabilities. This is actually very hard because as you make a system more flexible, the capabilities increase superlinearly.

This is no surprise and suggests a decent model for thinking about risk of agentic systems:

Agentic systems’ risk profile is related strongly to two factors:

1. Flexibility of the harness

2. Sensitivity of the harness.

Graph showing risk as it relates to flexibility (f) and sensitivity (s).

Note that flexibility grows more quickly than sensitivity.

To address sensitivity, we got our first security requirement for this project: no corporate (and definitely no production) data can be remotely possible to touch.

This is a guarantee we really needed. To ensure this we proposed an architecture where:

1. OpenClaw is deployed in a:

   1. Dedicated AWS account; and

   2. No one except a few IT and security people have access to the account; and

   3. In a dedicated VPC; and

   4. In its own EC2 instance; and

   5. Security group settings that limit any network movement in the VPC; and

   6. IAM policies that allow the EC2 instance to make Bedrock API calls and nothing else

At first glance, this seemed extremely thorough, but a key risk vector remained: ultimately our employees need to interact with this service (using a Slack channel) and we can’t guarantee they won’t accidentally provide something sensitive. We also didn’t want Slack to become an attack vector if the OpenClaw instance was to become compromised. This led to another slew of (flexibility constraint) requirements like:

1. Read-only file systems

2. Dedicated internal Slack app with minimal permissions

3. Disable “Elevated Tools” in OpenClaw

4. Disable “Plugins” in OpenClaw

5. Auto-update to ensure security updates are applied

6. TLS to ensure communication is encrypted

7. No incoming traffic (yes, even with Slack as an “input”. Thanks socket mode)

This is a lot, but it was important, especially for a phase 1 experiment. The 5-year risk of deploying OpenClaw without mitigations was estimated two orders of magnitude higher than with these applied.

At this point we felt confident that we could get something deployed and used by research without there being any significant risk to the business.

Solution: Lightsail

Since we have most of our LLM models in Bedrock, using a built-in solution in AWS to run the first v0 sounded like a good first iteration.

We pursued two approaches in parallel to see which worked better:

1. Deploying our own EC2 instance and configuring OpenClaw on it manually using Docker; or

2. Using the (at the time just released) AWS Lightsail option

We quickly discovered Lightsail was the better option for an MVP, since:

1. It gave us credential-less management

2. An OpenClaw deployment that was mostly configured in a “secure default” way

3. Automatic TLS management

4. Dedicated networking and EC2 instances

Of course there were downsides:

1. It’s not particularly configurable, so doing things like read-only filesystem mounts were less trivial.

2. Some things, like filesystem access control, is more opaque.

We discovered that some of these requirements were already built in for us and it was easier to set it up instead of using an EC2 instance - with the tradeoff of losing some capabilities like setting up read-only filesystem mount directory via Lightsail.

Some other interesting features built in for us:

1. Memory Snapshots: Version control of our setup saved in AWS

2. Buckets: Save the data in one of AWS instances to provide context to our openclaw agent

Initial Thoughts and Next Steps

Good news: It worked! We now have a persistent agent running that employees can communicate with.

Bad news: It basically just acts like ChatGPT when it was first released. It can answer a few questions, perform benign actions, and use the Slack channel as persistent context, but not much else. This is not surprising: we knew this was going to be heavily constrained.

This is the start of the journey of running agentic solutions and we have specific use cases we want to unlock next for both research purposes and productivity gains, specifically, we are interested in the following use cases:

• Tracking telemetry (OTEL)

• Ability to run cron jobs for long periods of time

• Give it write access to its own workspace to see if it can automate non-sensitive workflows

• Could we have it write software + push to a dedicated repo? What happens if you do?

We will continue searching for the value/risk optimum and how far can this technology help us.

The nominal per-token rate is identical to Opus 4.6 ($5/$25 per million), but a new tokenizer maps the same input to more tokens. Anthropic’s own stated range was 1.0-1.35x, but others such as Simon Willison are reporting inflation at levels up to 46%. (He saw this, for example, on the model’s own system prompt.)

Same prompt + same words → bigger bill.

Tomasz Tunguz framed this well in his recent newsletter: smarter models used to be cheaper per outcome. Opus 4.5 needed 76% fewer tokens than Sonnet to reach the same result, making it 60% cheaper despite a higher sticker price. Opus 4.7 reverses the pattern. He calls it a sawtooth: resolution goes up and cost goes up, then efficiency gains bring it back down. Rinse and repeat. The net effect across each cycle is more tokens consumed industrywide.

This connects directly to the data we’ve been tracking at Jellyfish. Recently I shared data showing 10x token cost for ~2x throughput at the highest usage decile... and that exponential cost curve just got steeper. If your heaviest AI users were already in the $90/PR range, they may have quietly crossed into $125+ territory in the last week without writing a single line of code differently.

So what has the tokenizer tax turned out to be in reality? Our current estimate: 17%

What’s going on? First, note that input tokens are just part of the equation. In practice, the real costs depend on how much you use Claude Code, the proportion of extended thinking (since input tokens are proportionally less of the total as thinking time increases), and the type of work you do. (For example, perhaps Opus 4.7 works more efficiently, resulting in fewer tokens overall?)

To sort this out, we took a look at Jellyfish data across 17K Claude Code users who switched from Opus 4.6 to 4.7. Using a statistical model to control for individual differences in the type and amount of work across developers, we saw that switching to 4.7 was associated with a 17% increase on average, with this “tax rate” decreasing as thinking time increases (and increasing as thinking decreases).

The chart above illustrates what this looks like for three scenarios, based on level of Claude Code token consumption: low (bottom 20%), median, and high (top 20%). These figures represent typical levels of weekly usage cost, proportion of extended thinking, and total expected change due to switching from Opus 4.6 to 4.7.

The headline: the tokenizer tax is real and significant, but also less than half what you’d estimate from first principles. It’s a good reminder that these systems are complex, and each engineering team’s use cases are unique – the only reliable way to know the actual cost impact is to measure it yourself.

Main Findings:

• Sessions on average are very short in span, but the top 0.1% of agents are running for more than an hour

• New models like Opus 4.7 are increasing the time agents can run on long-running tasks

• Long running agents are more productive both in commits and PRs and do more feature work and less refactoring work compared to short running agents

• +80% of long running turns present low supervision involvement and low supervision produces the most code during long tasks

Below we present our methodology and findings in detail.

Autonomy is a nascent concept in AI agents and the industry is just adapting to the idea of delegating to a system running for hours. The trend seems to be that users will get more comfortable with computers running and taking longer to arrive at a decision. According to METR, the length of tasks AI can do is doubling every 7 months. At this pace, by the end of 2026 we will have agents that will be able to run for as long as 4 hours on a single task.

We are developing this research to better understand how autonomous runs are evolving, how productive these agents are, what type of work autonomy is useful for, and whether there is an unexpected side effect for users running these long tasks.

To study autonomy we are going to focus into two traits:

• Length of task - to capture how long agents are capable of running productively

• Supervision - understand how humans are interacting with agents running long periods of time.

Methodology

To start: what is an agent and how should we define it? There are a variety of definitions of what an agent is and how we should measure it.

Another complexity of studying agents is the fast evolution of the agent landscape and its workflows, as we concluded in our previous research: In a span of months new workflows appeared moving the user away from the IDE to a fully agentic workflow

We took a pragmatic approach to studying agents and we used the definition that an agent is “An LLM agent runs tools in a loop to achieve a goal”.

For this research we won’t propose a definition of what autonomy is and simply define a threshold to categorize agents per turn duration and human involvement with the agent. To understand autonomy we are going to focus on agents at the top percentile, of turn duration. We define a turn as a sequence of consecutive CLI interactions bounded by human input. To analyze supervision, we condense turns into sessions, since productivity metrics align more naturally at the session level than at the turn level.

Our main findings are derived from a proprietary dataset that combines OTEL schema telemetry from Claude Code with productivity metrics sourced from GitHub signals. More information around metric definition can be found in the appendix.

Main Findings:

Sessions on average are very short in span, but the top 0.1% of agents are running for more than an hour

How long are agents running autonomously?

Figure 1. Distribution of Claude Code session duration in wall minutes across percentiles (P50, P90, P95, P99, P99.9). The median session lasts just 1 minute, but durations grow sharply in the long tail — reaching 8 minutes at P95, 27 minutes at P99, and 94 minutes (~1.6 hours) at P99.9.

We can see how the median session of an agent has a one-minute duration. To understand autonomy in agents, it is useful to zoom into the long tail and understand the behaviour of longer sessions, where only a subset of agents are running long-running tasks.

Understanding the evolution of long tasks provides us a view of how users are running agents in new creative ways and more complex tasks.

New models like Opus 4.7 are increasing the time agents can run on long-running tasks

Are long running agents running for longer periods of time?

One interesting pattern was to see the trend of how long-running tasks are evolving with new models. We found that for the last 2 months Opus 4.6 was stable in terms of long-running turns, while when the new model Opus 4.7 was released we realized that this new model pushed the boundaries on how long agents are capable of running.

Figure 2. P99.9 turn duration in wall minutes over time, segmented by model (Opus 4.6 vs. Opus 4.7), shown as a smoothed trend with daily values. Opus 4.6 holds steady around 70–100 minutes, while Opus 4.7 starts lower in mid-March, spikes above 150 minutes during early adoption, and stabilizes near 90 minutes by late April.

One interesting aspect is how the time an agent is able to run has evolved. We have checked this trend across models. It’s interesting to see how users are running longer tasks over a longer period of time, but this is only true when they run the latest models like Opus 4.7.

It seems that with each new model release users start exploring capabilities and gaining trust in how long new models can run, until it reaches a stable range where long runs are consistent. The increase in time isn’t immediate, and we think this happens because models get more efficient and users need to explore capabilities until they are able to inquire around longer tasks.

Long running agents are more productive both in commits and PRs

How do long running turns compare in productivity vs short running turns?

Figure 3. Average commits per turn and merged PRs per turn across four turn-duration cohorts (Below P90, P90–99, P99–99.9, P99.9+). Both metrics scale roughly linearly with duration: commits per turn rise from 0.020 to 0.404 (~20x), and PRs per turn rise from 0.016 to 0.178 (~11x). This analysis reflects all interactive Claude Code turns linked to git activity.

It’s interesting to see how productivity favours long-running agents. Although they spend more than an hour running these tasks, their productivity is substantially higher than that of agents running a short span of tasks.

This pattern is reflected in the idea of tokenmaxing: to see more-than-proportional gains in productivity we need to look at the top 0.1% of long-running tasks. It seems that long-running agents are paying off on whatever task they are doing, which uncovers the following question: what type of tasks are these long running agents running? Are long running agents generalists or specialists? We explore this in the following section.

Long running agents do more feature work and less refactoring

What type of tasks are long running agents working on?

Figure 4. Pull request type distribution (feat, fix, refactor, chore, other, test, docs, infra/style) by turn-duration cohort. The mix is largely stable across cohorts — feat ranges 22–25%, fix 25–27%, and refactor 11–14% — suggesting long-running agents are not specialized but rather doing the same work mix at greater volume. This analysis reflects PRs attributable to Claude Code turns across the duration distribution.

If we had to categorize long-running agents, they would look closer to generalists doing more work overall but in similar proportions to their short-running peers. With the caveat that they tend to do slightly more feature work (~3% more) and slightly less refactoring (~2% less).

Most of feature tasks are related to API or UI services

Feature development tasks by P99.9 sessions

Figure 5. Distribution of feature types built across long-running (P99.9) sessions, ranked by share of feature-development sessions. API & Backend Services (18.2%) and UI & Frontend (14.7%) dominate, followed by Authentication & Authorization (10.6%), Data Pipelines & ETL (9.4%), and Integration & External APIs (8.5%); the long tail covers Database & Storage, Notifications, Configuration, Monitoring, and Security & Compliance, each below 6%. This analysis reflects 341 P99.9 feature-development sessions classified by commit and PR text.

If we zoom in into feature development to understand if this is productive work or the agent is drifting off to non-related feature work we actually see how commits and PRs are building all sorts of features, being backend and frontend the most relevant in terms of sessions.

But why is it that agents run for long periods of time to accomplish these tasks, our hypotheses are: either agents are drifting from their main objective and taking longer than needed or because the agent needs to wait in a sequential workflow that has high external dependencies.

From a small sample of P99.9 long runs we were able to see how one agent was running over 3 hours because it was waiting for a CI/CD process to finish or because an agent was running a cron job every 10 minutes without any output just checking almost as a “bot” and less than an agent.

We studied in depth each of these feature types that are acting like agents (excluding any bot behaviour) and reached on why each of these features might be taking longer times to finish by analyzing commit text and PR text.

Figure 6. Qualitative deep-dive into the top six feature categories from long-running (P99.9) sessions, pairing each category’s share with a “why it runs long” annotation. API & Backend (18.2%) drives duration through iterative refinement and waits on CI/CD, integration tests, and cross-service coordination; UI & Frontend (14.7%) and Data Pipelines & ETL (9.4%) run long mainly through idle waits on E2E suites, visual regression checks, and Spark/schema-migration jobs; Auth & Authorization (10.6%) shows the lowest idle ratio — meaning more genuinely active development on security and edge cases; Integration & External APIs (8.5%) is dominated by waits on third-party systems; AI/ML & Analytics (6.5%) reflects model-training and experiment-validation cycles.

Although agents run for long periods of time, the agent probably isn't active the whole time — it's waiting on external dependencies, depending on what the task is doing. Long agent runtimes might just be a reflection of how today's software is built, and as agents compress their internal cycle time, inefficiencies in sequential workflows will become more evident.

Supervision

We define supervision as the percentage of time a human is involved per turn. We want to understand how supervision affects long running tasks and code quality output.

Is supervision affecting quality of work, and if so how?

+80% of long running turns present low supervision involvement

Figure 7. Share of turns falling into Low (≤30%), Medium (30–50%), and High (>50%) supervision bands across the four turn-duration cohorts, where supervision is defined as the percentage of turn time the human is actively involved. Low supervision rises sharply with duration — from 9% in sub-P90 turns to 83% in P99.9+ turns — while high-supervision turns nearly disappear in the long tail.

Low supervision produces the most code during long tasks

Sessions where the human barely types (≤30% user share) generate 11x more net lines per turn than high-supervision sessions.

Figure 8. Average net lines of code (additions minus deletions) per turn, split by duration cohort (Short, Medium, Long) and supervision band (Low, Medium, High). Long-duration, low-supervision turns produce ~11,800 net lines on average — roughly 11x the output of long-duration, high-supervision turns (~1,000 net lines) — and the gap is invisible in shorter cohorts.

As we observe, the long-duration turns tend to create more lines of code when there is low supervision, reflecting the new workflow of “letting it run” and letting agents build the code without checking or corroborating. What’s interesting is that when supervision is high for the longest runs, the net added code is less than that of the previous (medium-duration) cohort. This could be because users running agents for longer also have a higher bar on correcting what Claude is generating as a solution or simply because the type of task requires less code and more external dependencies.

We don’t know if human supervision it’s because there is degradation after running for long hours(and more human correction is required), or simply because of the type of task, which requires less code when there is human involvement. Low supervision may also correlate with sessions where the agent drifts — taking the user down a rabbit hole without converging on a stable solution. Drift, coherence, and supervision are areas we plan to continue monitoring in future research.

What’s Next?

If 2026 was the year of coding agents, 2026-2027 will be the year of autonomous agents running 24/7. We suspect that new bottlenecks will emerge: like infrastructure and reviewing the work agents are running in the background. Infrastructure will become critical as agents scale up, where having a surface area available for any numbers of agents requesting programmatic access will be business critical.

In the future we want to follow our research on how code quality might be affected by autonomous agents and how solutions like Openclaw, Hermes agents, NemoClaw and headless Claude Code agents are being used under the new SDLC model. If these solutions end up being the new industry standard then agents will diffuse into other areas beyond coding.

Notes & Limitations:

• Data coverage: Can’t differentiate between a human interrupting a run vs. providing more context to the agent.

• User linkage: Only sessions from companies with GitHub integrations (via Jellyfish) produce commit/PR linkage. Companies without GitHub data contribute turn-duration metrics but not productivity metrics.

• OTEL Schema: The schema we use as data source, doesn’t expose a direct invocation-mode field, so we’d have to infer it from terminal.type and query_source, which we haven’t done.

Appendix

• Agent: Unique session of a Claude Code instance - compatible with Simon Willison’s definition of an agent.

• Session: One unique session_id in the OTEL telemetry. Represents a single continuous Claude Code process invocation.

• Turn: A stretch of consecutive CLI-active minutes within a session, bounded by human interactions. Starts at the first CLI minute after a user minute (or session start) and ends just before the next user minute (or session end).

• Active minutes: Count of minutes in the turn with at least one CLI heartbeat.

• Wall minutes: (turn_end − turn_start) / 60 + 1. Includes idle gaps where the CLI was not active

We found that not only does investing in these customizations increase pull request output, the results compound – the more you invest in these tools, the more additional PRs you ship.

Controlling for other factors (e.g. individual baseline output, temporal trends, and overall AI usage intensity), developers with 6+ customizations ship an additional 2.3 PRs/week over what they would have done otherwise.

Overall, doubling your cumulative investment in these tools is associated with an additional +29% increase in PRs merged per person per week.

So, developers who are spending time teaching their AI tools are shipping meaningfully more code. Will be interesting to see how this trend evolves as context engineering becomes more of a team-level practice rather than an individual one!

I get the appeal. It’s a crude metric, but it can be useful in a crisis... kind of like checking someone’s pulse tells you they’re not dead. But just like your heart rate, you want to be thoughtful about when and how you try to max it out.

This chart tells the broader story: pull request throughput and tokens per PR by token usage decile across ~7,500 engineers in Q1.

The PR throughput curve (purple) is encouraging: it climbs steadily from 0.77 PRs/week at the lowest token usage to 2.15 at the highest. More tokens = more output.

But look at the token cost curve (orange). Tokens per PR goes up exponentially. The median developer uses about 7M tokens per PR, versus 69M (!!) at the top decile. That’s roughly 10x more tokens for about 2x the throughput.

The practical implication is that when it comes to tokenmaxxing, there’s a sweet spot. You’ll get way more bang for your buck by getting everyone in your org into the middle of the adoption curve (D4 through D6) than by pushing a small group into the stratosphere. Broad, moderate adoption beats narrow, extreme usage.

What about the promise of massive productivity gains from fleets of autonomous agents? Those can absolutely be unlocked, but they require serious investments in agent infrastructure, sandboxed environments, and context engineering, and most companies in 2026 are still in the early stages of addressing these challenges. Until those are overcome, there's still an "agentic barrier" – a speed of light you can't get past, no matter how many tokens you burn.

We analyzed millions of data points from Claude Code telemetry and we asked: How many agents each user is running at the same time ? Are users spending most of their time with multiple agents ? How many companies are running multi-agentic systems ?

Main Findings:

User activity findings

• Multi-agents systems are just starting out and users are mostly running one or two agents most of the time. Using multiple agents at once is not the most adopted pattern yet. 84% of users are still running one or two agents at a time.

• Users running multi-agent spend most of their time with one agent. Users running more than three agents at a time, still require to spend most of their time in a single agent terminal. Even for users using multiple agents, 80% of their time is spent within a single agent. We can’t verify how this workflow is being used but we have some hypotheses:

  • Attention Span: Users hit the attention barrier when managing more than two agents at the same time.

  • Background Autonomy: Some agents run in the background with minimal supervision needed, reducing the need for active attention showing early sparks of background autonomy.

Users running multi-agent systems are more productive per hour, and more engaged overall. They commit more, are more productive per hour and run agents with more background autonomy. Engagement and maybe curiosity seems to be what is sparking this set of users to run multiple workflows in parallel.

Company activity findings

• 2.5% of companies are using a “fleet of agents”, the majority of companies are running 2 agents in parallel.Agents are being implemented in an orchestrated pattern but we are not seeing a significant number of companies implementing agents at scale. We suspect this is related to infrastructures challenges, budget limitation or educational learning process.

• Companies running a fleet of agents spend only 14% of their time in using multi-agentic mode. We can’t validate if this is a consequence of the multi-agentic system or a limitation of the current state of companies implementing this pattern. This pattern follows the same user conclusion. Are companies running true background autonomy agents that don’t require supervision or interactive agentic workflows per user are limited by the initial experiment and system design?

Below we present our methodology and findings in detail. Our main conclusion is that managing multi-agent systems will require a new paradigm. Neither current infrastructure nor human attention is sufficient to scale agent concurrency through interactive management alone. The transition from interactive concurrency to background autonomy—what we might call the “agentic barrier”—is the phase change the industry needs to navigate. Novel solutions are being developed by startups and power users to build orchestration layers that reduce the need for constant human attention. This was our first step toward understanding how users run agents concurrently, the current stage of maturity, and what comes next.

Methodology

What is an agent and how should we define it? There are a variety of definitions of what an agent is and how we should measure it.

Another complexity of studying agents is the fast evolution of the agent landscape and its workflows, as we concluded in our previous research1 - In a span of months new workflows appeared moving the user away from the IDE to a fully agentic workflow.

With these two challenges, how should we study and measure agents running concurrently?

We took a pragmatic approach to studying agents and we used the definition that an agent is “An LLM agent runs tools in a loop to achieve a goal”.2

Next, we defined a set of metrics to measure agentic concurrency using Claude Code telemetry data. The dataset covers a rolling 30-day window with ~19M rows of active time data. This dataset offers a useful sample for studying concurrency at the session level, with its own limitations: it captures only one tool in the market and does not reflect agent concurrency across different providers.

Main Findings:

Multi-agents systems are just starting out and users run one or two agents

How many agents are users running concurrently? We measured this by tracking how many Claude Code sessions are active in the same time interval per user.

Most users run agents solo or in pairs—84% of users still run one or two agents at a time. To our surprise, we expected a heavier distribution toward three or four-plus concurrent agents.

Figure 1. Distribution of users by peak concurrent agents. The vast majority of users (84%) run only one or two agents at a time, with a steep drop-off at three agents (11.4%) and only 4.7% reaching four or more. Data reflects the 30-day window of Claude Code telemetry.

It turns out the average user may still be in the early stages of the learning curve. But there is also a deeper question: Can we sustain multiple spans of attention to run different projects at the same time?

One limitation of our data: we cannot determine whether users running multiple agents are breaking down the same project across sessions or running entirely separate projects. We also cannot see whether individual sessions spawn Claude Code sub-agents internally. Nevertheless, we asked: Are users interacting simultaneously with all their agents equally?

Users running multi-agent spend most of their time with one agent

Can users actively work with multiple sessions in parallel ?

It is surprising to see that running multiple agents simultaneously doesn’t mean that agents are being used equally but rather in a more passive manner. Is this a feature or a bug ? We can’t verify how this workflow are being used but we have some hypotheses:

1. Attention Span: Users struggle to sustain the same level of attention across all of their agents. As Andrej Karpathy put it, “We will need a bigger IDE”3. The current tooling assumes a single-focus interaction model, and splitting attention across multiple terminals degrades the quality of each interaction.

2. Background “autonomy” emerging: Some tasks require less active supervision than others, allowing certain agents to run semi-independently. This is not true background autonomy—where agents execute end-to-end without human input—but rather a transitional pattern where one agent gets active attention while others idle or run simple tasks.

Figure 2. Time-weighted concurrency distribution by user tier. Even users who have reached higher peak concurrency levels spend the overwhelming majority of their active time—over 80%—focused on a single agent. Only Sustained users (peak ≥ 4) show a meaningful share of multi-agent time, with 13.8% at two agents and 3.9% at four or more.

The attention barrier limits interactive concurrency. When a user manages multiple agents interactively, they are context-switching — prompting one agent, waiting, switching to another, prompting again. The agents take turns being active because the human can only drive one at a time. Adding more terminals does not add parallel throughput; it adds more things to serialize through. Current tooling assumes a single-focus interaction model, and splitting attention across agents degrades the quality of each interaction rather than multiplying output.

Background autonomy is what unlocks actual concurrency. The small fraction of time that users do sustain multiple agents maps almost directly to moments where one agent runs semi-independently while the user drives another. This is not yet full background autonomy — agents executing end-to-end without human input — but a transitional pattern where one agent receives active attention while others coast on simpler or longer-running tasks. It is this emerging autonomy, not multitasking skill, that creates the real multi-agent minutes in the data.

The implication is clear: scaling AI leverage per developer is not a matter of opening more terminals. It requires agents capable of independent execution and tooling designed to orchestrate them. Until then, human attention remains the binding constraint on AI throughput.

Multi-agent users are more more productive per hour and more engaged overall

Figure 3. Behavioral comparison across peak concurrency tiers. Sustained users (peak ≥ 4) show dramatically higher autonomy (47.5% vs 0%), commits per hour (2.3 vs 0.0), total active hours (32.5 vs 0.7), and total commits (69.5 vs 0) compared to single-agent users. Each panel shows median values per tier across a 30-day window

Company View

We wanted to explore how companies are running agents to see whether organizational patterns differ from individual user behavior. In this section, we analyze company-level concurrency patterns and the maturity stages that emerged.

We categorized each company’s maturity stage by evaluating the behavior of the majority of its users and assigning the corresponding level.

2.5% of companies are using a “fleet of agents”, majority of companies are running 2 agents in parallel

What level of maturity are companies exhibiting ? How many agents are companies running concurrently?

Figure 4. Agentic maturity distribution across companies. The majority of companies (58.1%) fall into the Orchestrator tier, typically running two agents in parallel, while 39.4% remain at the Starter stage. Only 2.5% of companies have reached the Fleet+ tier, indicating that large-scale multi-agent deployments are still rare.

Interestingly when we analyze behaviour at a company level, we see that the majority of companies are acting as orchestrators. Meaning that the majority of users per company are running two agents in parallel.

We suspect this is part of a learning curve where users are getting more proficient and companies providing richer tools for orchestration and budget to test and learn. We can’t validate if this is a trend where we will see more companies running fleets of agents. But based on our previous research, it seems that we will have more advanced infrastructure to run a fleet of agents, which could accelerate movement along the maturity curve.

Companies running fleet of agents, spend only 14% of their time in using multi-agentic mode

Figure 5. Global and per-stage breakdown of time spent in multi-agent versus single-agent mode. Across all companies, only 6.1% of active time involves multiple concurrent agents, out of +7 million total active minutes

Companies running fleets of agents spend only 14% of their active time in multi-agent mode. This mirrors the individual user finding: interactive concurrency—actively managing multiple agents—consumes a small fraction of total time, likely because human attention remains the binding constraint. At the same time, we can’t validate if these are the first sparks of background autonomy of agents running in the background without requiring the user’s attention.

Independently if any of these statements are true, this suggests a future where managing this fleet of agents could require less active time on each individual agent but more time in a meta system organizing and orchestrating agents - something like what Cursor built around self-driving codebases orchestrating thousands of agents simultaneously or coordination apps emerging like Conductor app

More active days, more multi-agents minutes, and agents running outside of normal hours

Are mature companies doing something different vs nascent ones ?

Figure 6.Behavioral triggers that differentiate company maturity stages across four dimensions. Fleet+ companies show substantially higher active days (21 vs 5), sessions per active day (7 vs 2), multi-agent minutes (194 vs 2), and off-hours activity (223 vs 20 minutes) compared to Starter companies.

The off-hours activity signal is particularly noteworthy. Fleet+ companies show 10x more off-hours agent activity, suggesting early signs of background autonomy—agents running outside of normal working hours with reduced human supervision. Could this be the first spark of experiments of long running agents in the background, trying to replicate an OpenClaw experience running their own computer 24/7 ? That could also be probably true.

Notes & Limitations:

This research was a first step. We analyzed what we can with current agentic activity and we encourage transparency of the limits that our data can’t tell:

• We analyzed Claude Code telemetry at the session level. We defined a Claude Code session as a unit of agent activity, but we could not measure sub-agents spawned by Claude Code within a session.

• This analysis covers a 30-day period, which is our current analytical window. Agentic patterns like concurrency are emerging and evolving quickly. We plan to extend this analysis over time to detect trends, but the current time frame limits what can be observed.

• We wanted to contrast different providers to compare Claude Code’s agentic features against alternatives, but we were limited by the data currently available to us.

What’s Next ?

Seeing only a small fraction of companies showing multi-agentic patterns reminds us that the industry is moving fast, but organizations and companies require time to implement new features and patterns in their current workflows. Understanding how to measure agentic behaviour like concurrency was a relevant first step to understanding how the development cycle of software is evolving.

The agentic barrier(or attention barrier) is a central challenge. Our data shows that the transition from interactive concurrency to background autonomy is where the industry stalls. Users can spin up multiple agents, but they cannot meaningfully interact with more than one or two at a time. Crossing this barrier—through better orchestration tooling, trust frameworks, and infrastructure—is the phase change that will unlock true multi-agent productivity.

Infrastructure for background autonomy is needed. Companies should start asking how their infrastructure will support running a fleet of agents that operate with minimal human oversight. This goes beyond simply scaling terminals—it requires new patterns for monitoring, error handling, and human-in-the-loop escalation.

From a research perspective studying autonomy is an aspect that we will monitor to understand how this transition is happening and how users are solving some of the challenges mentioned.

Appendix

• Agent:Unique session of Claude Code instance - compatible with Simon Willison definition of an agent.

• Concurrency: How many agents run simultaneously

• Interactive Concurrency: Agent running at the same time requiring human interaction to run

• Background Autonomy: Agents capable of running with minimal or null human intervention

• Peak Concurrency: Maximum number of agents running at a specific time

• Autonomous Streak: Consecutive minutes in a single session with only CLI activity, unbroken by any user action (type=user). Gaps in agent activity (e.g., waiting for API or build responses) do not break the streak; only human intervention does.

• Autonomous CLI Minutes: Total CLI-only minutes within qualifying autonomous streaks for a user or session. Minutes from short, non-qualifying streaks are excluded.

• Autonomous Hours: This measures the total independent agent work time in qualifying streaks.

• Autonomous Streak Count: The number of qualifying autonomous streaks for a user or session.

Concurrency Type:

User tiers are classified by peak:

• Single (peak = 1)

• Experimenter (peak = 2)

• Explorer (peak = 3)

• Sustained (peak ≥ 4)

Company maturity stages use the company median max peak:

• Starter (≤ 1)

• Orchestrator (≤ 3)

• Fleet Manager (≤ 6)

• Fleet+ (> 6)

Companies are categorized by their dominant active cohort to assign a maturity stage

We analyzed usage from three AI coding tools across our customer base throughout 2025 — GitHub Copilot, Cursor, and Claude Code — and asked: what does each tool actually do, how should we measure, and what would a fair comparison look like?

What We Measured and How

Our dataset covers 90M suggestions from January through December 2025, aggregated through the Jellyfish platform from vendor APIs. For each tool, we tracked suggestion acceptance rate, line acceptance rate, lines per suggestion, and lines per acceptance.

We then categorized each tool to a different group that reflects how engineers are using these new tools and new workflows emerging.

A Framework for Understanding AI Coding agents

We started by analyzing and comparing suggestion acceptance rates, and immediately saw that each tool sits at a different point on the spectrum. On one end is the full agentic experience of Claude Code; on the other are GitHub Copilot’s inline completions and Chat at lower to mid acceptance rates. This tells us something important: each tool is being used in a fundamentally different way, for a different purpose, at a different level of trust.

We use suggestion acceptance rate (%) to position each tool and define four distinct workflow tiers:

We drew some distinctions within each product to compare fairly, within the limits of our data taxonomy. For example, GitHub Copilot spans W1 and W2. Cursor operates at W3. Claude Code operates at W4.

Comparing normalized suggestions

What Is a “Suggestion”?

Suggestion is defined different across tools:

• Copilot/Cursor: How often users Tab-accept the inline ghost completions or block of code.

• Claude Code: How often users approve the proposed file changes or commands

Since we identified four different workflows, we wanted to understand whether this behavioral difference was reflected in both suggestion acceptance and line acceptance rates:

Findings

Normalizing by lines rather than suggestions exposes something important about how each workflow actually works.

1. Copilot completions: A 5-point gap between suggestion rate and line rate. The units are small enough that partial acceptance rarely makes sense — what gets accepted, stays.

2. Cursor: A 40-point gap. Developers accept 4 out of 5 Cursor suggestions — but then keep only about half the lines. The real work happens after acceptance: reviewing the diff, editing what doesn’t fit, deleting what was wrong. Cursor’s 81% suggestion rate describes the starting point of a review process, not the end of one. We also think that user when using the IDE have a workflow of reviewing the code and rejecting lines because the review is part of the process. Since Cursor uses multiple models there is a follow up question: Could it be that some models are affecting the output more than others? Maybe

3. Claude Code: Line acceptance rate can’t be calculated from available data, but the workflow is inherently binary. When you accept Claude Code output you’re taking an entire implementation — there’s no line-by-line diff to partially approve. Developers either trust the result and move forward, or reject it and iterate. This workflow makes sense, it’s the type of continuous agent workflow that developers work with a forward mindset - instead of reviewing and deleting what doesn’t work the user accepts most of the code in blocks, checks if it’s functional and re-accepts another code of block to fix bugs.

Trends: How are workflows evolving ?

We can see how the trend and evolution of acceptance rates can show us how each workflow evolved in the market.

In May–June 2025, Cursor released its 1.0 experience with significant enhancements including “background agent” and “tool search enhancement.” This was a pivotal moment: acceptance rates spiked and the tool shifted firmly into W3 (IDE-Native), moving away from the chat-heavy W2 experience that had characterized earlier usage. Once Cursor stabilized above 90% acceptance, that metric essentially lost signal.

On the other side, Copilot Chat started 2025 around 50% acceptance and stabilized at 10–20% — a substantial drop. We think this reflects a shift in how teams are using it: less for direct code generation, more as a contextual thinking tool. Chat becomes a way to understand a problem or explore an approach before actually building, not a way to ship lines. An open question worth following: did users migrate from using W2 alone, or did they converge on a combined W2 (understanding) + W3 (building) workflow? Something for future research.

This suggests two things:

a) Tools are evolving quickly on their core use case — not their peripheral features.

b) Knowing which workflow a tool occupies — and when that changes — is critical for measurement. After Cursor’s 1.0 release, it moved firmly into W3 territory, which means acceptance rate no longer has much signal. The right metric to reach for shifts to how developers are editing the code the IDE generates.

Now that we understand how each workflow behaves, we can find the metrics that actually correlate with quality in each one.

One Metric per Workflow

To find the right metric for each workflow, it helps to invert the question: how do we know when a team is using a workflow poorly?

• For Claude Code (W4), frequent interruptions signal that teams aren’t letting the agent run — which undermines the core value of full agentic output.

• For Cursor (W3), heavy post-acceptance editing means suggestions aren’t landing well, often due to insufficient context engineering in the repo.

• For Copilot Chat (W2), if the chat isn’t providing useful answers, engineers switch to other sources and the workflow breaks down.

• For Copilot Inline (W1), a flat or low acceptance rate tells you directly whether suggestions match the patterns engineers actually want to keep.

Each metric is designed to detect process breakdowns — so we can surface signals about where the workflow is working and where it isn’t.

Industry Benchmarks:

Comparing AI tool performance across companies is genuinely difficult — especially given how fast these tools are evolving and how differently teams use them. Still, looking at benchmark ranges helps teams understand where they stand and calibrate their expectations.

We decided to explore how it looks like and see if we find interesting points to consider when using this framework:

Copilot improving, Cursor with a healthy rejection rate, Claude Code with near-zero revert rates

• Copilot Inline: Approximately 3 out 4 suggestions are still rejected, but we are seeing how this numbers are improving for the past 6 months. Even among top users, acceptance peaks below 28%, suggesting there’s still room to improve suggestion fit.

• Copilot Chat: Counter-intuitively, most teams using Chat are moving slower, not faster. Our hypothesis is that teams are reaching for Chat on harder, more ambiguous problems — which increases cycle time by design. This makes Chat’s value harder to see in throughput metrics, but that doesn’t mean it’s absent. (See below for detail.)

• Cursor: Median rejection rate of 13.7%. A lower rate indicates stronger fit between generated code and what developers actually keep. (Note: True post-acceptance edit rate wasn’t directly available; suggestion rejection rate was used as a proxy.)

• Claude Code: Agentic output is being merged with near-zero revert rates at the median. This could mean strong output quality — or it could reflect the early adoption effect, where teams running lower-stakes experiments simply aren’t generating the kind of code that gets reverted later. Separating these explanations will require more time and higher-volume data.

A Closer Look at Copilot Chat

The “AI makes teams slower” finding deserves more detail on nuances. We split PRs into Simple and Complex (by number of files changed) and compared cycle times to see if teams where spending more time on harder challenges.

AI-assisted vs. non-AI cycle times across percentiles:

What we found:

• Simple PRs: Chat actually widens the gap — AI-assisted PRs are slower across all percentiles. Suggests developers are over-engineering or over-consulting via chat for work that is small and considered simple ? Maybe, we need to validate this further.

• Complex PRs: At Median and P75, the gap persists — AI is still slower. Selection bias (using Chat on harder work) could explains part of this, but not all.

  • At P90 on Complex PRs the gap essentially vanishes: 7.92d (AI) vs 8.00d (non-AI). This means that best teams, on the hardest problems, are moving faster with AI to solve the most complex challenges

  • The teams who extract that value (P90) do so on complex PRs where the alternative isn’t “do it faster without AI” — it’s “get blocked.”

Conclusions

Measuring AI coding tools fairly requires accepting a foundational premise: these tools don’t all do the same thing. Comparing Copilot’s suggestion acceptance rate to Claude Code’s revert rate isn’t an apples-to-apples comparison — it’s comparing individual keystrokes to entire implementations. Having a framework to understand different workflows gives teams a principled way to identify which measurement approach fits each tool in their stack. Like we have seen, tools and workflows are still evolving and that means checking in what is the right metric up-to-date.

We also seen how some teams have extracted good value from this tools like increasing cycle time or reverting only a fraction of code. Some of this suggest a natural learning curve that we are seeing. On one hand only advanced teams are gaining efficiencies, but on the other time we see cycle time increase suggesting that gains remain to be seen for the beginners.

Methodology

• Coverage: January 1 – December 31, 2025

• Metric definitions

• Suggestion acceptance rate: % of AI-generated suggestions accepted by the developer

• Line acceptance rate: % of suggested lines retained after acceptance

• Post-acceptance edit rate / Suggestion rejection rate: % of accepted suggestions where generated lines were subsequently removed or modified

• Cycle time delta: Difference in PR cycle time (days) between AI-assisted and non-AI PRs, compared at matching percentiles (P50, P75, P90)

• PR revert rate: % of merged PRs subsequently reverted, segmented by AI tool usage

In fact, we saw on average that Jellyfish customers added about 30 more bugs (per 100 engineers) to their backlogs than they resolved last quarter.

However, I’m hearing from a lot of folks pushing hard on AI transformation – especially those getting good context engineering in place and deploying autonomous agents – that they’re using these tools to better tame their bug backlogs.

Interestingly, the data bears it out. The plot above shows net bugs added in Q4 2025 versus AI adoption level (percentage of engineers that are frequent AI coding tool users) for a set of 500+ Jellyfish customers.

At the lowest AI adoption levels, companies add ~43 net bugs per 100 engineers per quarter. At the highest? Around 22 – cut nearly in half.

Maybe someday AI coding tools and agents will help us get to a place where bug backlogs actually shrink instead of grow... or maybe we’ll have a whole new set of quality challenges to deal with!

This chart shows average lines of code added per person per week, broken out by AI coding tool, from Jellyfish data over the past two months. In January, Claude Code users averaged 2.6x the lines added compared to engineers not using AI tools. Cursor came in at 1.6x, GitHub Copilot at 1.5x.

One thing that jumped out: look at the end of December. Activity (predictably) dipped across the board for the holidays... except Claude Code, which spiked. If you were online over the break, you may have heard folks talking about “Claude Christmas”, where the November Opus 4.5 release + increased usage limits + holiday downtime led to a big surge in usage. Our data corroborates the hype: whatever was happening over the holidays, Claude Code users were shipping.

To be clear: lines of code isn’t the whole story – quality is paramount, more code isn’t always better code, and all this volume has the potential to create bottlenecks elsewhere in the SDLC. But this is a striking signal about how much raw output varies by tool, and it’s consistent with the PR throughput patterns we’ve been tracking.

Last year was about exploration. This year is all about scaling. If January is any indication, it’s going to be a wild ride!

1. Data now outputs in TOON format instead of JSON. How you send data to AI models matters more than you might think—send too much or in the wrong format, and you’re wasting tokens. We ran an analysis on Jellyfish data comparing formats and found that TOON, a YAML-like format, is more token-efficient than JSON in most cases. The MCP now automatically translates data into TOON before sending it to your AI host application.

2. Rebuilt from Python to Node.js. The MCP server and Claude Desktop extension now share the same codebase in Node.js, which means faster updates and more consistent behavior across platforms. The Python version is no longer being updated, but the source code for the previous version is still available here.

3. Docker support. We published a Docker image on Docker Hub so you can run the MCP server in a container. Once configured, Docker handles everything—no manual installs, and updates pull automatically so you’re always on the latest version.

What this means for you: Nothing changes about how you use the MCP. Just make sure you’re on the latest version. If you want to try the new Docker setup, follow the directions in the README, and once configured, it will automatically stay up to date.

The hardest part of building an MCP isn’t the engineering — it’s managing context. Every tool response sends data to an LLM, and figuring out how much context to include is an ongoing challenge when the model is essentially a black box. There’s no formula for getting it right, at least not yet. So we’re iterating: testing different approaches, exploring new ideas, and learning as we go. The switch to TOON came directly out of this process — and it won’t be the last change we make. If you’re building an MCP too, our experience has been that this kind of iteration is where the real progress happens.

An emerging question is the return on investment (ROI) for AI and how should companies analyze it. Particularly we focused on how the nature of the programming language used significantly influence the acceptance rate and, consequently, the ROI?

We analyzed 750+ million lines of AI-suggested code across companies using multiple AI coding tools and discovered that programming language choice affects the AI ROI.

Introduction

Every three-six months, a new LLM record is shattered on a public leaderboard, yet for the average developer, the “revolutionary” shift rarely feels reflected in the daily git commit. This was true until Anthropic released Opus 4.5 and suddenly the impact was a “before-after” Opus 4.5 era, code started to go at another quality and speed level.

But here’s the question that experience raises: how do you actually measure that impact?

While industry benchmarks focus on capabilities like quality, performance and safety, they often miss the metrics that truly matter to a business: productivity gains or value generated.

Quantifying the value derived from lines of code is inherently complex, subjective, and prone to disagreement. Ultimately, value is a business-level concept, tied directly to the organization’s mission and assessed through core business metrics. However, we can use the adoption of generated “tokens” as a proxy for value creation. If an AI tool produces valuable output, its tokens are adopted; if not, they are rejected. By measuring the ratio of accepted to rejected tokens within our codebase, we could gain a clearer perspective on the value (or at least the productivity) generated by using AI, and consequently a proxy of its ROI.

In this piece, we will measure acceptance rate not with tokens, but rather with lines of code. If tokens are the new “currency” of AI, then acceptance rate(lines accepted/lines generated) is the exchange rate that tells you what that currency is actually worth in the real world.

Main Findings

Our analysis revealed a clear and consistent pattern across the industry:

• The Language Gap is Real: Acceptance rates for code generated by code languages like Go, Python, and Ruby are significantly higher, achieving 22–30%, compared to configuration languages such as JSON and YAML, which see only 10–20% acceptance. This represents a persistent 2–3x performance difference across all companies.

• Language Type Predicts Success: Go is the most accepted language among the options, leading with a 30.28% acceptance rate across 121 companies. Python, despite having a high volume of suggestions (16.2M lines), has a 24.74% acceptance rate. JSON trails significantly, with only a 10.30% acceptance rate across 151 companies.

• Volume Doesn’t Explain It: There is no correlation between languages acceptance rate and volume of code generated - quantity of lines of code generated does not affect accepted lines.

Coding Language: The most valuable language per output

When we group languages by category: Code,Config and Markup the separation becomes clear: Coding leads the value generated in contrast to markups. Comparing the medians between coding vs markup languages we see almost a 2x between these two. We think the reason why this gap exist is because of three characteristics innate of each language: Complexity, Context and Variation

• Code languages: Median ~24%, range 21-30%

• Config languages: Median ~15%, range 10-20%

• Markup languages: Median ~13%, range 10-14%

Language Categories Distribution

Complexity, Context and Variation

Languages that have diverse patterns, flexibility and rich signals like coding languages tend to thrive in LLM environments, in contrast to config files that are rigid and limited - below some examples of differences between languages:

AI offers high acceptance rate in code languages (like Python, Go, TypeScript, Java) due to their complexity, varied patterns, and rich contextual information (imports, types, hierarchies). This allows AI to provide genuinely helpful alternatives and completions.

Conversely, acceptance rate is lower in formulaic configuration languages (like JSON, YAML). These have low pattern complexity, require one correct format, and are often context-poor (standalone key-value pairs). Since these formats are brittle, developers reject “close enough” suggestions, limiting AI’s value beyond basic autocomplete.

Language Matters More Than You Think: Top 10 is predominantly code language

Across 750+ million lines of AI-suggested code from multiple AI coding tools, we found a clear hierarchy. Traditional programming languages consistently achieve 2-3x higher acceptance rates than configuration and markup languages. The top ten acceptance rates by languages show how this is predominantly led by code language. If your team writes primarily config files, even the best AI tool will show “disappointing” overall numbers. That’s not a tool failure, it’s the “language effect” in action with Go having the highest acceptance rate.

Language Acceptance Rates(%)

Volume vs Acceptance: No Correlation

One question that we wanted to validate is: “Is the volume of code generated affecting the quality of tokens generated ?” The answer seems no, volume doesn’t predict acceptance. Python (most widely used) and Go (mid-volume) both achieve high acceptance rates because they’re code languages. JSON (high volume) and Markdown (low volume) both struggle because they’re configuration/markup formats.

If volume mattered, we’d see a diagonal trend. Instead, we see horizontal clustering by language type. This might be because acceptance rate is more closely related to the performance of the actual LLM in the different context engineering environments and its conditions. Although LLMs tend to experience degradation over time depending on the context length and task difficulty the correlation with language vs volume disappears.

Volume vs Acceptance

Practical Implications: What This Means for Your Team

Measuring ROI

• Measure Time Saved, Not Just Cost: Track the duration of content creation and review cycles pre- and post AI.(Jellyfish AI Impact tracks this for you !)

• Convert Accepted Lines into Financial Value: Measure acceptance rate of tokens and calculate ROI using the following function:

  ROI = [(Acceptance Rate × Total Tokens × Time Saved per Token × Dev Cost) - AI Cost] / AI Cost

Conclusion

Programming language choice has a larger impact on AI code acceptance rates than which AI tool you use. Our analysis of 750+ million lines across 239+ companies and multiple AI coding tools shows that traditional programming languages (Go, Python, Ruby, TypeScript) achieve 22-30% acceptance rates, while configuration and markup formats (JSON, YAML, Markdown) achieve 10-20% - a consistent 2-3x difference.

Before you analyze your AI coding assistant for low ROI, check your token efficiency and how language distribution affects the ROI.Before you switch tools, measure acceptance rates by language category. Before you set team targets, adjust expectations based on what languages your team actually writes.

Methodology

Data Source: Jellyfish AI Coding Assistant Usage Analytics

Time Period: Jan 2025 - December 2025

Coverage: 750+ million lines suggested across multiple AI coding tools

Analysis Approach:

• Language analysis aggregated across all tools to identify universal patterns

• Acceptance rate = (lines accepted / lines suggested) × 100

• Categories defined by language purpose: Code, Config, Markup

Key Metrics:

• Acceptance rate by language (aggregated across all tools)

• Total lines suggested per language

• Company adoption breadth per language - Category-level distributions

Large Language Models (LLMs) are powerful tools capable of executing complex tasks. As we continue to leverage and implement them, it has become increasingly clear that while a good prompt is essential, it is only half the battle. The difference between a helpful response and a hallucination often lies not just in what you ask, but also in the context you provide.

Tackling more complex applications - such as asking an LLM to refactor a codebase or analyze team performance metrics - requires providing the right context. But context is finite and fragile. Even as context windows expand from 200K (Claude) to 400K (GPT-5.2) or 1M (Gemini) tokens1, larger volumes of data can negatively affect a model’s attention, leading to lower accuracy. While newer models like GPT-5.2 are designed to improve long-context reliability and cross-document reasoning, the risk of attention saturation remains a primary bottleneck for performance.

Managing this finite space has given rise to Context Engineering. As we discover more about how contexts fail, developers are learning how to fix their context by curating the optimal set of tokens during inference. As Anthropic describes it, this practice refers to “the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference.” Limiting the number of tokens passed to an LLM is no longer just about avoiding limits or costs - it is about ensuring the model remains sharp and focused.

Jellyfish MCP

Earlier this year, we released the Jellyfish MCP server to provide data and analytics from the Jellyfish platform on team performance. The MCP wraps our external API to return information on metrics, allocations, deliverables, and more. Each MCP tool corresponds to a specific API endpoint, and our current implementation returns the endpoint’s raw response in JSON format.

As we’ve continued building the MCP and learning about LLM best practices, a constant theme has emerged concerning format optimization: structured text formats, such as Markdown and YAML, tend to be significantly more context-efficient and easier for LLMs to interpret than verbose formats like JSON.

Because the MCP simply relays output from our API, this question is central to our design: Does changing the output format meaningfully reduce context consumption without compromising the accuracy of the data? We explored format optimization as a low-complexity, high-impact technique to manage context bloat.

Methodology

Since the MCP currently returns raw JSON from the Jellyfish API, we decided to test three different output formats:

1. Pretty JSON: Human-readable format with indentation and line breaks—easy to scan but heavy on whitespace.

2. Minified JSON: Strips all unnecessary whitespace from JSON, producing a compact but less readable output optimized for machine processing.

3. TOON (Token-Oriented Object Notation): Format designed specifically for LLM consumption, transforming JSON into a tabular notation that aims to reduce token count while maintaining readability. According to its documentation, TOON uses roughly 40% fewer tokens than JSON while achieving 74% accuracy compared to JSON’s 70%.

Pretty JSON:

{
  "context": {
    "task": "Our favorite hikes together",
    "location": "Boulder",
    "season": "spring_2025"
  },
  "friends": [
    "ana",
    "luis",
    "sam"
  ],
  "hikes": [
    {
      "id": 1,
      "name": "Blue Lake Trail",
      "distanceKm": 7.5,
      "elevationGain": 320
    },
    {
      "id": 2,
      "name": "Ridge Overlook",
      "distanceKm": 9.2,
      "elevationGain": 540
    }
  ]
}

Minified JSON:

{"context":{"task":"Our favorite hikes together","location":"Boulder","season":"spring_2025"},"friends":["ana","luis","sam"],"hikes":[{"id":1,"name":"Blue Lake Trail","distanceKm":7.5,"elevationGain":320},{"id":2,"name":"Ridge Overlook","distanceKm":9.2,"elevationGain":540}]}

TOON:

context:
  task: Our favorite hikes together
  location: Boulder
  season: spring_2025
friends[3]: ana,luis,sam
hikes[2]{id,name,distanceKm,elevationGain}:
  1,Blue Lake Trail,7.5,320
  2,Ridge Overlook,9.2,540

We tested these three formats across 25 endpoints from the Jellyfish API, each with different response characteristics—some returned deeply nested structures while others returned flat lists. After converting each response into all three formats, we measured token counts using Anthropic’s tokenizer for Claude Opus 4.5. Once we measured the token count, we compared the differences to see which format performs the best.

Results

Overall Performance Comparison

Across all 25 endpoints tested, TOON consistently outperformed Pretty JSON by consuming fewer tokens, achieving a median reduction in token count of 42.69%. This result is unsurprising—Pretty JSON relies heavily on whitespace for readability, using indentation and line breaks to make nested structures human-parsable. TOON achieves similar readability through its compact notation while eliminating most of this whitespace overhead, resulting in substantial token savings across the board.

The comparison between Minified JSON and TOON reveals a more nuanced picture. For many endpoints, TOON remained the more token-efficient format. This is because TOON excels with flat, uniform arrays of objects—when every item in an array shares the same fields at the same level, TOON can collapse the structure into a compact tabular format, declaring field names once as a header and representing each object as a simple CSV-like row. This eliminates the repeated key names that inflate JSON.

Performance by Endpoint Category

Interestingly, the 11 allocation endpoints told a different story: Minified JSON was consistently more token-efficient than TOON for these 11 endpoints. Upon investigating why these endpoints behaved differently, we found that their responses share specific structural characteristics: nested objects within repeated entries and dictionaries that use dynamic keys as data values. When array entries contain nested objects or sub-arrays, TOON cannot flatten them into tabular rows and falls back to YAML-like indentation, which adds whitespace overhead. Similarly, when dictionaries use dynamic keys as values rather than a consistent schema, there’s no repeating structure for TOON to factor out. In these cases, Minified JSON’s approach of stripping all whitespace produces a more compact result.

MCP Implementation

Based on our findings, we decided to implement TOON as the standard output format for the Jellyfish MCP’s Node server. Although Minified JSON proved more efficient for some nested endpoints like allocations, TOON provided a general reduction in context bloat. This efficiency is particularly valuable since there is an option to send the output to both Hugging Face (for Prompt Guard) and the LLM host, meaning the token savings result in a double reduction in processing overhead. Beyond context management, there is a secondary benefit: human readability. Unlike dense JSON blocks, TOON’s tabular output makes it significantly easier for users to quickly scan and verify the raw data returned by the MCP.

Practical Application

We wanted to move beyond theoretical benchmarks and test if these token savings translate directly to usability in a real-world workflow. To do this, we conducted a functional test in Claude Desktop using the Claude Opus 4.5 model. We paired the Jellyfish MCP with a Jellyfish Visualization Skill - a custom Claude Skill we created that allows the LLM to generate dashboard-style visualizations using the Jellyfish style. It is important to note that the Visualization Skill is not optimized for context, but it was used primarily to test the real-world application of the Jellyfish MCP.

We tasked the model with two sequential prompts in a single thread, comparing the total token output of the tool calls in Minified JSON versus TOON. Our token measurement focused strictly on the output tokens generated by the tool calls, excluding the Visualization Skill definition or conversational overhead.

When configured to return Minified JSON, Claude executed 15 distinct tool calls. After switching the output to TOON, it triggered 17 calls - 15 of which were identical to the JSON run. Despite this higher volume of requests, the TOON format achieved a 9.39% reduction in total token consumption. Consistent with our earlier findings, the allocations tools remained the only outlier where TOON consumed more context than Minified JSON. In fact, if we exclude the allocations tools from the calculation, the difference skyrockets to a 23.33% decrease in token count when using TOON.

Conclusion

Our testing demonstrated two clear findings:

1. TOON was the most effective format for our MCP, providing the best balance of readability and context efficiency across the majority of our tools.

2. Minified JSON was most effective in cases where the data structure was nested and dynamic, such as the allocations endpoint.

Based on our findings, our next set of updates for the Jellyfish MCP’s Node server will prioritize the reduction of context bloat, establishing TOON as the standard. These changes, along with several housekeeping updates and the new DevEx API endpoint, are scheduled to be released to the repository shortly, providing teams with deeper visibility into their developer experience metrics.

Looking Ahead

Looking at the broader landscape, this work is just one piece of the puzzle. As we continue to refine how we feed data to LLMs, we are keeping an eye on emerging solutions that streamline this process further. Platforms like Tessl are pioneering “Spec-Driven Development,” which creates structured, AI-ready context by default. Similarly, frameworks like LangChain are introducing middleware for agents, allowing developers to programmatically filter and compress context before it ever reaches the model.

Ultimately, our work with TOON and Minified JSON is about maximizing the value of every token. But efficient formatting is only the immediate hurdle. The more space we save on raw data, the more capacity we create for Memory Engineering. While context engineering optimizes the current session, memory engineering focuses on how agents persist information, learn from past interactions, and evolve over time. By reducing the bloat of today’s context, we are effectively clearing the stage for the persistent, evolving memory of tomorrow.

We analyzed more than a hundred companies and thousands of users and discovered some interesting patterns. We defined adoption in terms of usage percentage—the proportion of workdays on which an AI coding tool was used—and categorized users into three buckets: Low (0–20%), Moderate (21–80%), and High (81–100%) usage.

Our analysis revealed that we don’t live in a world of Luddites vs lovers in the AI age: rather, there is a middle ground of adoption. Some of our findings show:

1. Partial Polarization:

   1. Usage distribution shows a right-skewed pattern, with 31.7% of users in the low usage bucket (0-20%); and only a small secondary peak at high usage.

   2. Our measure of polarization - the Polarization Index - remains stable over time, with only a slight decrease over the past few months.

2. Tool-Specific Differences: GitHub Copilot shows the highest Polarization Index (1.28), while Amazon Q shows the lowest (0.48), indicating tool choice significantly influences adoption patterns.

3. Middle Segment Holding: The moderate usage group (21-80%) remains stable at ~40% of users over the past months, showing no evidence of collapse.

Rather than polarization, AI usage follows a graduated adoption curve where most users fall into low-to-moderate usage, with a long tail of power users. This suggests AI tools are widely tried, but deeply adopted by only a subset of users.

Usage Distribution: Skewed Toward Low Usage, But With a Middle Ground

Distribution Histogram

The distribution is weakly bimodal — we see clustering at the low end (a clear peak at 1-10% usage), but no corresponding peak at high usage, just a minor mode (at 61-70% usage) within a gradual tail. The median user uses AI tools 28.4% of available workdays, suggesting moderate adoption is typical.

Polarization Index Remains Stable, Slight Decrease Over Time

Polarization Trend

We define a Polarization Index as: (Low Usage + High Usage) / Moderate Usage. For the past 6 months our data shows that polarization of usage has been going down, and that users are increasingly open to adopting AI — the slight negative slope suggests more users are moving into moderate usage ranges.

Significant Tool-Specific Polarization Differences

Tool Comparison

A different story emerges when we analyze polarization at the tool level. GitHub Copilot contrasts significantly with Amazon Q, suggesting that each tool drives different usage patterns. Breaking down each tool by usage bucket reveals distinct adoption profiles.

Tool Bucket Comparison

The varying levels of tool usage suggest that the specific use case and the characteristics of each tool are independently influencing how much they are adopted.

• GitHub Copilot (most mature tool) shows highest polarization, with nearly half of users in the low usage range

• Cursor shows a more balanced distribution, with the majority in moderate usage range

• Claude Code has the highest proportion of power users, with 28%

• Amazon Q shows most uniform distribution across usage ranges

Mature tools (GitHub Copilot) have had more time to accumulate inactive/low-usage licenses, while newer tools (Claude Code) are attracting more committed early adopters, creating tool-specific polarization patterns.

Distribution Evolution Shows Stability, Not Polarization

Monthly Snapshots

The moderate and high usage segments are growing faster than the low usage segment, contradicting polarization toward extremes. This suggests that new users are entering at moderate usage levels, some users are graduating from occasional to frequent usage, and the user base is expanding rather than bifurcating.

Conclusion

What we’re seeing is a lopsided adoption curve: most engineers are dipping their toes in (with low to moderate use), while a smaller group is fully committed. The encouraging news is that this divide is leveling out, with more users moving into the middle ground over time.

Tool choice matters significantly. People may stick with what just plain works, but newer tools tend to attract committed early adopters with more frequent usage patterns. This has practical implications for engineering leaders:

• If you’re piloting newer tools, know that early adopters tend to go all-in — this is a huge opportunity to use their enthusiasm to build internal champions.

• The middle-usage segment is your conversion opportunity. Focus on understanding what would move occasional users to regular usage. The path to AI-powered engineering isn’t about forcing adoption — it’s about creating the conditions where moderate users naturally become power users.

Methodology

• Usage percentage: Unique days a user accessed their AI tool ÷ available workdays (Monday–Friday, from seat activation to analysis end date)

• Usage buckets: Low (0–20%), Moderate (21–80%), High (81–100%)

• Polarization Index: (Low Users + High Users) ÷ Moderate Users — values >1.5 suggest strong polarization toward extremes

Cross-posted at jellyfish.co

Here at Jellyfish we recently teamed up with OpenAI to study how AI coding tools are affecting developer productivity. We found that adoption of AI tools correlated with significant increases in productivity, particularly PR throughput. In fact, achieving full adoption of AI coding tools (i.e. every engineer using AI every time they code) corresponded to a 2.1x change in PRs merged.

However, we all know that software engineering teams are complex entities, each with their own unique constraints, strengths, and challenges. Perhaps you’re seeing exciting gains… or perhaps you’re finding the impact of your AI investments underwhelming. In either case, there are many factors beyond adoption that affect PR throughput. Let’s talk about the most important: your code architecture.

What Do We Mean By “Code Architecture”?

By code architecture, we mean the strategy by which your products and services are organized and coordinated across your source code repositories. Think monorepo vs. polyrepo, monolith vs. microservices, or a centralized vs. federated product/platform strategy.

One simple metric for understanding your code architecture is weekly active repositories: how many distinct repositories did you merge code into this week? However, this metric grows linearly with the number of engineers – larger companies tend to have more repositories, representing more products and services as they scale. Alternatively, here at Jellyfish we’ve developed a new metric: weekly active repositories per engineer. This metric is more useful in practice, characterizing your organization’s repo strategy independent of scale. A low number of active repos per engineer indicates a more consolidated architecture (monorepos, monolithic services, and/or centralized products) while a high number indicates a more distributed architecture (polyrepos, microservices, and/or federated products).

To understand different code architectures and how they affect AI productivity gains, we investigated data from 321 Jellyfish customers from January to August 2025, representing more than 3.8 million pull requests across 130,000 code repositories.

The chart below shows the distribution of weekly active repos per engineer over the data set (9,602 total “company-weeks”, i.e. snapshots of each company for the various weeks over the observation period).

This chart illustrates the wide range of distinct code architectures that companies adopt. We’ve broken the distribution down into quartiles, labeling each with the code architecture that most characterizes it. The table below summarizes the different regimes and the types of architecture strategies that give rise to each.

How Code Architecture Affects AI Impact

As in previous analyses, we specify a company-wide AI adoption rate that measures how regularly engineers in an organization are using AI coding tools. It is defined as the fraction of active coding days an engineer used AI, averaged across all engineers on the team.

When we segment the data by code architecture, we find that coding behaviors and the effects of AI adoption vary widely across the cohorts. The chart below contains four scatter plots, where each data point is a snapshot of a single company and week, indicating the PRs merged per engineer and AI adoption rate for that company-week. Each scatter plot corresponds to a different code architecture (quartile) – centralized, balanced, distributed, and highly distributed. We also fit trend lines to measure the relationship between change in AI adoption rate and changes in PR throughput.

The first thing to notice is that the average volume of PRs varies across the code architectures, increasing as code becomes less centralized. Independent of AI coding practices, teams with highly distributed architectures merge approximately 2x the PRs per engineer compared to those with centralized architectures (5.3 vs. 2.7 PRs per engineer on average).

Note that this difference does not suggest a disparity in overall productivity. Rather, the higher PR volume associated with more distributed architectures is primarily due to the higher level of cross-repo coordination required. In a distributed architecture, shipping a single feature typically requires changes across many services, and a single upgrade or code migration may have to be duplicated across multiple repositories, resulting in a higher number of pull requests on average.

Most interestingly, the correlation between AI adoption and PR throughput also varies across code architectures. Teams with centralized and balanced architectures can expect a ~4x change in PRs per engineer (approximately double what we observed in our previous study) when going from 0% to 100% adoption of AI coding tools. Teams with distributed architectures can expect a ~2x change – about half of what we observe for centralized and balanced architectures.

For the top quartile the relationship between AI adoption and throughput is the noisiest, and the correlation indicates a slight negative trend, indicating that adoption of AI tools may actually be slowing down teams with highly distributed code architectures. We summarize all of the results in the table below.

The obvious question is, what is causing these disparities? One explanation is the importance of context to the effectiveness of AI coding tools. In the past year we have seen huge advances in the abilities of LLMs to perform indexing and search as well as tool use, and in many ways today’s AI tools are limited only by the quality of the context provided to them. This may be why code architecture matters: the more centralized the code, the easier it is for AI coding tools to find the context they need to complete work accurately, effectively, and with all relevant changes included. By contrast, highly distributed systems likely rely on human experts who understand the architecture and can manage the complex cross-repo coordination required to push code effectively.

Does this mean that organizations with highly distributed architectures are doomed? Not at all: context engineering and AI agents are evolving fast, and these organizations may have a lot to gain through better automation of coding work that spans repositories, services, and products, especially if it is highly redundant and can be easily validated. Coding, testing, and review agents that can coordinate work across repositories could yield huge gains for these kinds of architectures.

For now though, the data is clear: your code architecture is a critical (and perhaps unexpected) factor in the return on your AI investments. As you scale your use of these powerful new tools, take a hard look at your code architecture – a more consolidated codebase may be the key to unlocking the promise of bigger AI productivity gains.

Cross-posted at jellyfish.co

Last month we shared some exciting work we did in collaboration with folks at OpenAI investigating the impact of AI coding tools across millions of pull requests for more than 500 companies.

Among other things, we found that full adoption of AI coding tools correlated with a 113% increase in PR throughput. (Check out OpenAI’s take on the analysis here and our full post here.)

AI-assisted PRs are getting bigger

As we continue to dig deeper into the data, one interesting discovery is that in addition to increasing PR throughput and improving cycle times, teams with high levels of AI adoption also are pushing larger PRs.

As before, we defined a company-wide “AI Adoption Rate” which reflects how regularly engineers on a team are using AI tools (defined as the fraction of active coding days an engineer used AI, averaged across all engineers on the team). For the plot below, each data point is a snapshot of the average additions per PR and the level of AI adoption for a given company and week.

What we see is that going from 0% to 100% AI adoption corresponds to an increase from 74.8 additions per PR to 88.4 additions per PR – an 18.2% increase. In other words, as companies increase AI adoption, their product features and platform enhancements tend to involve more lines of code than before.

Why are PRs getting bigger?

Many factors could be driving this change. To look a little deeper, let’s compare some metrics for the top quartile of companies in the plot above (75–100% AI adoption) versus the bottom quartile (0–25% AI adoption).

Looking at these metrics, we can see that most of the change appears to be from net new code, not rewritten code. Companies in the top quartile are only pushing 4.1% more deletions on average, versus 14.5% more net additions. Interestingly, we don’t see a big increase in the number of files per PR, suggesting AI is generating more code within the same files, not necessarily finding more changes to make across files.

If we take a look across file types, we see that companies in the top quartile are using certain languages and file formats at a much higher rate (up to 76% more), particularly TypeScript, Python, YAML, and Markdown. This trend aligns with the fact that LLMs generally perform better when working with these highly structured, well-documented programming languages and supporting configuration and documentation formats. The AI-assisted code that high-adoption companies are writing in these formats tends to be more explicit, with more comments and other conditional code (e.g. exception handling) than might otherwise be written without it.

Is bigger code better?

Here at Jellyfish we’ve found that an increase in PR throughput is typically a good thing, corresponding to more value delivered to customers. But what about bigger code? That depends. It could mean code that is more robust and well-commented, but it could also mean code that is overly complex and harder to debug, iterate on, and maintain. As AI technology continues to evolve, we’ll keep looking at how AI coding assistants and agents are impacting code generation – both quantity and quality.

Cross-posted at jellyfish.co

When it comes to determining the impact of AI coding tools, reliable data has been few and far between, and published results can tell conflicting stories. For instance, a recent study from METR found that in some cases and under some conditions, AI tools can actually slow developers down, while an earlier study from Microsoft showed significant gains.

At Jellyfish, we’ve been collecting a robust set of AI impact data across more than 500 companies, representing the coding activities of tens of thousands of engineers, comprising millions of pull requests and billions of lines of code. Real-world data at this scale can help us understand what’s really going on out in the field as companies engage in the journey of AI transformation.

We recently teamed up with the folks at OpenAI to take a closer look at how AI tools are affecting coding productivity and quality. Here’s what we found:

Increasing AI adoption means shipping more code, and faster

We defined a company-wide “AI Adoption Rate” which reflects how regularly engineers on a team are using AI tools. It is defined as the fraction of active coding days an engineer used AI, averaged across all engineers on the team. In the following plots, each data point is a snapshot of a given performance metric and a company’s level of AI adoption for a single week, allowing us to see trends as companies achieve higher levels of AI adoption.

What we see is that in terms of PR throughput, going from 0% AI adoption to 100% adoption corresponds to an increase from 1.36 PRs per engineer on average to 2.90 PRs – a 113% increase.

Increasing AI adoption also correlates with pushing code to production faster. In going from 0% to 100% AI adoption, an average company can expect their median cycle time to drop four hours from 16.7 to 12.7 – a 24% reduction.

But what about code quality?

One of the things we looked at was the fraction of total PRs that are linked to bug tickets. We see that the highest levels of AI adoption correspond to an increase of 26.8% in the proportion of tickets (9.5% versus a baseline of 7.5%). In other words, companies with high AI adoption are pushing more code, and a higher fraction of those PRs are bug fixes.

It may be the case that higher AI use is causing more bugs, or it may be helping teams fix more bugs – or it may be a combination of both. One interesting observation is that we aren’t seeing a comparable increase in revert PRs (pull requests whose primary purpose is to undo previous changes, typically due to a critical failure), suggesting that AI doesn’t seem to be causing major quality issues across the board.

AI tools still need human judgment to deliver quality at speed

While errors are few and far between, they still diminish the time savings and productivity gains from AI and show just how vital human skill and oversight remain in the AI era. As AI models improve and training and usage increase, we expect quality and speed to further improve.