Agents Are Uneven by Language, Domain, and Framework; Scope to Their Strengths

A coding agent is good at some languages, domains, and frameworks and measurably worse at others, and the variation is large enough that "can the agent do this" has no single answer. The cleanest way to see it is to hold the model fixed and move the task: Claude 3.7 Sonnet resolves 63% of issues on SWE-bench Verified, which is Python, and 43% on SWE-bench Multilingual. Scoping work follows directly: trust and autonomy should track where the model is strong.

Competence tracks training data, and Python is the high-water mark

The unevenness by language is the best-measured axis. On SWE-bench Multilingual (SWE-agent with Claude 3.7 Sonnet), per-language resolution ranges from Rust at 58% and Java at 53% down to JavaScript and TypeScript around 35% and C/C++ at 29%, against the 63% Python baseline. Multi-SWE-bench (ByteDance) reports the same direction more starkly under a different scaffold, with the best configuration resolving 52% of Python issues but only 5% of JavaScript and 2% of TypeScript, and states the finding plainly: models "demonstrate strong performance in resolving Python issues but struggle to generalize effectively across other languages." McEval, spanning 40 languages, finds models do better on high-resource object-oriented languages and worse on low-resource functional and procedural ones.

Cite the direction with confidence and the exact deltas only with the scaffold named, because magnitude is scaffold-dependent. The pattern underneath is corpus frequency, which is why TypeScript can rank "popular" and still resolve poorly.

The Python ceiling collapses on long-horizon enterprise tasks

High benchmark scores on Python come from short, well-isolated GitHub issues, and they do not survive contact with large enterprise tasks. On SWE-bench Pro, built from longer-horizon, enterprise-grade problems, top models resolve roughly 23%, against the 70%-plus they reach on SWE-bench Verified. The same gap appears in the field: METR's randomized trial found experienced developers 19% slower on their own mature repositories.

Frontend means visual correctness, and that is the weakest axis

On SWE-bench Multimodal, built from 17 JavaScript interface libraries, the top system resolves only 12% of tasks, about half its text-Python rate, and the authors attribute it to "limitations in visual problem-solving." Design2Code found a strong model could produce a single passable page but lagged "in recalling visual elements and generating correct layout designs." Anthropic describes models converging "toward generic, on distribution outputs," which "creates what users call the AI slop aesthetic," and notes the model "defaults to safe choices unless explicitly encouraged otherwise." A structured design skill measurably moves this, winning 21 of 28 A/B comparisons (p=0.006), with the largest gains on smaller models, which says the visual quality is prompt-dependent rather than reliable by default.

Intercom's production data shows AI-authored frontend code reverting at a lower rate than its backend code, which can look like "agents are good at frontend." It is not that. Revert rate measures whether a change broke production badly enough to roll back, not whether the layout is right, and it compares a whole AI-and-review pipeline against a human one, not raw authoring skill.

Legacy languages are a modernization problem, not a translation problem

For COBOL, Fortran, and mainframe code, the weakness is structural. IBM Research's own quality evaluation of COBOL-to-Java transformation documents hallucinated output, errors in CICS, SQL, and IMS middleware parameters, and the blunt arithmetic that at 90% per-paragraph accuracy a 10-paragraph program has only a 34% chance of being fully correct, against an estimated 200 billion lines of COBOL still running. IBM's only quantified watsonx Code Assistant productivity figures, 85% acceptance and 20% to 45% gains, are for its Ansible product, not its Z/COBOL product, which ships no quantified accuracy claim. When Anthropic claimed Claude Code could rapidly refactor COBOL, IBM's Rob Thomas answered that "translation captures almost none of the actual complexity," because the hard part is the batch throughput, JCL orchestration, and integrations the language sits inside, not the syntax. Treat legacy modernization as a domain where the agent assists a heavily verified human process, not one it owns.

Framework knowledge is stale by default

Even in well-supported languages, the model's framework knowledge ages out at the training cutoff. The clearest acknowledgement comes from a framework author: the AGENTS.md that Next.js now generates tells the agent, verbatim, "Your training data is outdated, the docs are the source of truth," and Next.js ships version-matched documentation in the package and points agents at it. The general failure is models emitting deprecated patterns or non-existent calls for fast-moving libraries. The mitigation is to put current docs in the agent's context, either bundled like Next.js does or through a documentation MCP server such as Context7 that "injects live, version-specific documentation." The instruction file is where that pointer belongs.

Scope and trust by domain, and let verification expand what is delegable

Intercom carves production infrastructure out of its agent's trusted envelope on the explicit grounds that its largest outages were infrastructure and third-party issues that "PR review was never going to catch," and the market is segmenting the same way, with Gartner noting specialized agents for modernizing Java, .NET, and COBOL rather than one agent assumed equally capable.

The lever that widens the trusted set is verification. A frontend change is delegable once a browser or screenshot check can confirm it, a migration is delegable because its output is mechanically checkable, and an AI reviewer can set thresholds per language and per category instead of one bar for everything.

The strengths are real and worth using at full autonomy: test generation is reliable, with Meta's TestGen-LLM reporting 73% of recommended tests accepted into production, completion and small assists are mature, and well-scoped changes in a familiar stack revert far less than human work. The 2025 DORA finding that AI still correlates negatively with delivery stability is the downstream cost of trusting a weak surface too far.