Benchmarking Frameworks for Coding Agents: A Comprehensive Review of Current Practices, Challenges, and Future Trends

Introduction: Foundational Concepts and Significance of Benchmarking for Coding Agents

AI coding agents represent a pivotal evolution in the software development landscape, transitioning from conventional, static code generation tools to interactive, iterative, and tool-augmented workflows 1. These systems, powered by Large Language Models (LLMs), are engineered to autonomously plan, execute, and refine complex software development tasks 1. Their fundamental purpose is to streamline and elevate the coding process by generating, optimizing, and even repairing code with remarkable speed and accuracy, thereby allowing human developers to focus on higher-level challenges 2. Given their growing autonomy and integration into critical development pipelines, benchmarking these coding agents is indispensable for fostering reliable, efficient, and trustworthy AI systems capable of independently executing software tasks .

The comprehensive evaluation of AI coding agents is paramount for several strategic objectives. Firstly, it ensures their reliability and robustness, guaranteeing consistent performance across diverse scenarios and mitigating errors or hallucinations in generated code . Secondly, rigorous evaluation is vital for identifying failure modes and potential misalignments, uncovering unexpected behaviors, security vulnerabilities, or deviations from business objectives that autonomous agents might exhibit . Thirdly, benchmarking provides the quantitative feedback necessary for continuous iteration and improvement, enabling teams to objectively compare different models, prompt strategies, and architectural decisions 3. Furthermore, it aids in resource management by identifying inefficiencies and optimizing the significant computational resources consumed by agent operations 3. Crucially, as these agents assume more critical roles, comprehensive evaluation is essential for ensuring safety and trustworthiness, guaranteeing that they are not only accurate and helpful but also adhere to ethical guardrails and mitigate risks like unpredictable errors or context drift .

The methodologies for evaluating code generation and related AI systems have evolved considerably, reflecting the advancements in the field. Early efforts in program synthesis from the 1960s to the 1980s focused on generating provably correct programs from formal specifications, with evaluation primarily verifying correctness against these specifications 1. The 1990s saw the emergence of code completion tools, which enhanced developer productivity by predicting code snippets based on context, with evaluation centered on the accuracy of suggestions and productivity gains 1. The advent of pre-trained Large Language Models (LLMs) in the 2010s marked a significant shift, enabling robust few-shot and zero-shot capabilities in code generation and translation, with evaluation then focusing on the quality and correctness of generated code snippets 1. The current era of AI agentic programming leverages LLMs as autonomous entities capable of multi-step reasoning, tool interaction, and iterative refinement 1. This paradigm necessitates a departure from traditional testing, as these agents introduce non-determinism, operate across complex workflows, and maintain context, requiring new evaluation approaches that assess dynamic behavior, reasoning processes, and the entire trajectory of actions, rather than just final outputs . This evolution underpins the sophisticated benchmarking frameworks being developed today to critically assess the capabilities of these advanced coding agents.

Existing Benchmarking Frameworks and Methodologies for Coding Agents

The burgeoning field of AI coding agents, which autonomously plan, execute, and refine software development tasks 1, necessitates robust evaluation beyond traditional code generation metrics. Unlike static LLM evaluation or conventional software testing, AI agents present unique evaluation challenges due to their non-determinism, multi-step workflows, external tool interactions, and context retention 4. Consequently, several specialized academic and industry-standard benchmarking frameworks have emerged to rigorously assess their capabilities.

These frameworks are designed with specific principles, target tasks, evaluation environments, and datasets to provide comprehensive insights into agent performance. The following table summarizes prominent frameworks:

Beyond these core coding agent benchmarks, several other frameworks address related capabilities crucial for comprehensive agent evaluation 5. For general AI assistant capabilities, often integrated into coding agents, benchmarks like GAIA, OSWorld, AssistantBench, BrowseComp, and BFCL assess reasoning, multi-modality, web browsing, tool-use, and function calling 5. Cybersecurity-specific evaluations utilize frameworks such as CVEBench, Cybench, CyberMetric, CyberSecEval, InterCode, GDM Dangerous Capabilities, SEvenLLM, SecQA, and 3CB to gauge an agent's ability to identify or exploit vulnerabilities, perform cybersecurity tasks, or respond to incidents 5. Finally, AgentHarm and Mind2Web-SC are designed to evaluate the potential for harmfulness and the effectiveness of safety guardrails in AI agents 5. These diverse frameworks collectively contribute to a multifaceted understanding of coding agent performance across various domains and operational contexts.

Key Performance Indicators and Evaluation Metrics for Coding Agents

Evaluating coding agent performance necessitates a multi-layered approach that assesses various aspects, from the quality of the model's output to its broader application-level outcomes and operational efficiency 3. This comprehensive evaluation combines quantitative and qualitative metrics to ensure reliability, identify failure modes, enable iterative improvement, and manage resources effectively 3. Benchmarking frameworks often leverage a combination of these indicators to provide a holistic view of an agent's capabilities.

Key Performance Indicators (KPIs) and Metrics

1. Exact Match and Semantic Correctness

   • Exact Match: This metric quantifies how precisely an agent's outputs align with expected, pre-defined results 3. It is particularly crucial for tasks where a single correct answer or output format is required.
   • Semantic Correctness: Beyond mere syntactic agreement, semantic correctness assesses the factual accuracy of generated code, explanations, and other responses, ensuring they align with ground truth and domain-specific requirements 3. This also encompasses Faithfulness, which verifies that agent responses are well-grounded in retrieved context and avoid generating hallucinatory information 3. Semantic Similarity further refines this by comparing agent outputs to reference responses using embeddings to evaluate meaning-based alignment, allowing for functionally equivalent yet structurally different solutions 3. Benchmarks like HumanEval and MBPP, by requiring correct Python functions or programs from natural language, implicitly evaluate both exact match (for test cases) and semantic correctness.

2. Pass@k Pass@k is a common metric in code generation benchmarks. It involves generating k independent code samples for a given problem; if at least one of these samples successfully passes all associated unit tests, the problem is considered solved 6. While not always explicitly named "Pass@k" in the provided text, the concept of successfully passing tests after code generation is fundamental to many coding benchmarks, including SWE-Bench Pro, which uses a "Resolve Rate" requiring issues to be resolved (pass-to-fail tests turn into pass-to-pass tests) without introducing regressions 7. This metric acknowledges the probabilistic nature of LLM outputs and provides a more robust assessment of an agent's capability to solve a problem.

3. Code Efficiency Code efficiency metrics focus on the operational performance and resource consumption of coding agents.

   • Efficiency and Step Count: Measures how efficiently an agent achieves its objectives by minimizing unnecessary steps and avoiding unproductive loops 4. Step Utility is closely related, ensuring that each action taken by the agent meaningfully contributes to task completion 3.
   • Latency: The total time taken from submitting a query to receiving the final response, including model inference time, tool execution, and data retrieval 3.
   • Cost: Reflects the computational resources consumed, such as the number of tokens used per interaction, API calls to LLMs, and underlying infrastructure expenses 3. These metrics are crucial for deploying agents in real-world scenarios where operational costs and speed are significant considerations.

4. Readability Clarity and Conciseness evaluate whether an agent's responses and generated code are clear, easy to understand, well-structured, use appropriate language, and are appropriately brief without unnecessary verbosity 3. For coding agents, readable code is essential for maintainability and collaboration among human developers.

5. Security Vulnerabilities As coding agents become more integrated into critical systems, their security implications must be rigorously evaluated.

   • Robustness to Adversarial Inputs: Measures an agent's resilience against malicious prompts, data poisoning, or injection attacks that could lead to unintended or harmful behaviors 3.
   • PII Detection: Validates that agents do not inadvertently expose sensitive Personally Identifiable Information, ensuring compliance with privacy regulations 3.
   • Vulnerability Detection/Introduction: This assesses an agent's ability to identify existing security vulnerabilities within code or to prevent the introduction of new ones during code generation or modification. Benchmarks like CVEBench specifically evaluate an agent's capability to interact with and resolve real-world web application vulnerabilities 5.

6. Test Coverage For agents tasked with generating or modifying code, test coverage measures the extent to which the generated or changed code is covered by tests 3. Automated testing is an integral component of modern software development, and a capable coding agent should either generate well-tested code or be able to generate tests for its code.

7. Human Evaluation Human evaluation is indispensable for ground truth assessment, particularly for complex, nuanced, or safety-critical tasks that automated metrics cannot fully capture . Human experts provide domain-specific correctness validation, assess subtle quality attributes, make judgments on safety and appropriateness, and help identify unforeseen edge cases 3. Frameworks like SWE-Bench Pro and SWE-bench Verified extensively leverage human review to annotate and verify benchmark tasks, ensuring clarity, fairness, and the real-world applicability of the evaluations . This qualitative assessment often guides the interpretation and weighting of quantitative metrics.

Other Important Metrics and Evaluation Strategies

Beyond these core KPIs, several other metrics and evaluation strategies contribute to a comprehensive understanding of coding agent performance:

• Task Success Rate: The most fundamental metric, often a binary assessment of whether an agent successfully completes its assigned task, sometimes including graded partial completion . In SWE-Bench Pro, this is termed "Resolve Rate" and requires both issue resolution and no regressions 7.
• Agent Trajectory Quality: Evaluates the logical reasoning paths, intermediate decisions, and overall efficiency of the approach taken by the agent through multi-step tasks 3.
• Tool Selection Accuracy: Assesses whether agents correctly identify and invoke the most relevant external tools with appropriate parameters for a given task 3.
• User Satisfaction: Measures the end-user's perception of the agent's performance through direct feedback (e.g., ratings, surveys) and implicit signals (e.g., conversation continuation) .
• Consistency: Evaluates whether agents provide similar, stable answers for similar queries over time and across different invocations 3.
• Adaptability: Measures an agent's ability to adjust to new scenarios, generalize beyond its training distribution, and perform effectively on novel tasks 3.

Evaluation strategies involve Automated Evaluation using statistical (BLEU, ROUGE) and programmatic (rule-based checks) tools 3; LLM-as-Judge Evaluation, employing other language models to assess subjective qualities ; Simulation-Based Evaluation for testing across synthetic scenarios 3; and Online Evaluation for continuous monitoring of production agents 3. These strategies provide the mechanisms through which the aforementioned metrics are collected, analyzed, and used to inform improvements and ensure that AI coding agents meet the stringent demands of modern software development.

Challenges and Limitations of Current Benchmarking

Benchmarking coding agents presents numerous inherent difficulties, unresolved issues, and shortcomings within current frameworks. These challenges stem from the complex nature of software development, the rapid evolution of large language models (LLMs), and the need to accurately measure agent performance in realistic and dynamic environments. This section details these challenges, highlighting why current evaluation methodologies are often insufficient.

Task Complexity

Early code-agent benchmarks often target isolated and static problems, such such as algorithmic tests, function-level code completion, or program repair 8. This narrow focus overlooks the broader scope of real-world developer practices, which involve navigating extensive documentation, understanding code dependencies, and dynamically generating, modifying, or debugging code 8. Many programming task benchmarks remain technical-oriented, failing to assess an agent's ability to leverage open-source repositories for solving complex, end-to-end tasks in a user-centric setting 8.

Modern agent evaluation is more complex than simple one-shot LLM calls, involving multi-step processes where agents hold context, call tools, read and write to internal databases, and must complete tasks reliably 9. Open-ended tasks, such as those in GAIA, may require an arbitrarily long sequence of actions and multimodal understanding 10. Benchmarks like MINT evaluate interactive tasks where models must use external tools, respond to feedback, and adjust their approach over multiple turns, testing resilience and self-correction 10. Current agents frequently struggle with complex workflows, especially multimodal tasks involving model-based processing, dependency installation, weight downloading, and runtime configuration 8. Issues include getting stuck, clicking wrong links, misunderstanding web layouts, losing track of long-term goals, or misinterpreting interface elements 10. The rapid evolution of LLMs has also outpaced benchmark development, making many datasets insufficiently challenging or comprehensive 11. Furthermore, widely used benchmarks like HumanEval and MBPP suffer from flaws such as incorrect tests, insufficient test coverage, flawed canonical solutions, and imprecise problem definitions 12. For example, HumanEval's Task 47 incorrectly states the median of a list 12. As models achieve near 100% scores on saturated benchmarks, there is a continuous need to elevate program complexity 12.

Dataset Biases

A significant limitation is the presence of biases in datasets and the lack of standardization in benchmark creation . Early benchmarks tend to feature "isolated, static problems" and focus narrowly on technical tasks, rather than replicating real-world scenarios that involve leveraging diverse open-source repositories 8. This creates a bias towards problems that are easier to simulate but less representative of actual developer challenges 8. The proliferation of benchmarks has led to fragmented knowledge across tasks and difficulties in selecting contextually relevant benchmarks 12. Flaws in foundational benchmarks can propagate biases and lead to an overestimation of technical progress 12. Data contamination and benchmark overfitting are critical concerns, where models may memorize flawed solutions, artificially inflating performance scores. An instance of this is ChatGPT-3.5 reproducing an incorrect result from HumanEval's Task 47 12. Many benchmark variants, intended to improve language support or test coverage, often build upon these original flawed datasets, thereby duplicating existing issues or generating new test cases based on incorrect canonical solutions without rigorous quality control 12.

Generalizability to Real-World Scenarios

Most early benchmarks evaluate agents in simplified or synthetic environments, failing to assess their real-world problem-solving capacity 8. Real-world applicability is crucial, as LLM-based agents often operate as black-box models, generating probabilistic solutions that can contain hallucinations, low effectiveness, security vulnerabilities, or logic errors 11. Popular benchmarks like SWE-Bench, WebArena, and AgentBench often run agents in contained environments with public tools, but they typically avoid the complex interactions with internal databases and dynamic user interactions that are common in business workflows 9. This limitation means they primarily test tool-use mechanics rather than an agent's ability to complete real business tasks under realistic constraints 9. There is a lack of alignment with real-world scenarios and a pressing need for more comprehensive and realistic datasets that include interactive and multi-modal contexts 11. The importance of managing dependencies, handling unforeseen errors, and understanding complex build processes, as demonstrated in multimodal tasks like image processing, remains a major hurdle for current agents 8.

Computational Cost

The computational cost associated with benchmarking coding agents is a significant practical consideration 8. Evaluating cost-efficiency is crucial because replacing human labor with agents is not always economically viable 8. Agents incur tangible operational costs, such as API fees for proprietary LLMs or hardware expenses for open-source solutions 8. Benchmarking efforts must quantify potential cost savings and efficiency gains to determine practical utility 8. The introduction of metrics like the "alpha value" attempts to integrate task completion quality, agent token usage, and market-rate human labor costs into a unified framework for assessing economic benefits 8.

Experiments show that performance can vary greatly between different framework-LLM pairings, affecting both effectiveness and efficiency. For example, OpenHands with Claude 3.7 may offer the best performance but at a higher cost, while GPT-4.1 can be more cost-efficient for similar performance 8. Fine-tuning parameters like timeout and max_iteration can boost performance but also incur higher token usage and costs, underscoring the trade-off between effectiveness and computational efficiency 8. Even for low-cost tasks, agent operational costs can quickly lead to negative returns, highlighting the need for careful cost control in commercial applications 8.

Dynamic Code Environments

A major challenge lies in evaluating agents within dynamic code environments 8. Traditional benchmarks often neglect agents' ability for autonomous environment setup and leveraging open-source repositories 8. Real-world tasks necessitate agents to independently manage environment provisioning, including installing dependencies (e.g., pip install -r requirements.txt) and resolving dependency issues in a sandbox 8. In GitTaskBench, environment setup errors (E1) were the most common failure type, accounting for 65.04% of all failures 8. These errors typically arise from dependency conflicts, missing binary wheels, or absent system-level libraries, demonstrating the critical and unavoidable nature of environment management in practical agent applications 8. Agent behavior is conditional on a mutable state that exists outside the model, which evaluation systems must be able to handle 9. Stateful benchmarks that use mocked or real databases and simulate user interactions are emerging to address this, as they can validate not only the final answer but also the resulting changes in the database state 9. However, adapting these stateful benchmarks to real business workflows introduces further complexities such as managing complex, evolving data models and ensuring idempotent test runs 9.

Subjectivity of Human Assessment

While objective metrics are sought after, human assessment inevitably plays a role in establishing ground truth and evaluating the quality of agent outputs 8. Benchmarks like GitTaskBench employ human-designed, automated evaluation scripts that rely on practical success criteria 8. The 'alpha' metric incorporates a quality factor derived from human assessment, where experts compare agent outputs to human-generated ground truth and assign a score (0 to 1) 8. This process involves multiple raters independently assessing outputs against a standard, with the majority choice determining the final value 8. The market value of tasks, also used in the 'alpha' metric, is based on publicly listed freelance fees, which can vary and introduce a degree of subjectivity in economic valuation 8. Despite efforts to standardize, human interpretation of task requirements and quality can introduce variability 8. Furthermore, some benchmarks use human feedback or simulation of user interactions (e.g., in MINT), where the human element directly influences the dynamic evaluation process 10. The need to account for how a model arrives at answers, not just if it's correct (e.g., distinguishing between true reasoning and memorization), also requires qualitative human judgment or carefully designed adversarial tests 10.

In conclusion, benchmarking coding agents effectively requires addressing multifaceted challenges spanning task complexity, dataset quality, real-world applicability, computational expense, and dynamic environmental interactions. The ongoing development of benchmarks like GitTaskBench and stateful evaluation methodologies aims to move beyond isolated, static tests towards more comprehensive, realistic, and economically-aware assessments. Efforts are needed to improve dependency management, execution planning, repository comprehension, resource handling, and instruction following for more robust and reliable agent performance in real-world scenarios 8.

Latest Developments, Trends, and Research Progress (2023-2025)

Benchmarking frameworks for coding agents are undergoing rapid development and refinement between 2023 and 2025, driven by the increasing sophistication of AI agents and the urgent need for robust, reliable, and ethically sound evaluation methods . This progress addresses limitations of earlier, more static performance metrics by embracing dynamic, multi-step, and human-aligned assessments. The field is moving towards holistic evaluations that capture the complexities of real-world software development, multi-agent collaboration, specialized coding tasks, and crucial ethical considerations.

1. Emerging Benchmarking Frameworks and Environments (2023-2025)

A new wave of benchmarks has emerged to measure how well AI systems reason, act, and recover across complex workflows, encompassing both general agentic evaluation frameworks and those specialized for coding tasks 13. The landscape highlights a significant shift towards real-world, dynamic environments that challenge agents in more sophisticated ways than previous evaluations.

Key emerging benchmarking frameworks for coding agents include:

Beyond static benchmarks, dynamic and interactive evaluation environments are crucial for agentic AI 16. Benchmarks like Terminal-Bench 13, WebArena 16, OSWorld 16, and FieldWorkArena 16 test agents in realistic or simulated environments that require dynamic adaptation. Visual development platforms such as Latenode are also emerging to simplify the creation, prototyping, and scaling of AI agents by bridging complex frameworks with intuitive interfaces 17.

2. Advancements in Multi-Agent Collaboration Evaluation

As AI agents become more sophisticated, evaluating their ability to collaborate has become a significant focus . Multi-agent systems involve multiple interacting entities specializing in perception, planning, or execution, working collectively to solve complex problems 16. Recent developments include frameworks designed for orchestrating and evaluating these collaborative behaviors:

• LangGraph: An extension of LangChain, designed for stateful, multi-actor applications involving planning, reflection, and multi-agent coordination through a graph-based representation of agent interactions .
• CrewAI: Orchestrates role-playing AI agents, each with specific roles and responsibilities, to collaborate on complex tasks, mimicking human organizational structures .
• Microsoft AutoGen: An open-source framework for building advanced multi-agent systems, enabling cooperation among agents through a multi-agent conversation framework and customizable roles .
• OpenAI Swarm: A lightweight, experimental multi-agent orchestration framework that focuses on agent coordination via "Agents" and "Handoffs" 18.
• SCOOP (March 2025): A framework for proactive collaboration and social continual learning through natural language interaction and causal reasoning 19.
• ACPs (Agent Collaboration Protocols) (May 2025): Protocols specifically designed for the Internet of Agents 19.
• Meta's LOKA Protocol (April 2025): A decentralized framework for trustworthy and ethical AI agent ecosystems 19.

Despite these advancements, evaluation gaps remain, as most benchmarks currently score only final answers rather than the quality of planning, tool selection, or the collaborative processes themselves 16.

3. Benchmarks for Specialized Coding Tasks

To address the unique demands of coding, specialized benchmarks are moving beyond general language tasks to focus on specific aspects of software development:

• Secure Coding: While not explicitly a standalone benchmark, efforts like "Protected Materials for Code" flag AI-generated code matching licensed GitHub repositories, touching upon intellectual property and potentially secure coding practices 15. Terminal-Bench specifically includes security tasks as part of its evaluation 13. Cybersecurity and safety frameworks like MITRE ATLAS™ and red teaming methodologies are critical for identifying risks and vulnerabilities in agentic AI systems .
• Performance Optimization: Benchmarks are beginning to indirectly address this through efficiency metrics. For instance, Context-Bench exposes the cost-to-performance ratio of context management 13. Furthermore, performance benchmarking standards like IEEE 2937-2022 are emerging for AI server systems 19, and AI agents have demonstrated the ability to match human expertise in tasks like writing Triton kernels, delivering faster results 14.
• Domain-Specific Languages (DSLs): The need for benchmarks covering DSLs is reflected in benchmarks for polyglot codebases such as SWE-PolyBench and those for enterprise Java workflows like Spring AI Bench 13, indicating a focus on specific programming ecosystems and their unique challenges.

4. Integration of Ethical AI Considerations into Evaluation

Ethical AI, trustworthiness, transparency, and accountability are increasingly central to the evaluation of AI agents . This integration is crucial for deploying agents responsibly, addressing potential biases, and ensuring safe operation.

• Governance Frameworks:

  • NIST AI Risk Management Framework (AI RMF): Provides comprehensive guidance for organizational approaches to AI risk .
  • EU AI Act: Establishes regulatory requirements for AI systems, with obligations phased for 2025 .
  • ISO/IEC Standards: Include standards for software testing of AI systems (ISO/IEC AWI TS 29119-11), AI trustworthiness (ISO/IEC 23053), and bias in AI systems (ISO/IEC TR 24027:2021) .
  • GovTech Singapore's Agentic Risk & Capability (ARC) Framework (August 2025) and MI9: An Integrated Runtime Governance Framework for Agentic AI (2025) are specific frameworks targeting agentic systems governance 19.
  • OpenAI's Practices for Governing Agentic AI Systems (August 2024) 19.
  • Microsoft's Responsible AI Transparency Report (2025) highlights investments in tools, policies, and practices for responsible AI, anticipating agentic systems as a significant innovation area and complying with regulations like the EU AI Act 15.

• Transparency and Explainability: Initiatives like IEEE P2976 (Standard for XAI) and IEEE 7001-2021 (Transparency of Autonomous Systems) aim to improve clarity and understanding of AI decisions 19. Policy Frameworks for Transparent Chain-of-Thought Reasoning in LLMs are also emerging to enhance explainability 19.

• Fairness and Bias Prevention: Continuous data auditing, algorithm testing across demographics, diverse teams, and ongoing monitoring are emphasized to prevent bias 20. While LLMs are trained to be unbiased, implicit biases, such as racial or gender biases, can unfortunately persist 14.

• Safety and Reliability: Rigorous testing, continuous monitoring, and human-in-the-loop controls are crucial for ensuring the safety and reliability of AI agents 20. New benchmarks like HELM Safety and AIR-Bench specifically assess factuality and safety 14. Red teaming operations are critical for identifying vulnerabilities in AI systems, including agentic AI, by simulating adversarial attacks . Microsoft has also expanded its measurement pipeline to detect protected materials (including code) and harmful content across modalities 15.

Despite these advancements, standardized responsible AI evaluations remain rare 14. Furthermore, the complexity of algorithmic prompts and potential for high costs due to token consumption in multi-agent workflows present ongoing challenges 18.

5. Forward-Looking Trends

The benchmarking landscape for coding agents is rapidly evolving, converging on several key trends:

• Holistic Evaluation: A significant shift is occurring towards lifecycle-aware assessment, integrating multi-step reasoning, robustness, and ethical alignment into unified evaluation paradigms, rather than solely scoring final answers 16.
• Real-World Complexity: Benchmarks are increasingly designed to mirror the messy, multi-step workflows faced by real developers, requiring agents that can consistently reason and act in complex, dynamic environments 13.
• Autonomous Capabilities: There is a rising demand for agents that can optimize their performance and adapt to changing requirements without constant human intervention, although fully autonomous complex reasoning remains a challenge .
• Integration of Governance: Continuous oversight, detailed action logs, clear decision rationales, and verifiable data lineage will become standard requirements for auditing agent behavior and ensuring accountability 16.
• Hybrid Approaches: Organizations will continue to adopt a mix of open-source and closed-source tools for flexibility, transparency, cost-efficiency, and data privacy, leveraging open-source options for greater control over models and code 21.
• Improved Tooling for Responsible AI: Increased investment in tools for risk measurement, mitigation, and compliance, particularly for agentic systems, will continue in 2025 and beyond 15.