ISO/IEC TS 25058:2024 AI System Quality Assessment Management System

Key guidelines for quality assessment of artificial intelligence systems

As a key technical specification in the field of AI, ISO/IEC TS 25058 fills the gap in AI feature evaluation in traditional software quality standards. It combines the complexity of AI systems, such as dynamic learning and data dependency, and proposes targeted evaluation indicators and methods to help organizations improve the reliability, security, and compliance of AI systems.
The standard revolves around the core quality characteristics of AI systems, constructing six evaluation dimensions and refining them into actionable indicators and methods:
1. Functional applicability
Evaluate whether the AI system meets the predetermined functional requirements, including task completion, algorithm accuracy, and scenario adaptability. For example, medical diagnostic systems need to verify the coverage of their diagnostic logic through multi case testing.

2. Performance efficiency
Covering indicators such as response time, resource utilization, and scalability. The standard recommends combining dynamic load testing (such as peak data processing capability) to evaluate system performance and ensure its stability in high concurrency scenarios.

3. Security
Emphasize data privacy protection, defense against attacks, and vulnerability management. The standard requires adversarial testing of AI models (such as simulating input perturbation attacks) to verify their robustness.

4. Reliability and availability
Measure the continuous operation capability of the system through indicators such as failure rate and fault tolerance recovery ability. For example, the auto drive system needs to verify its emergency response mechanism when the sensor is abnormal.

5. Maintainability and Scalability
Focus on modular design of system architecture, version iteration compatibility, and integration capability of new features. The standard recommends using code readability analysis and module coupling testing for evaluation.

6. Documents and Transparency
Require complete development documentation, algorithm explanations, and decision logic explanations to ensure technical traceability and user understanding. For example, financial risk control systems need to disclose model feature weights and decision basis

（I.） Key application scenarios
1. Compliance audit: Meet the regulatory requirements of strong regulatory industries such as finance and healthcare (such as GDPR, HIPAA), and provide quantitative quality evidence (such as security vulnerability rates, system availability).
2. Supplier evaluation: Objectively evaluate the quality of third-party software through unified measurement standards (such as defect density, compatibility test pass rate) to reduce procurement risks.
3. DevOps integration: Embed quality metrics (such as test coverage and API response time) in the CI/CD process to achieve automated monitoring and real-time feedback.
4. User experience optimization: Quantify user behavior data (such as task completion rate, error rate), and link quality characteristics (such as usability) to drive product iteration.

（II.） Core values
1. Improve system reliability
By using quantitative indicators such as failure rate and MTBF, as well as evaluation techniques such as fault injection and reliability growth models, we can systematically reduce failure rates and minimize downtime losses.

2. Reduce compliance risks
Collaborate with standards such as ISO 9001 and IEC 62304 to meet regulatory requirements in industries such as healthcare and automotive (ISO 26262), and avoid legal disputes caused by reliability defects.

3. Optimize resource allocation
Early reliability requirement definition and testing validation (such as agile reliability engineering) can reduce rework costs and shorten product launch cycles.

4. Enhance market competitiveness
Reliability indicators, such as availability commitments, have become key factors in customer decision-making, helping enterprises establish technological trust in high security areas such as cloud computing and AI.

5. Support continuous improvement
Through on-site data analysis and the construction of a reliability culture, promote the organization to embed reliability practices throughout the entire SDLC process, and form a quality improvement loop.

（I.） Application materials
When implementing ISO/IEC TS 25058, organizations need to prepare the following materials:
1. Standard Text: Obtain the standard text of ISO/IEC TS 25058:2024.
2. Evaluation team formation: Form a cross departmental team, including AI system developers, quality management personnel, data scientists, security experts, testing engineers, etc.
3. Training materials: Prepare training materials related to ISO/IEC TS 25058 to ensure that team members are familiar with the content and evaluation methods of the standard.
4. Evaluation tools and methods: Choose appropriate evaluation tools and methods, such as functional testing, performance testing, security vulnerability scanning, etc.
5. Data collection tools: Prepare tools for collecting operational data of AI systems, including system logs, user feedback, etc.
6. Improvement Plan Template: Develop a template for improvement plans, specifying improvement measures, responsible persons, and timelines.
Through the above preparations, organizations can better implement ISO/IEC TS 25058 and improve the quality of AI systems.

（II.） Application requirements
ISO/IEC TS 25058 is not a certification standard, but a technical specification that provides guidance for quality assessment of artificial intelligence (AI) systems. When implementing ISO/IEC TS 25058, organizations should meet the following conditions:
1. Organizational type: Applicable to all types of organizations engaged in the development and use of artificial intelligence.
2. System scope: Clearly define the specific scope of the AI system that needs to be evaluated, including its functional modules, application scenarios, data sources, etc.
3. Evaluation objectives: Set specific goals for AI system quality evaluation, such as improving system accuracy, enhancing system security, and improving user experience.
4. Evaluation Plan: Develop a detailed evaluation plan, including an evaluation schedule, resource allocation, evaluation methods, and tool selection.

The following is a full process guide for ISO/IEC TS 25058:2024 Artificial Intelligence System Quality Assessment Management System, covering the key steps for enterprises from building from scratch to obtaining certification, suitable for internal compliance or third-party certification scenarios within organizations: