57% reduction in AI maintenance costs

Business context

A client in the financial sector was developing advanced AI applications, including both generative and classic models. With the upcoming AI Act requirements and the goal of achieving ISO 42001 certification, it was necessary to standardise, automate, and control the internal audit and compliance of their AI systems.

The client was struggling with:

• High costs related to error handling, quality improvement, and auditing AI systems.
• Poor scalability of manual testing and compliance processes.
• The need to document compliance with regulations, including the AI Act.

Methodology for estimating AI Governance costs

The Speednet team estimated the AI Governance costs for a single AI system using data the client provided in a specially prepared survey.

The probability of at least one error occurring is calculated using the normalised probabilities of each individual error.

Sources for probability calculations:

Based on the calculated probability of at least one error per query, and knowing the query volume from the survey, we can calculate the expected number of errors for each system over a year. Below is an example of the estimated annual number of errors with an expected monthly request volume of 800,000.

This figure represents the minimum expected number of random errors from the list of detectable errors mentioned above. Not all errors in this set will be critical; some may only relate to the quality and presentation of the response.

The error estimation serves as a basis for calculating the personnel costs for managing tests and errors. To estimate the cost, the current, actual cost of handling a single error is first calculated from the survey data.

The cost of handling an error varies depending on the type of management: at the testing level, the compliance level, and the Data Science and MLOps level. The current situation, due to a lack of automation, does not allow for the identification and handling of all potential errors. Knowing the cost factors, we can calculate the cost of managing all the estimated annual errors.

The cost of handling all errors is calculated using a logarithmic function, a method that reflects real-world conditions. The reason for this is that the number of new edge cases and test cases grows logarithmically, not linearly, with the number of user requests. This means the volume of new, specific error cases that weren’t covered by previous tests doesn’t scale directly. Most unique errors and edge cases appear at the very beginning of an AI system’s life. As more data flows through it, the probability of repeating the same or similar errors increases, causing the discovery of new issues to slow down naturally.

In the next step, the personnel costs for the audit are calculated, which are closely linked to the compliance needs arising from the AI Act. These costs are similar for every application and depend mainly on the complexity of the system’s architecture, the volume of documentation, the supplier chain, and many other factors. For this estimation, we made assumptions about the time needed to complete each stage of the audit.

The assumed monthly time required to complete the different audit stages was based on data from our clients’ compliance departments and partner law firms that provide AI system audit services. The calculation of individual costs takes into account the number of people currently employed in specific roles at our client’s company, essentially assuming “how long this process would take today.” With the time-consuming nature of the process calculated, we can determine its cost using the rates included in the survey we received from the client, assuming market rates.

Below is an example of the calculated personnel costs related to preparing for and conducting an internal audit of a single AI system.

Auditor’s automated features reduce the cost of the process inventory and internal audit by 90% and completely eliminate the expense of collecting data and evidence.

Cost comparison

Below, we present a comparison of the costs associated with AI Governance for the specified system without a suitable tool and with the use of Speednet Auditor. Estimated annual number of errors to handle: 864,564.

Results

• Total AI Governance costs were cut by over 57% annually.
• The time required for process inventories and internal audits fell by 90%, and
• The costs for gathering data and compliance evidence were eliminated entirely.