Evaluating AI Solutions

How do you make sure the AI service you buy is right?

Deciding which AI to buy for your organisation is a minefield, with so many new applications, services and features. The pace of change makes it nearly impossible to know which service is right today, let alone next week.

I’ve been considering this problem for some time, but it wasn’t until my clients started to ask me if they should switch from ChatGPT+ to Microsoft CoPilot that I decided I needed to provide some more objective guidance.

This week I went into some detail using an analytic hierarchy process to help me find a way to objectively evaluate AI solutions. While this isn’t perfect, the process of going through this helped to provide clarity on what is important and how businesses can deal with this rapidly changing landscape.

For this process I've create the following criteria for evaluation, (I've also included in brackets the indicative weighting that I give each of these criteria).

Model (27%): Evaluates the underlying algorithms, data quality, speed, breadth of knowledge, and perceived intelligence of the AI.

Flexibility (17%): Considers the AI's adaptability to various use cases and its multimodality, including handling different data types like images and documents.

Privacy (11%): Focuses on the risks of data breaches, GDPR compliance, and whether the AI uses private data for training or offers a privacy-focused model.

UX (User Experience)(11%): Assesses the intuitiveness and user-friendliness of the AI service, especially for new users in a professional context.

Features (8%): Looks at the unique functionalities of each AI service, such as image recognition or database auto-completion.

Workflow (7%): Measures how well the AI aligns with specific tasks and processes, and facilitates the creation and repetition of workflows.

Access and Integration (7%): Evaluates the ease of accessing the AI service and how well it integrates with existing systems.

Private Data (6%): Considers the AI's capability to analyse and infer from the user's own data, ensuring it is used effectively without compromising privacy.

Public Data (4%): Examines the AI's ability to access and use data from the public domain, like web searches or social media information.

Cost (2%): Analyses the pricing model, subscription costs, API call charges, and the overall value for money considering the features offered.

While these criteria or weightings might not be right for every organisation, I believe they are right for the current position of GenAI services in a modern, innovative business. 

In future newsletters I'll be illustrating how these criteria translate to directly rating AI solutions and comparing them so you can make an informed decision on which services to pay for and which to avoid.