Identifying AI Problems and Defining Success Criteria

6.1 Start With the Problem

A common mistake in AI development is choosing a model before understanding the problem.

A better approach is to identify the user need, determine the expected inputs and outputs, establish a baseline, and decide how success will be measured.

Consider a company whose employees spend several hours each day categorizing support tickets manually.

The goal is not simply to build an AI system. The goal is to reduce categorization effort while ensuring that tickets reach the right team.

6.2 A Practical Problem-Solving Framework

Step 1: Define the problem. Describe the difficulty, the affected users, and why it matters.

Step 2: Identify inputs and outputs. Determine what information the system receives and what it must produce.

Step 3: Establish a baseline. Measure how the current process performs without the proposed solution.

Step 4: Select success metrics. Choose measurements that reflect the actual business or user objective.

Step 5: Evaluate feasibility and risk. Consider data availability, privacy, operating costs, error consequences, and maintenance.

6.3 Choose the Right Approach

The simplest approach that meets the requirements is often the best starting point.

6.4 Define Measurable Success

Suppose a company receives 10,000 support tickets per month.

A proposed classifier could be evaluated using:

• Accuracy: The proportion of all predictions that are correct.
• Precision: Of the tickets assigned to a category, how many truly belong to it?
• Recall: Of all tickets that truly belong to a category, how many are identified?
• Time saved: How much manual effort is reduced?
• Cost: Does the solution provide enough value to justify operating and maintenance expenses?
• Reliability: How does the system handle uncertain or unusual cases?

One metric is rarely sufficient. A classifier might achieve high overall accuracy while performing poorly on a rare but urgent category. Teams should evaluate the metrics that matter most for the intended use.

6.5 Practical Activity: Plan an AI Project

Choose a problem from your work, studies, or everyday life and complete the following project outline.

Project name: Support Ticket Classification

Problem: Support tickets are manually categorized, delaying assignment to the appropriate team.

Input: Ticket subject and message text.

Expected output: A predicted category, such as billing, technical issue, or account access.

Potential approach: A machine learning classifier trained on labeled historical tickets.

Success criteria:

• Measure precision and recall for each category.
• Reduce manual categorization time.
• Track how often employees correct predictions.
• Identify high-risk cases that require manual review.

Potential risks: Poor-quality training data, misclassified urgent tickets, privacy concerns, and changing message patterns.

Before building the solution, compare it with a simpler rule-based baseline. If rules solve the problem adequately, a machine learning model may not be necessary.