Menu

Explainability in Deep Learning

Explainability in Deep Learning

Why Black Box Models Are a Problem

As models get more complex, businesses and stakeholders need transparency in predictions to build trust, stay compliant, and debug issues. A deep network might make great predictions, but if nobody, not even the person who built it, can explain why it made a specific call, that is hard to trust, especially in regulated industries. 

Picture a bank using a model to approve or deny loans. If it denies someone and cannot explain why, that is a problem both for the customer who deserves an answer and for the bank, which may need to meet fair lending regulations. This exact worry is what gave rise to explainable AI, often shortened to XAI, built specifically to crack open these black boxes.

SHAP and LIME Explained

LIME (Local Interpretable Model-agnostic Explanations)

SHAP (SHapley Additive exPlanations)

Treats the model as a black box and explains individual predictions.Explains predictions by measuring each feature’s contribution using game theory.
Focuses on local explanations around a single prediction.Provides fair contribution scores for each feature in a prediction.
Builds a simple approximation model around the instance being explained.Uses Shapley values to distribute prediction “credit” among features.
Explanations are accurate only near the selected data point.More mathematically consistent and stable across explanations.
Easy to apply to any model.Also model-agnostic but more computationally expensive.