LIME, SHAP & Attention Visualization
Hands-on tools to explain individual black-box model predictions
Local vs Global Explanations
There are two types of model explanations: **Global explanations** describe how the model behaves overall: feature importances, partial dependence plots, global SHAP summary. Useful for understanding the model's general strategy. **Local explanations** describe why the model made a specific prediction for a specific input: why was this loan denied? Why did the model flag this scan? Useful for auditing individual decisions. LIME and SHAP both provide local explanations. SHAP also aggregates into global explanations.
SHAP Values
SHAP (SHapley Additive exPlanations) is grounded in game theory. Each feature's SHAP value measures its contribution to the prediction relative to the average prediction across all inputs.
import shap
import xgboost as xgb
model = xgb.XGBClassifier()
model.fit(X_train, y_train)
# Compute SHAP values
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
# Explain a single prediction
sample_idx = 42
print(f"Prediction: {model.predict_proba(X_test[sample_idx:sample_idx+1])[0, 1]:.3f}")
print("\nTop contributing features:")
feature_shap = list(zip(X_test.columns, shap_values[sample_idx]))
feature_shap.sort(key=lambda x: abs(x[1]), reverse=True)
for feature, value in feature_shap[:5]:
direction = "increases" if value > 0 else "decreases"
print(f" {feature}: {direction} prediction by {abs(value):.3f}")
# Global summary
shap.summary_plot(shap_values, X_test, plot_type="bar")Prediction: 0.847 Top contributing features: annual_income: decreases prediction by 0.312 credit_history_months: increases prediction by 0.241 debt_to_income: increases prediction by 0.198 num_late_payments: increases prediction by 0.167 loan_amount: increases prediction by 0.089