Explaining a simple OR function

This notebook examines what it looks like to explain an OR function using SHAP values. It is based on a simple example with two features is_young and is_female, roughly motivated by the Titanic survival dataset where women and children were given priority during the evacuation and so were more likely to survive. In this simulated example this effect is taken to the extreme, where all children and women survive and no adult men survive.

[2]:
import numpy as np
import xgboost
import shap

Create a dataset following an OR function

[3]:
N = 40000
M = 2

# randomly create binary features for (is_young, and is_female)
X = (np.random.randn(N,2) > 0) * 1

# force the first sample to be a young boy
X[0,0] = 1
X[0,1] = 0

# you survive only if you are young or female
y = ((X[:,0] + X[:,1]) > 0) * 1

Train an XGBoost model to mimic this OR function

[4]:
model = xgboost.XGBRegressor(n_estimators=100, learning_rate=0.1)
model.fit(X, y)
model.predict(X)
[21:39:39] WARNING: src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.
[4]:
array([9.9998689e-01, 9.9998689e-01, 1.3232231e-05, ..., 9.9998701e-01,
       1.3232231e-05, 9.9998701e-01], dtype=float32)

Explain the prediction for a young boy

Using the training set for the background distribution

Note that in the example explanation below is_young = True has a positive value (meaning it increases the model output, and hence the prediction of survival), while is_female = False has a negative value (meaning it decreases the model output). While one could argue that is_female = False should have no impact because we already know that the person is young, SHAP values account for the impact a feature has even when we don’t nessecarily know the other features, which is why is_female = False still has a negative impact on the prediction.

[5]:
explainer = shap.TreeExplainer(model, X, feature_dependence="independent")
shap_values = explainer.shap_values(X[:1,:])
print("explainer.expected_value:", explainer.expected_value.round(4))
print("SHAP values for (is_young = True, is_female = False):", shap_values[0].round(4))
print("model output:", (explainer.expected_value + shap_values[0].sum()).round(4))
explainer.expected_value: 0.7507
SHAP values for (is_young = True, is_female = False): [ 0.3761 -0.1268]
model output: 1.0

Using only negative examples for the background distribution

The point of this second explanation example is to demonstrate how using a different background distribution can change the allocation of credit among the input features. This happens because we are now comparing the importance of a feature as compared to being someone who died (an adult man). The only thing different about the young boy from someone who died is that the boy is young, so all the credit goes to the is_young = True feature.

This highlights that often explanations are clearer when a well defined background group is used. In this case it changes the explanation from how this sample is different than typical, to how this sample is different from those who died (in other words, why did you live?).

[6]:
explainer = shap.TreeExplainer(model, X[y == 0,:], feature_dependence="independent")
shap_values = explainer.shap_values(X[:1,:])
print("explainer.expected_value:", explainer.expected_value.round(4))
print("SHAP values for (is_young = True, is_female = False):", shap_values[0].round(4))
print("model output:", (explainer.expected_value + shap_values[0].sum()).round(4))
explainer.expected_value: 0.0
SHAP values for (is_young = True, is_female = False): [1. 0.]
model output: 1.0

Using only positive examples for the background distribution

We could also use only positive examples for our background distribution, and since the difference between the expected output of the model (under our background distribution) and the current output for the young boy is zero, the sum of the SHAP values will be also be zero.

[7]:
explainer = shap.TreeExplainer(model, X[y == 1,:], feature_dependence="independent")
shap_values = explainer.shap_values(X[:1,:])
print("explainer.expected_value:", explainer.expected_value.round(4))
print("SHAP values for (is_young = True, is_female = False):", shap_values[0].round(4))
print("model output:", (explainer.expected_value + shap_values[0].sum()).round(4))
explainer.expected_value: 1.0
SHAP values for (is_young = True, is_female = False): [ 0.1689 -0.1689]
model output: 1.0

Using young women for the background distribution

If we compare the sample to young women then neither of the features matter except for adult men, in which both features are given equal credit for their death (as one might intuitively expect).

[8]:
explainer = shap.TreeExplainer(model, np.ones((1,M)), feature_dependence="independent")
shap_values = explainer.shap_values(X[:10,:])
shap_values[0:3].round(4)
[8]:
array([[ 0. , -0. ],
       [ 0. , -0. ],
       [-0.5, -0.5]])