Commit a1866315 authored by Eszter Schoell's avatar Eszter Schoell
Browse files

moved from content

parents
Loading
Loading
Loading
Loading

.gitignore

0 → 100644
+2 −0
Changes for .gitignore: 2 added lines, 0 removed lines.
Original line number Diff line number Diff line
.ipynb_checkpoints/
.DS_store

Dockerfile

0 → 100644
+86 −0
Changes for Dockerfile: 86 added lines, 0 removed lines.
Original line number Diff line number Diff line
# Base debian system
FROM debian:8.5
ENV LANG=C.UTF-8 LC_ALL=C.UTF-8

# Update OS
RUN apt-get update --fix-missing && apt-get install -y wget bzip2 ca-certificates \
    libglib2.0-0 libxext6 libsm6 libxrender1 \
    git mercurial subversion

# Add user joyvan
RUN useradd -ms /bin/bash joyvan

# Anaconda Python 3.5
RUN echo 'export PATH=/opt/conda/bin:$PATH' > /etc/profile.d/conda.sh && \
    wget --quiet https://repo.continuum.io/archive/Anaconda3-4.2.0-Linux-x86_64.sh -O ~/anaconda.sh && \
    /bin/bash ~/anaconda.sh -b -p /opt/conda && \
    rm ~/anaconda.sh

RUN apt-get install -y curl grep sed dpkg && \
    TINI_VERSION=`curl https://github.com/krallin/tini/releases/latest | grep -o "/v.*\"" | sed 's:^..\(.*\).$:\1:'` && \
    curl -L "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini_${TINI_VERSION}.deb" > tini.deb && \
    dpkg -i tini.deb && \
    rm tini.deb && \
    apt-get clean

ENV PATH /opt/conda/bin:$PATH

ENTRYPOINT [ "/usr/bin/tini", "--" ]
CMD [ "/bin/bash" ]

# Java
RUN apt-get -f -y install default-jdk

# H2o deps
RUN pip install --upgrade pip && pip install requests \
    tabulate \
    six \
    future \
    colorama

# H2o
RUN pip uninstall h2o || true && \
    pip install -f https://h2o-release.s3.amazonaws.com/h2o/rel-weierstrass/2/Python/h2o-3.14.0.2-py2.py3-none-any.whl --trusted-host h2o-release.s3.amazonaws.com h2o

# Git
RUN apt-get -y install git
RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && \
    apt-get install git-lfs

# XGBoost
RUN apt-get -y install gcc g++ make && \
    conda install libgcc && \
    git clone --recursive https://github.com/dmlc/xgboost.git && \
    cd xgboost && \
    make && \
    cd python-package && \
    python setup.py install #--user

ENV PATH /xgboost:$PATH

# GraphViz
RUN apt-get -y install graphviz

###############################
RUN pip install matplotlib==2.0.2 \
		seaborn==0.8.1

# Launchbot labels
LABEL name.launchbot.io="ormlanders/interpretable-ml-python-xgboost-h2o"
LABEL workdir.launchbot.io="/home/jovyan"
LABEL 8888.port.launchbot.io="Jupyter Notebook"

# Set the working directory
WORKDIR /home/joyvan

# Add files
COPY *.ipynb /home/joyvan/
COPY default_of_credit_card_clients.xls /home/joyvan/default_of_credit_card_clients.xls

USER joyvan

# Expose the notebook port
EXPOSE 8888

# Start the notebook server
CMD jupyter notebook --no-browser --port 8888 --ip=* --NotebookApp.token='' --NotebookApp.disable_check_xsrf=True

README.md

0 → 100644
+71 −0
Changes for README.md: 71 added lines, 0 removed lines.
Original line number Diff line number Diff line
# Interpretable-machine-learning-with-Python-XGBoost-and-H2O

Usage of AI and machine learning models is likely to become more commonplace as larger swaths of the economy embrace automation and data-driven decision-making. While these predictive systems can be quite accurate, they are usually treated as inscrutable black boxes that produce only numeric predictions with no accompanying explanations. Unfortunately, recent studies and recent events have drawn attention to mathematical and sociological flaws in prominent weak AI and ML systems, but practitioners don’t often have the right tools to pry open machine learning models and debug them. This Oriole is part of a series that introduces several new approaches to that increase transparency, accountability, and trustworthiness in machine learning models. If you are a data scientist or analyst and you want to explain a machine learning model to your customers or managers (or if you have concerns about documentation, validation, or regulatory requirements), then this series of Jupyter notebooks is for you!

The notebooks highlight techniques such as:
* [Monotonic XGBoost models, partial dependence, and individual conditional expectation plots](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/xgboost_pdp_ice.ipynb)
* [Decision tree surrogates, reason codes, and ensembles of explanations](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/dt_surrogate_loco.ipynb)
* [LIME](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/lime.ipynb)
* [Sensitivity and residual analysis](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/resid_sens_analysis.ipyn)

### Enhancing Transparency in Machine Learning Models with Python and XGBoost - [Notebook](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/xgboost_pdp_ice.ipynb)

![](./readme_pics/pdp_ice.png)

Monotonicity constraints can turn opaque, complex models into transparent, and potentially regulator-approved models, by ensuring predictions only increase or only decrease for any change in a given input variable. In this notebook, I will demonstrate how to use monotonicity constraints in the popular open source gradient boosting package XGBoost to train a simple, accurate, nonlinear classifier on the UCI credit card default data.

Once we have trained a monotonic XGBoost model, we will use partial dependence plots and ICE plots to investigate the internal mechanisms of the model and to verify its monotonic behavior. Partial dependence plots show us the way machine-learned response functions change based on the values of one or two input variables of interest, while averaging out the effects of all other input variables. ICE plots can be used to create more localized descriptions of model predictions, and ICE plots pair nicely with partial dependence plots.


### Increase Transparency and Accountability in Your Machine Learning Project with Python - [Notebook](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/dt_surrogate_loco.ipynb)

![](./readme_pics/dt_surrogate.png)

Gradient boosting machines (GBMs) and other complex machine learning models are popular and accurate prediction tools, but they can be difficult to interpret. Surrogate models, feature importance, and reason codes can be used to explain and increase transparency in machine learning models. In this Oriole, we will train a GBM on the UCI credit card default data. Then we’ll train a decision tree surrogate model on the original inputs and predictions of the complex GBM model and see how the variable importance and interactions displayed in the surrogate model yield an overall, approximate flowchart of the complex model’s predictions. We will also analyze the global variable importance of the GBM and compare this information to the surrogate model, to our domain expertise, and to our reasonable expectations.

To get a better picture of the complex model’s local behavior and to enhance the accountability of the model’s predictions, we will use a variant of the leave-one-covariate-out (LOCO) technique. LOCO enables us to calculate the local contribution each input variable makes toward each model prediction. We will then rank the local contributions to generate reason codes that describe, in plain English, the model’s decision process for every prediction.


### Explain Your Predictive Models to Business Stakeholders with LIME using Python and H2O - [Notebook](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/lime.ipynb)

![](./readme_pics/lime.png)

Machine learning can create very accurate predictive models, but these models can be almost impossible to explain to your boss, your customers, or even your regulators. This Oriole will use LIME to increase transparency and accountability in a complex GBM model trained on the UCI credit card default data. LIME is a method for building linear surrogate models for local regions in a data set, often single rows of data. LIME sheds light on how model predictions are made and describes local model mechanisms for specific rows of data. Because the LIME sampling process may feel abstract to some practitioners, this Oriole will also introduce a more straightforward method of creating local samples for LIME.

Once local samples have been generated, we will fit LIME models to understand local trends in the complex model’s predictions. LIME can also tell us the local contribution of each input variable toward each model prediction, and these contributions can be sorted to create reason codes -- plain English explanations of every model prediction. We will also validate the fit of the LIME model to enhance trust in our explanations using the local model’s R2 statistic and a ranked predictions plot.

### Testing Machine Learning Models for Accuracy, Trustworthiness, and Stability with Python and H2O - [Notebook](https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O/blob/master/resid_sens_analysis.ipynb)

![](./readme_pics/resid.png)

Because machine learning model predictions can vary drastically for small changes in input variable values, especially outside of training input domains, sensitivity analysis is perhaps the most important validation technique for increasing trust in machine learning model predictions.
Sensitivity analysis investigates whether model behavior and outputs remain stable when input data is intentionally perturbed, or other changes are simulated in input data. In this Oriole, we will enhance trust in a complex credit default model by testing and debugging its predictions with sensitivity analysis.

We’ll further enhance trust in our model using residual analysis. Residuals refer to the difference between the recorded value of a target variable and the predicted value of a target variable for each row in a data set. Generally, the residuals of a well-fit model should be randomly distributed, because good models will account for most phenomena in a data set, except for random error. In this Oriole, we will create residual plots for a complex model to debug any accuracy problems arising from underfitting or outliers.

## Installation

The ideal way to access these notebook is through [O'Reilly Safari](https://www.safaribooksonline.com/). Doing so will enable video narration by the notebook author and no installation of software packages is required. To use these notebooks outside of the Safari platform, follow the instructions below.

  Install:

  1. Anaconda Python 4.2.0 from the [Anaconda archives](https://repo.continuum.io/archive/).
  2. [Java](https://java.com/download).
  3. The latest stable [h2o](https://www.h2o.ai/download/) Python package.
  4. [Git](https://git-scm.com/downloads).
  5. [XGBoost](https://github.com/dmlc/xgboost) with Python bindings.
  6. [GraphViz](http://www.graphviz.org/).

  Anaconda Python, Java, Git, and GraphViz must be added to your system path.

  From a terminal:

  7. Clone the mli-resources repository with examples.</br>
  `$ git clone https://content.oreilly.com/oriole/Interpretable-machine-learning-with-Python-XGBoost-and-H2O.git`

  8. `$ cd Interpretable-machine-learning-with-Python-XGBoost-and-H2O`

  9. Start the Jupyter notebook server.</br>
  `$ jupyter notebook`

  10. Navigate to the port Jupyter directs you to on your machine.
+5.28 MiB

File added.

No diff preview for this file type.

+943 −0

File added.

Preview size limit exceeded, changes collapsed.

Loading