Can’t a computer do that?

class: inverse, center, middle

# Can't a computer do that?

<img src="figures/Red_camera_eye.svg.png" width="25%" />

### Exploring automatic evaluation of statistical graphics

.large[Adam Loy | Gustavus Adolphus MCS Seminar | 30 Sept 2019]

---
class: center, middle

# Have you ever fit a regression model?

# Did you check residual plots?

# Have you ever seen a residual plot you weren't sure how to interpret?

---

# Sample homework problem

.pull-left[

`\(\widehat{\tt heart.rate} = b_0 + b_1 \cdot {\tt duration}\)`

![](gustavus_talk_sept2019_files/figure-html/pengiun-scatter-1.svg)
]

.pull-right[

Is there any evidence of structure?

![](gustavus_talk_sept2019_files/figure-html/penguin-residuals-1.svg)
]

---
class: middle

# apophenia

### the tendency to perceive a connection or meaningful pattern between unrelated or random things (such as objects or ideas)

.footnote[
"apophenia” Meriam-Webster Dictionary Online, September 2019, merriam-webster.com
]

---

background-image: url(figures/usual_suspects.jpg)
background-size: cover

# The lineup protocol

---

## Which residual plot is not like the others?

![](gustavus_talk_sept2019_files/figure-html/resid lineup-1.svg)

???
Observed plot: 1

---
class: center, middle
# What did we just do?

## We compared the **data plot** with **null plots** of samples where, by construction, there really is nothing going on

## This allows us to make decisions from our graphics on a firm foundation

---

# Inspiration

.large[
Classical hypothesis testing provides an established framework for inference.

- Formulate two competing hypotheses: `\(H_0\)` and `\(H_1\)`.

- Choose a test statistic that characterizes the information in the sample relevant to `\(H_0\)`.

- Determine the sampling distribution of the chosen statistic when `\(H_0\)` is true.

- Compare the calculated test statistic to the sampling distribution to determine whether it is "extreme."
]

---

.left-column[

Hypotheses

Test statistic

Reference distribution

Evidence against H0 if...

]

.pull-left[
.bold[Conventional Inference]

H0: linear relationship

]

.bold[Visual Inference]

H1: nonlinear relationship

---

.left-column[

Hypotheses

Test statistic

Reference distribution

Evidence against H0 if...

]

.pull-left[
.bold[Conventional Inference]

H0: linear relationship

Nested F-test

M1: `\(E(Y|X) = \beta_0 + \beta_1x\)`

M2: `\(E(Y|X) = \beta_0 + \beta_1x + \beta_2x^2\)`

]

.bold[Visual Inference]

H1: nonlinear relationship

---

.left-column[

Hypotheses

Test statistic

Reference distribution

Evidence against H0 if...

]

.pull-left[
.bold[Conventional Inference]

H0: linear relationship

Nested F-test

M1: `\(E(Y|X) = \beta_0 + \beta_1x\)`

M2: `\(E(Y|X) = \beta_0 + \beta_1x + \beta_2x^2\)`

]

.bold[Visual Inference]

H1: nonlinear relationship

---

.left-column[

Hypotheses

Test statistic

Reference distribution

Evidence against H0 if...

]

.pull-left[
.bold[Conventional Inference]

H0: linear relationship

Nested F-test

M1: `\(E(Y|X) = \beta_0 + \beta_1x\)`

M2: `\(E(Y|X) = \beta_0 + \beta_1x + \beta_2x^2\)`

the test statistic is "extreme"

]

.bold[Visual Inference]

H1: nonlinear relationship

the data plot is identifiable

---
class: clear

### Which Q-Q plot is most different?

![](gustavus_talk_sept2019_files/figure-html/qqplot lineup-1.svg)

???

Observed plot: 9

---
class: clear

### Is it rude to bring a baby on a plane?

![](gustavus_talk_sept2019_files/figure-html/bar lineup-1.svg)

???

Observed plot: 11

---
class: middle, center

# That's a neat idea, but where does it fit into my workflow?

---

# Applications of visual inference

.large[
1. Model diagnostics

2. Interpreting unfamiliar plots

3. When large-sample theory breaks down

4. Conducting research in (statistical) graphics
]

---
class: middle, center

# That's great!

# Can you train a computer do this?

---

# How?

.large[
We have to turn plot evaluation into a classification problem...

.pull-left[

.center[Good residual plot]

]

.pull-right[

.center[Bad residual plot]

]
]

---

# How can we model this?

.large[
1. Statistical paradigm

+ Logistic regression
  
  + .bold[Random forests]
  
  + Support vector machines (SVMs)

2. Computer vision paradigm

+ Neural networks
]

---
class: middle, center
# Statistical modeling

## work with Cari Comnick, Logan Crowl, Sophia Gunn, Aidan Mullan

---

# Major questions

.pull-left[

1. What's the response?

2. **What are the predictors?**

3. How are we going to relate the two?

4. Does that model fit?

5. How well does this model perform?
]

Good or bad class label

Hmm... that's hard

We'll use a random forest

Be sure to check your model

Need to gauge predictive accuracy

---
class: middle, center

# How can we summarize key features/characteristics of scatterplots?

# Summary statistics are not the answer!

---
class: center, middlep

background-image: url(figures/Datasaurus12.png)
background-size: cover

---

# Scagnostics (scatterplot diagnostics)

.large[
- Originally proposed by Tukey & Tukey (1985)
 
- Original idea was to provide indices to help guide exploration of a large scatterplot matrix
 
- Wilkinson, Anand, & Grossman (2005) proposed 9 graph-theoretic metrics
 
- Wilkinson & Wills (2008) explored the distribution of scagnostics on different classes of scatterplots
]

---

.left-column[
# Shape
## Stringy
]

.right-column[

.pull-left[
<img src="img/stringyLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/stringyHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[

.Large[

Low

Medium

High
]
]

]

???

Diameter / length of the MST

---

.left-column[
# Shape
## Stringy
## Skinny
]

.right-column[

.pull-left[
<img src="img/skinnyLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/skinnyHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

???

`\(1 - \dfrac{\sqrt{4\pi \cdot area(A)}}{perimeter(A)}\)`

---

.left-column[
# Shape

## Stringy
## Skinny
## Convex

]

.right-column[

.pull-left[
<img src="img/convexLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/convexHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

???

area(alpha hull)/area(convex hull)

---

.left-column[
# Association
## Monotonic
]

.right-column[

.pull-left[
<img src="img/monotonicLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/monotonicHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

???

spearman rank correlation squared

---

.left-column[
# Density
## Outlying
]

.right-column[

.pull-left[
<img src="img/outlyingLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/outylingHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

???

`\(\dfrac{length(T_{(outliers)})}{length(T)}\)`

---

.left-column[
# Density
## Outlying
## Sparse
]

.right-column[

.pull-left[
<img src="img/sparseLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/sparseHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

---

.left-column[
# Density
## Outlying
## Sparse
## Skewed
]

.right-column[

.pull-left[
<img src="img/skewedLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/skewedHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

---

.left-column[
# Density
## Outlying
## Sparse
## Skewed
## Clumpy

]

.right-column[

.pull-left[
<img src="img/clumpyLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/clumpyHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

---

.left-column[
# Density
## Outlying
## Sparse
## Skewed
## Clumpy
## Striated
]

.right-column[

.pull-left[
<img src="img/striatedLow.png" width="70%" style="display: block; margin: auto 0 auto auto;" />

<img src="img/StriatedHigh.png" width="70%" style="display: block; margin: auto 0 auto auto;" />
]

.pull-right[
.Large[

Low

Medium

High
]
]
]

---

.pull-left[
<img src="gustavus_talk_sept2019_files/figure-html/unnamed-chunk-34-1.svg" width="75%" style="display: block; margin: auto;" />

<table>
<tbody>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Outlying </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0865 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: silver !important;"> Skewed </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: silver !important;"> 0.7921 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: silver !important;"> Clumpy </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: silver !important;"> 0.2708 </td>
 </tr>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Sparse </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0835 </td>
 </tr>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Striated </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.1639 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: gainsboro !important;"> Convex </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: gainsboro !important;"> 0.2138 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: silver !important;"> Skinny </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: silver !important;"> 0.7661 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: silver !important;"> Stringy </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: silver !important;"> 0.5314 </td>
 </tr>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Monotonic </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0051 </td>
 </tr>
</tbody>
</table>

]

.pull-right[
<img src="gustavus_talk_sept2019_files/figure-html/unnamed-chunk-36-1.svg" width="75%" style="display: block; margin: auto;" />

<table>
<tbody>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Outlying </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.1231 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: gainsboro !important;"> Skewed </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: gainsboro !important;"> 0.6137 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: gainsboro !important;"> Clumpy </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: gainsboro !important;"> 0.0898 </td>
 </tr>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Sparse </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0814 </td>
 </tr>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Striated </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0703 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: silver !important;"> Convex </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: silver !important;"> 0.5133 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: gainsboro !important;"> Skinny </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: gainsboro !important;"> 0.5628 </td>
 </tr>
 <tr>
 <td style="text-align:left;font-weight: bold;color: black !important;background-color: gainsboro !important;"> Stringy </td>
 <td style="text-align:right;font-weight: bold;color: black !important;background-color: gainsboro !important;"> 0.3490 </td>
 </tr>
 <tr>
 <td style="text-align:left;color: black !important;background-color: white !important;"> Monotonic </td>
 <td style="text-align:right;color: black !important;background-color: white !important;"> 0.0041 </td>
 </tr>
</tbody>
</table>

]

---
class: middle, center

# Random forests

---

# Classification trees

---

.large[Start with all observations in one group]

.large[Find the variable/split that best separates the classes]

---

.large[Find the variable/split that best separates the classes]

.large[Within each split, find the best variable/split that separates the outcomes]

---

.large[Within each split, find the best variable/split that separates the outcomes]

.large[Continue until the groups are too small or sufficiently "pure"]

---

# Visualizing the tree

---

# Classification trees
.large[
.pull-left[
**Advantages**

- Easy to explain

- May more closely mirror human decision-making than other approaches

- Can be displayed graphically, and are easily interpreted even by a non-expert (especially if they are small)

- Can easily handle categorical predictors 
]

.pull-right[
**Disadvantages**

- Generally do not have the same level of predictive accuracy as many other classification  approaches

- High variability
]
]

---

# Growing a random forest

.large[
- Sample with replacement from your training data set to get B bootstrapped training sets

- Grow a tree for each of the B training sets (grow deep, don't prune)

+ For each split only consider a random sample of m out of p predictors

- Majority (plurality) vote to classify an observation
]

---
background-image: url(figures/ex_random_forest.png)
background-size: contain

---

# Training data: 14865 signal plots

---

#Training data: 14865 null plots

---

# Model evaluation
.large[
- Generated 1000 signal plots

- Calculated the 9 scagnostics on each

- Used random forest to classify as "signal" or "noise"
]

.content-box-blue[
.large[972 of the 1000 signal plots were correctly classified]
]

.content-box-yellow[
.large[
Also evaluated on 1000 lineups of size 20, with unspecified number of signal plots

- 93.5% of linear plots identified as signal; 5.3% false positive rate
]
]

---
class: middle, center
# Computer vision models

---
# Motivation: Giora Simchoni's blog post

---

## Giora trained a computer vision model two ways

.Large[
- classification: significant correlation vs not

- regression: to predict the correlation
]

---
class: clear

## Success, picked plot 16

---
class: clear

## Success, failed to pick plot 4

---
class: clear

## Fail! Doesn't see the strong nonlinear association. Picks the most linear.

---

# Deep learning

.footnote[
Source: [Abdellatif Abdelfattah](https://medium.com/@tifa2up/image-classification-using-deep-neural-networks-a-beginner-friendly-approach-using-tensorflow-94b0a090ccd4)
]

---

# Deep learning

.footnote[Source: [Chihuahua or Muffin? Brad Folkens](https://blog.cloudsight.ai/chihuahua-or-muffin-1bdf02ec1680)]

---

## Neural networks

- `\(x_i\)` = input variable

- `\(v_j\)` = function of linear combinations of the inputs (e.g. sigmoid)

- `\(y_k\)` = function of linear combinations of the `\(v_j\)` (e.g. softmax)

.footnote[
.scriptsize[Source: [Cheng & Titterington (1994)](https://projecteuclid.org/euclid.ss/1177010638)]
]
???

Derived features Zm are created from linear combinations of the inputs, and then the target Yk is modeled as a function of linear combinations of the Zm

---

# Regression as a neural network

.footnote[
.scriptsize[Source: [Cheng & Titterington (1994)](https://projecteuclid.org/euclid.ss/1177010638)]
]

---

# Images as data

---

# Filtering patterns

.footnote[
Source: [Deep Learning with R](https://www.manning.com/books/deep-learning-with-r)
]

---

# Pooling to find spatial hierarchies

.footnote[
Source: Di Cook's [DSSV slides](http://www.dicook.org/files/dssv19/slides#1)
]

???
to induce spatial-filter hierarchies

---

# Rinse & repeat to reveal other hierarchies

.footnote[
Source: Di Cook's [DSSV slides](http://www.dicook.org/files/dssv19/slides#1)
]

---
background-image: url(img/nn_cat.png)
background-size: cover

---

# Approach I

.large[
- Save the scatterplots as images, then train your CNN

- Di Cook and Shuofan Zhang at Monash University worked on this to detect linearity

- Trained Keras model with 60,000 training data sets for each class: linear vs. not

- Accuracy with simulated test data, 93% 
    + null error 0.0179
    + linear error 0.1176
]

---

# Approach II

.large[ 
- Create images showing the "shape" of the scatterplot, then train your CNN

.pull-left[
<img src="img/nn_hulls1.png" width="70%" style="display: block; margin: auto;" />
]

.pull-right[
<img src="img/nn_hulls2.png" width="70%" style="display: block; margin: auto;" />
]

- Elliot Pickens and I worked on this last spring
]

---

# Approach II
.large[
Trained Keras model on simulated 300 plots for each class

4200 total plots in the training set

.pull-left[
- **Uniform**

- **Spherical**

- **Binormal**

- **Funnel**

- **Exponential**
]

.pull-right[

- **Quadratic**

- **Clustered**

- **Doughnut**

- **Stripe**

- **Sparse** 
]
]

???
1. **Uniform** (2D Poisson process)
2. **Spherical** (spherical normal)
3. **Binormal** (bivariate normal with `\(\rho = \pm 0.6\)`)
4. **Funnel** (bivariate log-normal with `\(\rho = \pm 0.6\)`)
5. **Exponential** (exponential growth/decay plus random error)
6. **Quadratic** (positive/negative quadratic function plus random error)
7. **Clustered** (three separated spherical normals at the vertices of an equilateral triangle)
8. **Doughnut** (two polar uniforms separated by a moat of white space)
9. **Stripe** (product of Uniform and integer [1, 5])
10. **Sparse** (product of integer [1, 3] with itself)

---

## Approach II

.pull-left[
.large[
- 2100 images in the testing set (150 per class)

- Precision = true positive

- Recall = sensitivity
]
]

.pull-right[
Class | Precision | Recall
------|---|---
Uniform | 0.97 | 0.99 
Spherical | 0.70 | 0.61 
Binormal | 0.85 | 0.81 
N. Binormal | 0.96 | 0.87 
Funnel | 0.96 | 0.90 
N. Funnel | 0.94 | 0.93 
N. Expo | 0.97 | 0.99 
Expo | 0.97 | 0.97 
Quadratic | 0.96 | 0.97 
Clustered | 0.73 | 0.83 
Doughnut  | 0.73  | 0.83 
Stripe | 0.89  | 0.98 
Sparse | 1.00 | 1.00 
Logarithmic | 0.99 | 0.92 
]

???
Avg. precision and recall is  about 90%

Precision = proportion of plots classified as uniform that are actually uniform

Recall = proportion of uniform plots that were actually predicted to be uniform

---

# Discussion

.large[
- It's possible to automate the detection of plots, but the training sets are key

- If you use a statistical model/algorithm, you need to carefully consider your predictors

- Computer's haven't beaten human ability to detect plot type (Cook & Zhang)

- Promising results for model diagnosis, exploring large data sets, and prototyping new statistical graphics
]

---

# Joint work
.large[
#### Deep learning:  
[Giora Simchoni](http://giorasimchoni.com/2018/02/07/2018-02-07-book-em-danno/); [Di Cook](http://www.dicook.org/) & Shuofan Zhang; Elliot Pickens

#### Inference: 
Di Cook, Heike Hofmann, [Mahbub Majumder](http://mamajumder.github.io/html/experiments.html), Andreas Buja, Hadley Wickham, Eric Hare, [Susan Vanderplas](https://srvanderplas.netlify.com/), Niladri Roy Chowdhury, Nat Tomasetti

#### Contact: 
[](https://aloy.rbind.io/) aloy@carleton.edu [](https://github.com/aloy) aloy

]
---

# Further reading

.large[
- Buja et al (2009) Statistical Inference for Exploratory Data Analysis and Model Diagnostics, *Roy. Soc. Ph. Tr., A*

- Majumder et al (2013) Validation of Visual Statistical Inference, Applied to Linear Models, *JASA*

- Wickham et al (2010) Graphical Inference for Infovis, *InfoVis*

- Hofmann et al (2012) Graphical Tests for Power Comparison of Competing Design, *InfoVis*

- Loy et al (2017) Model Choice and Diagnostics for Linear Mixed-Effects Models Using Statistics on Street Corners, *JCGS*

]