Table of Contents
*Statistical Regression and Classification: From Linear Models to Machine Learning was awarded the 2017 Ziegel Award for the best book reviewed in Technometrics in 2017.*
Chapter One
Setting the Stage
Example: Predicting Bike-Sharing Activity
Example of the Prediction Goal: Body Fat
Example of the Description Goal: Who Clicks Web Ads?
Optimal Prediction
A Note About E(), Samples and Populations
Example: Do Baseball Players Gain Weight As They Age?
Prediction vs Description
A First Estimator
A Possibly Better Estimator, Using a Linear Model
Parametric vs Nonparametric Models
Example: Click-Through Rate
Several Predictor Variables
Multipredictor Linear Models
Estimation of Coefficients
The Description Goal
Nonparametric Regression Estimation: k-NN
Looking at Nearby Points
Measures of Nearness
The k-NN Method, and Tuning Parameters
Nearest-Neighbor Analysis in the regtools
Package
Example: Baseball Player Data
After Fitting a Model, How Do We Use It for Prediction?
Parametric Settings
Nonparametric Settings
The Generic predict() Function
Overfitting, and the Variance-Bias Tradeoff
Intuition
Example: Student Evaluations of Instructors
Cross-Validation
Linear Model Case
The Code
Applying the Code
k-NN Case
Choosing the Partition Sizes
Important Note on Tuning Parameters
Rough Rule of Thumb
Example: Bike-Sharing Data
Linear Modeling of _(t)
Nonparametric Analysis
Interaction Terms, Including Quadratics
Example: Salaries of Female Programmers and Engineers
Saving Your Work
Higher-Order Polynomial Models
Classification Techniques
It's a Regression Problem!
Example: Bike-Sharing Data
Crucial Advice: Don't Automate, Participate!
Mathematical Complements
Indicator Random Variables
Mean Squared Error of an Estimator
_(t) Minimizes Mean Squared Prediction Error
_(t) Minimizes the Misclassification Rate
Kernel-Based Nonparametric Estimation of Regression
Functions
General Nonparametric Regression
Some Properties of Conditional Expectation
Conditional Expectation As a Random Variable
The Law of Total Expectation
Law of Total Variance
Tower Property
Geometric View
Computational Complements
CRAN Packages
The Function tapply() and Its Cousins
The Innards of the k-NN Code
Function Dispatch
Centering and Scaling
Further Exploration: Data, Code and Math Problems
Chapter Two
Linear Regression Models
Notation
The "Error Term"
Random- vs Fixed-X Cases
Least-Squares Estimation
Motivation
Matrix Formulations
() in Matrix Terms
Using Matrix Operations to Minimize ()
Models Without an Intercept Term
A Closer Look at lm() Output
Statistical Inference
Assumptions
Classical
Motivation: the Multivariate Normal Distribution Family
Unbiasedness and Consistency
b_ Is Unbiased
Bias As an Issue/Nonissue
b_ Is Statistically Consistent
Inference under Homoscedasticity
Review: Classical Inference on a Single Mean
Back to Reality
The Concept of a Standard Error
Extension to the Regression Case
Example: Bike-Sharing Data
Collective Predictive Strength of the X(j)
Basic Properties
Definition of R
Bias Issues
Adjusted-R
The Leaving-One-Out Method"
Extensions of LOOM
LOOM for k-NN
Other Measures
The Practical Value of p-Values | Small OR Large
Misleadingly Small p-Values
Example: Forest Cover Data
Example: Click Through Data
Misleadingly LARGE p-Values
The Verdict
Missing Values
Mathematical Complements
Covariance Matrices
The Multivariate Normal Distribution Family
The Central Limit Theorem
Details on Models Without a Constant Term
Unbiasedness of the Least-Squares Estimator
Consistency of the Least-Squares Estimator
Biased Nature of S
The Geometry of Conditional Expectation
Random Variables As Inner Product Spaces
Projections
Conditional Expectations As Projections
Predicted Values and Error Terms Are Uncorrelated
Classical \Exact" Inference
Asymptotic (p + )-Variate Normality of b_
Computational Complements
Details of the Computation of ()
R Functions Relating to the Multivariate Normal Distribution
Family
Example: Simulation Computation of a Bivariate
Normal Quantity
More Details of 'lm' Objects
Further Exploration: Data, Code and Math Problems
Chapter Three
Homoscedasticity and Other Assumptions in Practice
Normality Assumption
Independence Assumption | Don't Overlook It
Estimation of a Single Mean
Inference on Linear Regression Coefficients
What Can Be Done?
Example: MovieLens Data
Dropping the Homoscedasticity Assumption
Robustness of the Homoscedasticity Assumption
Weighted Least Squares
A Procedure for Valid Inference
The Methodology
Example: Female Wages
Simulation Test
Variance-Stabilizing Transformations
The Verdict
Further Reading
Computational Complements
The R merge() Function
Mathematical Complements
The Delta Method
Distortion Due to Transformation
Further Exploration: Data, Code and Math Problems
Chapter Four
Generalized Linear and Nonlinear Models
Example: Enzyme Kinetics Model
The Generalized Linear Model (GLM)
Definition
Poisson Regression
Exponential Families
GLM Computation
R's glm() Function
GLM: the Logistic Model
Motivation
Example: Pima Diabetes Data
Interpretation of Coefficients
The predict() Function Again
Overall Prediction Accuracy
Example: Predicting Spam E-mail
Linear Boundary
GLM: the Poisson Regression Model
Least-Squares Computation for Nonlinear Models
The Gauss-Newton Method
Eicker-White Asymptotic Standard Errors
Example: Bike Sharing Data
The Elephant in the Room": Convergence Issues
Further Reading
Computational Complements
R Factors
Mathematical Complements
Maximum Likelihood Estimation
Further Exploration: Data, Code and Math Problems
Chapter Five
Multiclass Classification Problems
Key Notation
Key Equations
Estimating the Functions i(t)
How Do We Use Models for Prediction?
One vs All or All vs All?
Which Is Better?
Example: Vertebrae Data
Intuition
Example: Letter Recognition Data
Example: k-NN on the Letter Recognition Data
The Verdict
The Classical Approach: Fisher Linear Discriminant Analysis
Background
Derivation
Example: Vertebrae Data
LDA Code and Results
Multinomial Logistic Model
Model
Software
Example: Vertebrae Data
The Issue of \Unbalanced" (and Balanced) Data
Why the Concern Regarding Balance?
A Crucial Sampling Issue
It All Depends on How We Sample
Remedies
Example: Letter Recognition
Going Beyond Using the 0.5 Threshhold
Unequal Misclassification Costs
Revisiting the Problem of Unbalanced Data
The Confusion Matrix and the ROC Curve
Code
Example: Spam Data
Mathematical Complements
Classification via Density Estimation
Methods for Density Estimation
Time Complexity Comparison, OVA vs AVA
Optimal Classification Rule for Unequal Error Costs
Computational Complements
R Code for OVA and AVA Logit Analysis
ROC Code
Further Exploration: Data, Code and Math Problems
Chapter Six
Model Fit: Assessment and Improvement
Aims of This Chapter
Methods
Notation
Goals of Model Fit-Checking
Prediction Context
Description Context
Center vs Fringes of the Data Set
Example: Currency Data
Overall Measures of Model Fit
R-Squared, Revisited
Cross-Validation, Revisited
Plotting Parametric Fit Against Nonparametric One
Residuals vs Smoothing
Diagnostics Related to Individual Predictors
Partial Residual Plots
Plotting Nonparametric Fit Against Each Predictor
The freqparcoord Package
Parallel Coordinates
The regdiag() Function
Effects of Unusual Observations on Model Fit
The inuence() Function
Example: Currency Data
Use of freqparcoord for Outlier Detection
Automated Outlier Resistance
Median Regression
Example: Currency Data
Example: Vocabulary Acquisition
Classification Settings
Example: Pima Diabetes Study
Improving Fit
Deleting Terms from the Model
Adding Polynomial Terms
Example: Currency Data
Example: Programmer/Engineer Census Data
Boosting
View from the 30,000 Foot Level
Performance
A Tool to Aid Model Selection
Special Note on the Description Goal
Computational Complements
Data Wrangling for the Word Bank Dataset
Mathematical Complements
The Hat Matrix
Matrix Inverse Update
The Median Minimizes Mean Absolute Deviation
Further Exploration: Data, Code and Math Problems
Chapter Seven
Disaggregating Regressor Effects
A Small Analytical Example
Example: Baseball Player Data
Simpson's Paradox
Example: UCB Admissions Data (Logit)
The Verdict
Unobserved Predictor Variables
Instrumental Variables (IVs)
The IV Method
Stage Least Squares:
Example: Years of Schooling
Multiple Predictors
The Verdict
Random Effects Models
Example: Movie Ratings Data, Random Effects
Multiple Random Effects
Why Use Random/Mixed Effects Models?
Regression Function Averaging
Estimating the Counterfactual
Example: Job Training
Small Area Estimation: \Borrowing from Neighbors"
The Verdict
Multiple Inference
The Frequent Occurence of Extreme Events
Relation to Statistical Inference
The Bonferroni Inequality
Scheffe's Method
Example: MovieLens Data
The Verdict
Computational Complements
Movie Lens Data Wrangling
More Data Wrangling in the MovieLens Example
Mathematical Complements
Iterated Projections
Standard Errors for RFA
Asymptotic Chi-Square Distributions
Further Exploration: Data, Code and Math Problems
Chapter Eight
Shrinkage Estimators
Relevance of James-Stein to Regression Estimation
Multicollinearity
What's All the Fuss About?
A Simple Guiding Model
Wrong" Signs in Estimated Coefficients
Checking for Multicollinearity
The Variance Ination Factor
Example: Currency Data
What Can/Should One Do?
Do Nothing
Eliminate Some Predictors
Employ a Shrinkage Method
Ridge Regression
Alternate Definitions
Yes, It Is Smaller
Choosing the Value of _
Example: Currency Data
The LASSO
Definition
The lars Package
Example: Currency Data
The Elastic Net
Cases of Exact Multicollinearity, Including p > n
Why It May Work
Example: R mtcars Data
Additional Motivation for the Elastic Net
Bias, Standard Errors and Significance Tests
Generalized Linear Models
Example: Vertebrae Data
Other Terminology
Further Reading
Mathematical Complements
James-Stein Theory
Definition
Theoretical Properties
When Might Shrunken Estimators Be Helpful?
Ridge Action Increases Eigenvalues
Computational Complements
Code for ridgelm()
Further Exploration: Data, Code and Math Problems
Chapter Nine
Variable Selection and Dimension Reduction
A Closer Look at Under/Overfitting
A Simple Guiding Example
How Many Is Too Many?
Fit Criteria
Some Common Measures
No Panacea!
Variable Selection Methods
Simple Use of p-Values: Pitfalls
Asking \What If" Questions
Stepwise Selection
Basic Notion
Forward vs Backward Selection
R Functions for Stepwise Regression
Example: Bodyfat Data
Classification Settings
Example: Bank Marketing Data
Example: Vertebrae Data
Nonparametric Settings
Is Dimension Reduction Important in the
Nonparametric Setting?
The LASSO
Why the LASSO Often Performs Subsetting
Example: Bodyfat Data
Post-Selection Inference
Direct Methods for Dimension Reduction
Informal Nature
Role in Regression Analysis
PCA
Issues
Example: Bodyfat Data
Example: Instructor Evaluations
Nonnegative Matrix Factorization (NMF)
Overview
Interpretation
Sum-of-Parts Property
Example: Spam Detection
Use of freqparcoord for Dimension Reduction
Example: Student Evaluations of Instructors
Dimension Reduction for Dummy/R Factor
Variables
The Verdict
Further Reading
Computational Complements
Computation for NMF
Mathematical Complements
MSEs for the Simple Example
Further Exploration: Data, Code and Math Problems
Chapter Ten
Partition-Based Methods
CART
Example: Vertebral Column Data
Technical Details
Statistical Consistency
Tuning Parameters
Random Forests
Bagging
Example: Vertebrae Data
Example: Letter Recognition
Other Implementations of CART
Further Exploration: Data, Code and Math Problems
Chapter Eleven
Semi-Linear Methods
k-NN with Linear Smoothing
Extrapolation Via lm()
Multicollinearity Issues
Example: Bodyfat Data
Tuning Parameter
Linear Approximation of Class Boundaries
SVMs
Geometric Motivation
Reduced Convex Hulls
Tuning Parameter
Nonlinear Boundaries
Statistical Consistency
Example: Letter Recognition Data
Neural Networks
Example: Vertebrae Data
Tuning Parameters and Other Technical Details
Dimension Reduction
Statistical Consistency
The Verdict
Mathematical Complements
Edge Bias with k-NN and Kernel Methods
Dual Formulation for SVM
The Kernel Trick
Further Reading
Further Exploration: Data, Code and Math Problems
Chapter Twelve
Regression and Classification in Big Data
Solving the Big-n Problem
Software Alchemy
Example: Flight Delay Data
More on the Insufficient Memory Issue
Deceivingly Big- n
The Independence Assumption in Big-n Data
Addressing Big-p
How Many Is Too Many?
Toy Model
Results from the Research Literature
A Much Simpler and More Direct Approach
Nonparametric Case
The Curse of Dimensionality
Example: Currency Data
Example: Quiz Documents
The Verdict
Mathematical Complements
Speedup from Software Alchemy
Computational Complements
The partools Package
Use of the tm Package
Further Exploration: Data, Code and Math Problems
A Matrix Algebra
A Terminology and Notation
A Matrix Addition and Multiplication
A Matrix Transpose
A Linear Independence
A Matrix Inverse
A Eigenvalues and Eigenvectors
A Rank of a Matrix
A Matrices of the Form B'B
A Partitioned Matrices
A Matrix Derivatives
A Matrix Algebra in R
A Further Reading