A Comparison of CV and WAIC
Sumio Watanabe Homepage
Key words: Bayesian statistics, cross validation, information criterion
Let us compare cross validation with WAIC.
In general, information criteria will be evaluated from bias and variance.
Information criteria should be compared with the generalization error.
(1) First, we compare means and standard deviations of CV, WAIC, and DIC.
MATLAB program (1)
(2) Second, we compare generalization errors by models selectoions by CV, WAIC, and DIC.
MATLAB program (2)
Cross validation can be applied to almost all statistical problems,
but it seems that it has relatively large variance.
If a leverage sample is contained in the set of training samples, then
the variance of importance sampling cross validation becomes infinity.