A Comparison of CV and WAIC


Sumio Watanabe




Sumio Watanabe Homepage






Key words: Bayesian statistics, cross validation, information criterion



Let us compare cross validation with WAIC.




In general, information criteria will be evaluated from bias and variance.




Information criteria should be compared with the generalization error.




CV WAIC





(1) First, we compare means and standard deviations of CV, WAIC, and DIC.




MATLAB program (1)






CV WAIC





CV WAIC





(2) Second, we compare generalization errors by models selectoions by CV, WAIC, and DIC.




MATLAB program (2)





CV WAIC





CV WAIC





CV WAIC





CV WAIC





Cross validation can be applied to almost all statistical problems, but it seems that it has relatively large variance.

If a leverage sample is contained in the set of training samples, then the variance of importance sampling cross validation becomes infinity.