The bootstrap and permutation tests offer ways to help students better understand concepts such as sampling distributions, standard errors, confidence intervals, P-values, and statistical significance.
Here are notes about books and software for teaching using resampling.
What Teachers Should Know about the Bootstrap: Resampling in the Undergraduate Statistics Curriculum
Tim Hesterberg (2015), What Teachers Should Know about the Bootstrap: Resampling in the Undergraduate Statistics Curriculum, The American Statistician 69(4) 371-386, DOI: 10.1080/00031305.2015.1089789
Bootstrapping has enormous potential in statistics education and practice, but there are subtle issues and ways to go wrong. For example, the common combination of nonparametric bootstrapping and bootstrap percentile confidence intervals is less accurate than using t-intervals for small samples, though more accurate for larger samples. My goals in this article are to provide a deeper understanding of bootstrap methods—how they work, when they work or not, and which methods work better—and to highlight pedagogical issues. Supplementary materials for this article are available online.
Original (longer) version:
Mathematical Statistics with Resampling and R by Laura Chihara and Tim Hesterberg (Wiley, 2011) uses permutation tests and bootstrapping to introduce these concepts and to motivate more classical mathematical approaches.
For more information, see
Resampling helps students understand the meaning of sampling distributions, sampling variability, P-values, hypothesis tests, and confidence intervals. This groundbreaking book shows how to apply modern resampling techniques to mathematical statistics. Extensively class-tested to ensure an accessible presentation, Mathematical Statistics with Resampling and R utilizes the powerful and flexible computer language R to underscore the significance and benefits of modern resampling techniques.
The book begins by introducing permutation tests and bootstrap methods, motivating classical inference methods. Striking a balance between theory, computing, and applications, the authors explore additional topics such as:
Mathematical Statistics with Resampling and R is an excellent book for courses on mathematical statistics at the upper-undergraduate and graduate levels. It also serves as a valuable reference for applied statisticians working in the areas of business, economics, biostatistics, and public health who utilize resampling methods in their everyday work.
The bootstrap and permutation tests offer ways to help students
better understand concepts such as sampling distributions, standard
errors, confidence intervals, and P-values.
The publisher's page is here. See the related note about StatKey below.
More formally, this is Statistics: Unlocking the Power of Data, Robin H. Lock, Patti Frazer Lock, Kari Lock Morgan, Eric F. Lock, Dennis F. Lock, Wiley 2012.
This is on OpenIntro.
This intro stat book uses randomization tests (permutation tests) to introduce hypothesis testing. The treatment of the bootstrap in the first edition is lacking-they find that the bootstrap percentile interval is poor in small samples (true), and don't go deeper. See my 2014 arXiv article below for a deeper discussion.
More formally, this is Introductory Statistics with Randomization and Simulation, David M. Diez, Christopher D. Barr, and Mine Cetinkaya-Rundel, 1st edition, 2014.
There are different versions of BMPT, written as supplemental chapters for two different books, but all can be used independently as an introduction to bootstrap methods and permutation tests.
The first version ("BMPT/PBS") is a supplemental chapter for The Practice of Business Statistics: Using Data for Decisions by Moore, McCabe, Duckworth and Sclove. This is available from W. H. Freeman, ISBN 0-7167-5726-5 for about $7, or is available at http://bcs.whfreeman.com/pbs/cat_160/PBS18.pdf.
The second version ("BMPT/IPS5e") is a supplemental chapter for Introduction to the Practice of Statistics, 5th Edition by Moore and McCabe. This is available at http://bcs.whfreeman.com/ips5e/content/cat_080/pdf/moore14.pdf. See also http://www.whfreeman.com/ipsresample.
The third version ("BMPT/IPS6e") is a supplemental chapter for Introduction to the Practice of Statistics, 6th Edition by Moore, McCabe and Craig. This is available at http://bcs.whfreeman.com/ips6e/content/cat_040/pdf/ips6e_chapter16.pdf. See also the IPS 6e Resampling page.
The fourth version ("BMPT/IPS7e") is a supplemental chapter for Introduction to the Practice of Statistics, 7th Edition by Moore, McCabe and Craig. This is available at http://content.bfwpub.com/webroot_pubcontent/Content/BCS_4/IPS7e/Student/Companion%20Chapters/ips_chap16.pdf. See also the IPS 7e Resampling page.
For BMPT/PBS download PBSdata.zip.
For BMPT/IPS5e download IPSdata.zip.
For BMPT/IPS6e download IPSdata6.zip.
Download the appropriate package, unzip, then follow instructions in INSTALL.txt.
To use these packages, you need S+ and S+Resample, see below.
For a general introduction to S+, see the S-PLUS Guide for Moore and McCabe's Introduction to the Practice of Statistics,Fifth Edition. This works best with the IPS5e version of the data package.
The Lock^5 team have developed web apps to encourage the use of simulation methods (e.g. bootstraps intervals and randomization tests) to help students in introductory statistics courses understand the basic ideas of statistical inference. The result, called StatKey, is now freely available at http://lock5stat.com/statkey.
I've seen a demo, this could be very useful, with or without their book.
There are procedures for generating bootstrap distributions for a mean, median, standard deviation, proportion, difference in means, difference in proportions, slope, and correlation as well as constructing randomization distributions to test hypotheses about most of the same parameters.
In each of these situations students see a representation of the original sample, individual bootstrap/randomization samples, and a summary dotplot of the results for lots of simulated samples. Students can easily interact with the bootstrap or randomization distribution to find summary statistics, find percentiles, or check tail probabilities.
The bootstrap package is smallest, the boot package offers the most analytical capabilities, and the resample package is easiest to use. The R version of resample is a partial copy of the S+ version, but I'll add to it over time. The S+ version includes a menu interface, and offers some capabilities not in the other packages. For a quick comparison of all, see bootstrapComparison.txt. For a comparison of ease of use of boot and resample, see resamplePoster1407.pdf.bootstrap-short-course. I have given this course in various formats, ranging from a two-day hands-on course to half-day lecture-only, public or private, in Albuquerque, Boston, Chicago, Cincinnati, L.A., Little Rock, Miami, Minneapolis, Portland, Rochester MN, San Francisco, Washington D.C., Basel, Basingstoke UK, Bedford UK, London, Manchester, Montpellier FR, Toronto, and Zurich.
Since I am no longer at Insightful (now Tibco) I won't give this course as frequently. Contact me if you are interested in arranging a course.
For other articles (including references to published articles related to this software) see articles/