This lecture section is a cumulative review of material from the semester and is meant to serve as a study guide for students preparing for the upcoming final exam. Topics start at modeling fundamentals (what is the purpose of a model in general) to the specifics of designing statistical experiments with stochastic simulations.

IEE 475: Simulating Stochastic Systems

Lecture M (2020-12-01): Final Exam Review

 In this wrap-up lecture, we finish the treatment of Variance Reduction Techniques (VRT's) for stochastic simulation. We cover (or review) Common Random Numbers (CRN's), Control Variates (CV's), Antithetic Variates (AV's), and Importance Sampling. The lecture ends with some brief big-picture comments about the lifelong learning process of simulation modeling.

Lecture L (2020-11-24): Course Wrap-Up

In this lecture, we review different forms of Variance Reduction Techniques (VRT's) for stochastic simulation, which attempt to re-design simulation experiments to control for sources of variance and thus increase statistical power when making an estimate with a small number of replications. We start with common random numbers (CRN's) and Control Variates. We then pivot to discussing Antithetic Variates. The goal was to also cover Importance Sampling, but due to time constraints that topic will be saved for the next lecture period.

Lecture K2 (2020-11-19): Variance Reduction Techniques, Part 2 - AVs and Importance Sampling

This lecture primarily finishes the coverage of estimation of relative performance by walking through the three different 2-sample mean tests (paired-difference t test, pooled-variance t-test, and Welch's unpooled-variance t-test) and the assumptions required to use them. Confidence intervals for each of the mean differences are defined, requiring formulas for standard error of the mean and degrees of freedom for each of the three experimental cases. We also briefly discuss how to extend this to more than 2 systems (with ANOVA's and post-hoc tests), but due to time we shift into an introduction of the related topic of Variance Reduction Techniques (VRT's). We introduce common random numbers (CRN's), but the full discussion of CRN's and Control Variates will be saved until the next lecture.

Lecture K1 (2020-11-17): Variance Reduction Techniques, Part 1 - CRN's and Control Variates

In this lecture, we move from estimation of absolute performance from simulation studies to estimation of relative performance. We start with connecting confidence intervals with linear regression, as an alternative application of one-sample confidence intervals. We review the use of one-sample confidence intervals for relative performance estimation, and then we pivot to discussing the visualization of 2-sample tests with confidence intervals.

Lecture J4 (2020-11-12): Estimation of Relative Performance

This lecture continues to discuss issues related to estimating absolute performance from transient and steady-state simulations (of terminating and non-terminating systems, respectively). We continue to emphasize the importance and utility of interval estimations (over point estimates). We then move on to discuss experimental methodologies useful for steady-state simulations, particularly related to eliminating estimator bias and reducing computational time.

Lecture J3 (2020-11-10): Estimation of Absolute Performance, Part 3

In this lecture, we continue our discussion of the use of performance estimation strategies for absolute performance (particularly in the case of transient simulation models of terminating systems). We review the sources of variation within and across replications in a simulation study, followed by a definition of common point estimators (for mean as well as quantile estimation), and then we define measures of estimator variance (e.g., standard error of the mean, SEM). That allows us to introduce interval estimation, particularly confidence interval estimation, which represents the interval of hypotheses that would fail to be rejected by a one-sample t-test. We discuss interval estimation in terms of quantile estimation as well. Finally, we conclude with a method of using constraints on interval half width to guide how many simulation replications are needed for a given simulation study.

Lecture J2 (2020-10-05): Estimation of Absolute Performance, Part 2

In this lecture, we continue to discuss hypothesis testing -- introducing parametric, non-parametric, exact, and non-exact tests and reviewing the assumptions behind many popular parameterized tests (like the t-test and ANOVA) and non-exact tests (Chi-square test). We then move to discuss the multiple-comparisons problem ("fishing") and Bonferroni correct. We end with an introduction to the estimation of absolute performance in simulated systems.

Lecture J1 (2020-11-03): Estimation of Absolute Performance, Part 1

 In this lecture, we review the basics of hypothesis testing (type-I error, type-II error, statistical power) and the fundamental processes underlying hypothesis testing that create relationships among these things. We then dig deeper into the assumptions necessary for using parametric tests, like the Student's t-test, and non-exact parametric tests, like the Chi-square test (e.g., what the "continuity assumption" is with regard to the Chi-square test and the related inference).

Lecture I (2020-10-29): Statistical Reflections

In this lecture, we revisit some basics of hypothesis testing and then go on to introduce verification, validation, and calibration in the context of simulation models. This will ultimately move us away from goodness-of-fit tests of input models toward hypothesis tests of output performance (e.g., to detect differences from different simulations scenarios and confirm that simulations of real-world scenarios match our expectations from real-world data).

Lecture H (2020-10-27): Verification, Validation, and Calibration of Simulation Models

In this lecture, we continue our discussion of statistically rigorous methods for input modeling in simulation of stochastic systems. We first cover the basics of hypothesis testing, including a review of type-I error (alpha), p-values, and how they relate to critical values for goodness-of-fit tests (like Chi-squared and KS). We then review Q-Q and P-P probability plots to identify candidate families for input models from collected data. Then we discuss how maximum likelihood estimation (MLE) provides a bridge from summary statistics to mathematically justifiable choices for parameter values of the distributions we have chosen. Next time, we will discuss Chi-square testing and KS testing as applied to general probability distributions (i.e., not just as tests for uniformity).

Lecture G3 (2020-10-22): Input Modeling, Part 3

This is the second part in a unit on input modeling for simulating stochastic systems (stochastic simulation). In the this part, we describe how to start making sense of data collected from real-world systems. We start with an example that builds a model of a single-server, single-channel queue based on summary statistics alone and demonstrate that the resulting model is a poor fit for a realistic system. We then use a histogram to reveal insights into how the system can be re-structured to be more realistic while also requiring simpler input models. This leads into a discussion on building histograms to be maximally insightful.

Lecture G2 (2020-10-20): Input Modeling, Part 2

In this lecture period, we discuss student-generated questions as a means of reviewing for the upcoming midterm retake. Most of the discussion centers around the solution set from the first midterm.

Lecture F2 (2020-10-13): Review Before Midterm Retake

In this lecture, we re-motivate the topic of Input Modeling in stochastic simulation. Input modeling is the process of choosing probabilistic models to represent realistic variation that is statistically similar to measured data even though the probabilistic models leave out the real-world details underlying that variation. This lecture focus primarily on issues relating to collecting data (when to use old data, issues related to data censoring, issues related to correlated variables, issues related to multi-modal distributions, etc.).

Lecture G1 (2020-10-08): Input Modeling, Part 1

Midterm review lecture for Fall 2020. Starts with an introduction to inverse-transform sampling for discrete random variables (including sampling from empirical CDF's).

Lecture F (2020-10-01): Midterm Review

Archived lectures from undergraduate course on stochastic simulation given at Arizona State University by Ted Pavlic

hubhopper

IEE 475: Simulating Stochastic Systems

hubhopper