# Math 632 - Introduction to Stochastic Processes

Fall 2014

Meetings: TuTh 9:30AM - 10:45AM, VAN VLECK B115
Instructor: Benedek Valkó
Office: 409 Van Vleck
Phone: 263-2782
Email: valko at math dot wisc dot edu
Office hours: Tu 11-12 or by appointment

This is the course homepage that also serves as the syllabus for the course. Here you will find homework assignments, our weekly schedule, and updates on scheduling matters. The Mathematics Department also has a general information page on this course.

I will use the class email list to send out corrections, announcements, please check your wisc.edu email from time to time.

### Course description

Math 632 is a course on basic stochastic processes and applications with an emphasis on problem solving. Topics will include  discrete-time Markov chains, Poisson point processes, continuous-time Markov chains, and renewal processes. The material will be treated in a mathematically precise fashion although some proofs will be skipped due to time limitations.

To go beyond Math 632 in probability you should consider taking Math 635 - Introduction to Brownian Motion and Stochastic Calculus.

### Textbook

Durrett: Essentials of Stochastic Processes,  Springer, 2nd edition.
(a beta version of the second edition available on the author's website)

Note: the textbook uses scientific calculators in certain examples. We will skip these parts

Other textbooks which could be used for supplemental reading:
• Greg Lawler: Introduction to Stochastic Processes, Chapman and Hall
• Sidney Resnick: Adventures in Stochastic Processes, Birkhäuser.
• Sheldon Ross: Stochastic Processes, Wiley
• Sheldon Ross: Introduction to Probability Models, Academic Press

### Prerequisites

Good knowledge of undergraduate probability at the level of UW-Madison Math 431 (or an equivalent course: Math 331, Stat 309, 311, 313) is required. This means familiarity with basic probability models, random variables and their probability mass functions and distributions, expectations, joint distributions, independence, conditional probability and expectation, the law of large numbers and the central limit theorem.

If you need a thorough review of basics, the textbook A First Course in Probability by S. Ross is recommended. I will also upload a set of lecture notes on probability at Learn@UW which is currently being developed for Math 431. For those, who just need a quick refresher, this write-up summarizes some of the basic concepts of probability theory you should be familiar with. I strongly suggest that you review this before the first class. The appendix of the textbook also contains an overview of the required theory.

Familiarity with calculus and linear algebra is also needed. Some 500-level work in mathematics is recommended for background, preferably in analysis (521).

### Course content

The course consists of Chapters 1-4 of Durrett's book. The main topics covered in these chapters are discrete-time Markov chains, Poisson point processes, renewal theory and continuous-time Markov chains. We might discuss martingales and also give a brief introduction to Brownian motion, if time permits.
The list of covered topics will be updated after every week in the schedule below.

### Evaluation

Course grades will be based on home work assignments (20%), two midterm exams (20% each) and the final exam (40%).
You will be allowed to use a single written page of notes on the midterm exams and two written pages on the final exam. Calculators and other electronic devices will not be permitted during the exams.
The midterm exams will be evening exams scheduled on Wednesdays.

Exam dates:

• First midterm: Oct 15, 5:30PM-7PM, B302 Birge Hall
• Second midterm: Nov 19, 5:30PM-7PM, B302 Birge Hall
• Final exam: Dec 16, 12:25PM-2:25PM,  B239 Van Vleck
Midterm exams from previous years: math library exam reserves

The final grades will be determined according to the following scale:
A: [100,89), AB: [89,87), B: [87,76), BC: [76,74), C: [74,62), D: [62,50), F: [50,0].
There will be no curving in the class, but the instructor reserves the right to modify the final grade lines.

### Learn@UW

I will use the Learn@UW website of the course to post homework assignments, solutions and additional material.

### How to succeed in this course

We will cover a lot of material during the course and some of you will find the pace of the lectures a bit fast. Try to read ahead in the textbook (I will post the relevant section numbers below), this way it will be a lot easier to follow the lectures.
The midterm and final exams will contain problems which will be similar to the homework problems in difficulty. The best way to prepare for these is to do as many practice problems from the book as you can. This will also help you understand the theory a lot better!
If you have trouble solving the homework (or practice) problems then come see me in my office hours (or set up an appointment).

### Honors credit

If you are taking this course for an honors credit then you will need to do some extra work: this means reading some additional sections from the textbook and solving extra homework problems. Come and talk to me if you are interested

### Homework

Homework assignments will be posted on the Learn@UW site of the course. Weekly homework assignments are due Thursdays at the beginning of the class.

Note that the first homework assignment is due on September 4 (the first Thursday of the semester!).  In order to solve the problems on the first assignment you will only need the concepts learned in Math 431 (or any other introductory probability course). If you are having a hard time with this assignment then you should definitely spend some time reviewing the this write-up.

### Instructions for homework

• Observe rules of academic integrity. Handing in plagiarized work, whether copied from a fellow student or off the web, is not acceptable. Plagiarism cases will lead to sanctions.
• Homework is collected at the beginning of the class period on the due date. No late papers will be accepted. You can bring the homework earlier to the instructor's office or mailbox. You can also email your solution in an electronic form, but it has to arrive by 9:30AM on the due date.
• Organize your work neatly. Use proper English. Write in complete English or mathematical sentences. Answers should be simplified as much as possible. If the answer is a simple fraction or expression, a decimal answers from a calculator is not necessary. But for some exercises you might need a calculator to get the final answer.
• Numerical answers alone carry no credit. It's all in the reasoning you write down.
• Put problems in the correct order and staple your pages together.
• Do not use paper torn out of a binder.
• Be neat. There should not be text crossed out.
• Recopy your problems. Do not hand in your rough draft or first attempt.
• Papers that are messy, disorganized or unreadable will not be graded.

### Schedule

 Covered topics Suggested reading for next week Week 1 (9/2, 9/4) Review of basic concepts of probability. Discrete Markov chains: definitions and examples, the transition probability matrix, multistep probabilities, posted notes, Sections 1.1-1.2 Sections 1.3-1.5 Week 2 (9/9, 9/11) Classification of states, strong Markov property, transience and recurrence, closed and irreducible sets Section 1.3 Sections 1.4-1.6 Week 3 (9/16, 9/18) Stationary distributions, periodicity Sections 1.4, 1.5 Sections 1.6-1.7 Week 4. (9/23, 9/25) Limit behavior, expected return time, examples, birth and death process, proofs of the limit theorems Sections 1.5-1.7, 1.10 Sections 1.8-1.10 Week 5. (9/30, 10/2) Exit times and exit distributions,  the Metropolis-Hastings algorithm, extinction probability in a branching process Sections 1.8, 1.9, 1.10 Sections 2.1-2.2 Week 6 (10/7, 10/9) Properties of the exponential distribution, the Poisson process, number of arrivals in an interval, the spatial Poisson process Sections 2.1, 2.2 Section 2.3-2.4 Week 7 (10/14, 10/16) Compound Poisson process, transformations of Poisson processes First midterm exam Sections 2.3-2.4 Section 3.1 Week 8 (10/21, 10/23) Finish up Poisson process. Renewal processes. Sections 2.4, 3.1 Section 3.3 Week 9 (10/28, 10/30) Age and residual life Section 3.3 Section 4.1 Week 10 (11/4, 11/6) Continuous time Markov chains, Markov property in continuous time, the embedded discrete time MC, infinitesimal rates Section 4.1 Section 4.2 Week 11 (11/11, 11/13) Examples, the Poisson clock construction, Kolmogorov's backward and forward equations Section 4.1-4.2 Section 4.3 Week 12 (11/18, 11/20) Solving the Kolmogorov equations, stationary distribution, limiting behavior Second midterm exam Section 4.3 Section 4.3 Week 13 (11/25) Stationary distribution, detailed balance Section 4.3 Section 4.4-4.5 Week 14 (12/2, 12/4) Birth and death processes, exit times, queuing examples Week 15 (12/9, 12/11) Brief introduction to Brownian motion, review