Math 632 - Introduction to Stochastic Processes

Fall 2014

Meetings: TuTh 9:30AM - 10:45AM, VAN VLECK B115
Instructor: Benedek Valkó
Office: 409 Van Vleck
Phone: 263-2782
Email: valko at math dot wisc dot edu
Office hours: Tu 11-12 or by appointment

This is the course homepage that also serves as the syllabus for the course. Here you will find homework assignments, our weekly schedule, and updates on scheduling matters. The Mathematics Department also has a general information page on this course.

I will use the class email list to send out corrections, announcements, please check your email from time to time.

Important deadlines.

Course description

Math 632 is a course on basic stochastic processes and applications with an emphasis on problem solving. Topics will include  discrete-time Markov chains, Poisson point processes, continuous-time Markov chains, and renewal processes. The material will be treated in a mathematically precise fashion although some proofs will be skipped due to time limitations.

To go beyond Math 632 in probability you should consider taking Math 635 - Introduction to Brownian Motion and Stochastic Calculus.


 Durrett: Essentials of Stochastic Processes,  Springer, 2nd edition.
(a beta version of the second edition available on the author's website)

Note: the textbook uses scientific calculators in certain examples. We will skip these parts

Other textbooks which could be used for supplemental reading:


Good knowledge of undergraduate probability at the level of UW-Madison Math 431 (or an equivalent course: Math 331, Stat 309, 311, 313) is required. This means familiarity with basic probability models, random variables and their probability mass functions and distributions, expectations, joint distributions, independence, conditional probability and expectation, the law of large numbers and the central limit theorem.

If you need a thorough review of basics, the textbook A First Course in Probability by S. Ross is recommended. I will also upload a set of lecture notes on probability at Learn@UW which is currently being developed for Math 431. For those, who just need a quick refresher, this write-up summarizes some of the basic concepts of probability theory you should be familiar with. I strongly suggest that you review this before the first class. The appendix of the textbook also contains an overview of the required theory.

Familiarity with calculus and linear algebra is also needed. Some 500-level work in mathematics is recommended for background, preferably in analysis (521).

Course content

The course consists of Chapters 1-4 of Durrett's book. The main topics covered in these chapters are discrete-time Markov chains, Poisson point processes, renewal theory and continuous-time Markov chains. We might discuss martingales and also give a brief introduction to Brownian motion, if time permits.
The list of covered topics will be updated after every week in the schedule below.


Course grades will be based on home work assignments (20%), two midterm exams (20% each) and the final exam (40%).
You will be allowed to use a single written page of notes on the midterm exams and two written pages on the final exam. Calculators and other electronic devices will not be permitted during the exams.
The midterm exams will be evening exams scheduled on Wednesdays.

Exam dates:

Midterm exams from previous years: math library exam reserves

The final grades will be determined according to the following scale:
A: [100,89), AB: [89,87), B: [87,76), BC: [76,74), C: [74,62), D: [62,50), F: [50,0].
There will be no curving in the class, but the instructor reserves the right to modify the final grade lines.


I will use the Learn@UW website of the course to post homework assignments, solutions and additional material.

How to succeed in this course

We will cover a lot of material during the course and some of you will find the pace of the lectures a bit fast. Try to read ahead in the textbook (I will post the relevant section numbers below), this way it will be a lot easier to follow the lectures.
The midterm and final exams will contain problems which will be similar to the homework problems in difficulty. The best way to prepare for these is to do as many practice problems from the book as you can. This will also help you understand the theory a lot better!
If you have trouble solving the homework (or practice) problems then come see me in my office hours (or set up an appointment).

Honors credit

If you are taking this course for an honors credit then you will need to do some extra work: this means reading some additional sections from the textbook and solving extra homework problems. Come and talk to me if you are interested


Homework assignments will be posted on the Learn@UW site of the course. Weekly homework assignments are due Thursdays at the beginning of the class.

Note that the first homework assignment is due on September 4 (the first Thursday of the semester!).  In order to solve the problems on the first assignment you will only need the concepts learned in Math 431 (or any other introductory probability course). If you are having a hard time with this assignment then you should definitely spend some time reviewing the this write-up.

Instructions for homework


Covered topics
Suggested reading for next week
Week 1
(9/2, 9/4)
Review of basic concepts of probability. Discrete Markov chains: definitions and examples,
the transition probability matrix, multistep probabilities,
posted notes, Sections 1.1-1.2
Sections 1.3-1.5
Week 2
(9/9, 9/11)
Classification of states, strong Markov property, transience and recurrence, closed and irreducible sets
Section 1.3
Sections 1.4-1.6
Week 3
(9/16, 9/18)
Stationary distributions, periodicity
Sections 1.4, 1.5
Sections 1.6-1.7
Week 4.
(9/23, 9/25)
Limit behavior, expected return time, examples, birth and death process, proofs of the limit theorems
Sections 1.5-1.7, 1.10
Sections 1.8-1.10
Week 5.
(9/30, 10/2)
Exit times and exit distributions,  the Metropolis-Hastings algorithm, extinction probability in a branching process
Sections 1.8, 1.9, 1.10
Sections 2.1-2.2
Week 6
(10/7, 10/9)
Properties of the exponential distribution, the Poisson process, number of arrivals in an interval, the spatial Poisson process
Sections 2.1, 2.2
Section 2.3-2.4
Week 7
(10/14, 10/16)
Compound Poisson process, transformations of Poisson processes
First midterm exam
Sections 2.3-2.4
Section 3.1
Week 8
(10/21, 10/23)
Finish up Poisson process. Renewal processes.
Sections 2.4, 3.1
Section 3.3
Week 9
(10/28, 10/30)
Age and residual life
Section 3.3
Section 4.1
Week 10
(11/4, 11/6)
Continuous time Markov chains, Markov property in continuous time, the embedded discrete time MC, infinitesimal rates
Section 4.1
Section 4.2
Week 11
(11/11, 11/13)
Examples, the Poisson clock construction, Kolmogorov's backward and forward equations
Section 4.1-4.2
Section 4.3
Week 12
(11/18, 11/20)
Solving the Kolmogorov equations, stationary distribution, limiting behavior
Second midterm exam
Section 4.3
Section 4.3
Week 13
Stationary distribution, detailed balance
Section 4.3
Section 4.4-4.5
Week 14
(12/2, 12/4)
Birth and death processes, exit times, queuing examples

Week 15
(12/9, 12/11)
Brief introduction to Brownian motion, review