CHAPTER 10

Stochastic Processes

Study systems that evolve randomly through time and connect probability with dynamic mathematical models.

\[ \text{random variable} \rightarrow \text{random process} \rightarrow \text{trajectory} \]

STOCHASTIC PROCESSES

Random systems evolving through time

What Is a Stochastic Process?

Understand a family of random variables indexed by time or another parameter.

State Spaces

Describe the possible states that a stochastic system can occupy.

Discrete and Continuous Time

Distinguish processes observed at fixed time steps from processes evolving continuously.

Random Trajectories

Understand one realised path of a stochastic process through time.

The Markov Property

Understand when the future depends on the present state but not the full past history.

Discrete-Time Markov Chains

Study Markov processes that move between states at fixed time steps.

Transition Matrices

Represent probabilities of moving between states using a matrix.

Probability Vectors

Represent the probability distribution across all possible states.

Continuous-Time Markov Chains

Study Markov processes where events occur at random continuous times.

Transition Rates

Describe how quickly a continuous-time process moves between states.

Generator Matrices

Represent transition rates and the infinitesimal behaviour of a CTMC.

Waiting Times

Understand exponential waiting times between events in continuous-time processes.

Gillespie Simulation

Simulate one exact trajectory of a continuous-time Markov process.

The Poisson Process

Study a basic counting process with random event times and constant rate.

Brownian Motion

Introduce a continuous stochastic process with normally distributed independent increments.

The central idea

\[ \{X_t:t\ge0\} \]

A stochastic process does not describe one deterministic path. It describes a collection of possible random trajectories together with their probability structure.