STOCHASTIC PROCESSES 09
Continuous-Time Markov Chains
A CTMC moves between discrete states at random times while satisfying the Markov property.
The process
A CTMC is written
\[\{X_t:t\ge0\}\]with a discrete state space and continuous time.
1. Random event times
Unlike a DTMC, the process does not wait for fixed time steps. It remains in a state for a random waiting time and then jumps.
2. Transition rates
For \(i\ne j\), let \(q_{ij}\) be the rate of moving from state \(i\) to state \(j\). For a small interval \(\Delta t\),
3. Probability of no jump
If the total rate out of state \(i\) is
\[a_i=\sum_{j\ne i}q_{ij},\]then for small \(\Delta t\),
\[P(\text{no jump})=1-a_i\Delta t+o(\Delta t).\]4. Waiting time
The time until the next jump is exponentially distributed:
\[\tau\sim\operatorname{Exp}(a_i).\]5. Which jump occurs?
Given that a jump occurs from state \(i\),
\[P(i\to j\mid\text{a jump})=\frac{q_{ij}}{a_i}.\]6. Epidemic example
In an SIS model with \(i\) infectious individuals, an infection can move \(i\to i+1\) and a recovery can move \(i\to i-1\), each with its own state-dependent rate.