340 Pages 23 B/W Illustrations
    by CRC Press

    Clear, rigorous, and intuitive, Markov Processes provides a bridge from an undergraduate probability course to a course in stochastic processes and also as a reference for those that want to see detailed proofs of the theorems of Markov processes. It contains copious computational examples that motivate and illustrate the theorems. The text is designed to be understandable to students who have taken an undergraduate probability course without needing an instructor to fill in any gaps.

    The book begins with a review of basic probability, then covers the case of finite state, discrete time Markov processes. Building on this, the text deals with the discrete time, infinite state case and provides background for continuous Markov processes with exponential random variables and Poisson processes. It presents continuous Markov processes which include the basic material of Kolmogorov’s equations, infinitesimal generators, and explosions. The book concludes with coverage of both discrete and continuous reversible Markov chains.

    While Markov processes are touched on in probability courses, this book offers the opportunity to concentrate on the topic when additional study is required. It discusses how Markov processes are applied in a number of fields, including economics, physics, and mathematical biology. The book fills the gap between a calculus based probability course, normally taken as an upper level undergraduate course, and a course in stochastic processes, which is typically a graduate course.

    Review of Probability
    Short History
    Review of Basic Probability Definitions
    Some Common Probability Distributions
    Properties of a Probability Distribution
    Properties of the Expected Value
    Expected Value of a Random Variable with Common Distributions
    Generating Functions
    Moment Generating Functions
    Exercises
    Discrete-Time, Finite-State Markov Chains
    Introduction
    Notation
    Transition Matrices
    Directed Graphs: Examples of Markov Chains
    Random Walk with Reflecting Boundaries
    Gambler’s Ruin
    Ehrenfest Model
    Central Problem of Markov Chains
    Condition to Ensure a Unique Equilibrium State
    Finding the Equilibrium State
    Transient and Recurrent States
    Indicator Functions
    Perron-Frobenius Theorem
    Absorbing Markov Chains
    Mean First Passage Time
    Mean Recurrence Time and the Equilibrium State
    Fundamental Matrix for Regular Markov Chains
    Dividing a Markov Chain into Equivalence Classes
    Periodic Markov Chains
    Reducible Markov Chains
    Summary
    Exercises
    Discrete-Time, Infinite-State Markov Chains
    Renewal Processes
    Delayed Renewal Processes
    Equilibrium State for Countable Markov Chains
    Physical Interpretation of the Equilibrium State
    Null Recurrent versus Positive Recurrent States
    Difference Equations
    Branching Processes
    Random Walk in
    Exercises
    Exponential Distribution and Poisson Process
    Continuous Random Variables
    Cumulative Distribution Function (Continuous Case)
    Exponential Distribution
    o(h) Functions
    Exponential Distribution as a Model for Arrivals
    Memoryless Random Variables
    Poisson Process
    Poisson Processes with Occurrences of Two Types
    Exercises
    Continuous-Time Markov Chains
    Introduction
    Generators of Continuous Markov Chains: The Kolmogorov Forward and Backward Equations
    Connection Between the Steady State of a Continuous Markov Chain and the Steady State of the Embedded Matrix
    Explosions
    Birth and Birth-Death Processes
    Birth and Death Processes
    Queuing Models
    Detailed Balance Equations
    Exercises
    Reversible Markov Chains
    Random Walks on Weighted Graphs
    Discrete-Time Birth-Death Process as a Reversible Markov Chain
    Continuous-Time Reversible Markov Chains
    Exercises
    Bibliography

    Biography

    James R. Kirkwood

    "All chapters are followed by exercises that render this text-book attractive for teachers…"
    Zentralblatt MATH

    "Kirkwood…has published another significant mathematics monograph."

    "Suitable for audiences who strive to grasp the fundamental concepts of various types of Markov processes or to prepare for learning advanced stochastic processes… the monograph can serve as a textbook since it provides essential examples and exercise problems applied in economics, finance, engineering, physics, and biology."  
    —S-T. Kim, North Carolina A&T State University