Part 3.1 introduced Leonard Baum and the mathematical framework he helped build. This part slows down and looks inside it. Understanding roughly how a Hidden Markov Model works is what makes terms like "hidden state" and "inferred probability" mean something concrete, rather than just sounding technical.
The Problem in Plain Terms
Imagine you're trying to figure out whether it's raining outside, but
you're locked in a windowless room. You can't see the weather directly — that's
the "hidden" part. All you can observe is one thing: whether the
person who just walked in is carrying a wet umbrella.
A wet umbrella doesn't prove it's raining. Maybe they spilled
water on it. Maybe it stopped raining an hour ago and it just hasn't dried. But
across many people walking in over time, the pattern of wet versus dry
umbrellas gives you a way to estimate, probabilistically, what the actual
weather state most likely is — and to update that estimate continuously as new
people walk in.
That's the entire idea. There's a hidden state (the actual weather) you
can never observe directly. There are observable signals (umbrellas) that are
influenced by that hidden state but don't reveal it perfectly. And there's a
mathematical process for inferring the most probable hidden state, given a
stream of imperfect observations — while also allowing for the fact that the
hidden state itself can change over time.
Why "Markov" Matters
The Markov part of the name refers to a specific, useful simplifying
assumption: that the next hidden state depends mainly on the current
one, not on the entire distant history before it. Today's weather depends more
on yesterday's weather than on what happened a month ago. That assumption makes
the math tractable — you don't need infinite history to make a reasonable
inference about what's likely to happen next. You need a good read on the
current state, and a sense of how states tend to transition into one another.
Mapping This to Markets
Translate this directly onto price data. The "hidden state"
isn't the weather — it might be something like whether the market (or a
specific stock) is currently in a trending regime, a mean-reverting regime, a
high-volatility regime, or something else entirely. You can never see that
state labeled directly on a chart. All you can observe are price and volume —
the "umbrellas" — which are influenced by the hidden state but don't
announce it outright.
An HMM-style approach doesn't try to declare, with certainty, "the
market is now trending." It estimates the probability that the
market is in a trending state, given the recent pattern of observable data, and
continuously revises that probability as new price and volume information
arrives. It also incorporates the idea that regimes tend to persist for a while
once they start, and that transitions between regimes follow their own
statistical tendencies worth studying.
Why This Framing Is More Honest Than Most Technical
Analysis
Most chart-based analysis implicitly claims a kind of certainty it
hasn't earned — "this is a breakout," "this is a reversal"
— stated as fact rather than probability. The Hidden Markov framing is more
intellectually honest: it treats the market's current condition as
fundamentally unknowable in a direct sense, only inferable, and only ever
inferable with a confidence level attached, never certainty.
That's an uncomfortable way to think if you're used to conviction-based
trading. It's also, arguably, a far more accurate description of what's
actually knowable about markets at any given moment.
What This Means for Us
- Instead
of asking "what is the market doing right now," try asking
"what's the most probable state the market is in right now, given
what I can actually observe" — and hold that read loosely.
- Recognize
that your own indicators — moving averages, volume patterns, whatever you
use — are "umbrellas," not the weather itself. They're evidence,
not proof.
- Build
the habit of updating your read continuously as new data arrives, rather
than locking in a single interpretation and defending it until it's
obviously wrong.
This is the mathematical seed that ran through everything Renaissance
built afterward — not a market prediction machine, but a disciplined way of
estimating the unobservable from the observable, and never mistaking the
estimate for the truth.