Thoughts and Theory

Image by Hans Braxmeier via Pixabay

Policies in Reinforcement Learning (RL) are shrouded in a certain mystique. Simply stated, a policy π: s →a is any function that returns a feasible action for a problem. No less, no more. For instance, you could simply take the first action that comes to mind, select an action at…


Photo by Jennifer Capel on Unsplash

The textbook cliff walking problem continues to fascinate. Although exceedingly simple, it elucidates many interesting aspects of reinforcement learning algorithms. After treating some value-based implementation (SARSA and Q-learning here, Deep Q-learning here), now it is time to move to a policy-based implementation. Although many libraries exist nowadays, we manually implement…


Photo by Mitchel Lensink on Unsplash

Data-driven decision-making — the sales pitch

Stating that data radically transforms the way businesses operate is hardly a revelation. Whether it is high-frequency sensor data, real-time stock market prices or detailed user logs — we track, collect and store data at a scale unprecedented in history. The reason: these vast piles of data hide value.


Photo by Ross Parmly on Unsplash

It is human nature to find explanations for the things we observe. Accepting that an event unfolds ‘just because’ is deeply unsatisfactory. We always look for a compelling reason, an explanatory variable, an underlying cause. Curiosity and a desire to understand the universe has propelled mankind since the Stone Age.


Artistic impression of a volatility smile? Photo by Caju Gomes on Unsplash

When the Black-Scholes option pricing model was introduced in 1973, it forever changed the financial world, causing an explosion in the trade of derivatives[1]. Although naturally a simplification of reality, it captured real-world option prices pretty well, and provided a mathematical underpinning for the mechanisms that governed option values.

The…


A business, to be recognized by the open office space that fosters collaboration, the cozy sitting area with a flat screen, and a snake plant. Photo by Austin Distel on Unsplash

Even those completely unfamiliar with Reinforcement Learning (RL) probably know how DeepMind’s AlphaGo managed to beat world champion Lee Sedol in the ancient board game Go. For many, RL has become synonymous with playing games, be it chess, Super Mario, or simply crossing a frozen lake. Games are fun, intuitive…


Thoughts and Theory

Photo by Nathan Dumlao on Unsplash

At first glance, moving from vanilla Q-learning to deep Q-learning seems like a minor step. Just replace the lookup table with a neural network and you’re done. There’s more to it than that though — even for the simplest of problems deep Q-learning might struggle to achieve results.

To show…


Photo by Chad Walton on Unsplash

If you believe Deep Q-learning is simply a matter of replacing a lookup table with a neural network, you might be in for a rough awakening. Although Deep Q-learning allows handling very large state spaces and complicated non-linear environments, these benefits come at a substantial cost.

For this article, I…


Photo by Magnet.me on Unsplash

Being a subdomain of Machine Learning, Reinforcement Learning (RL) is often likened to a black box. You try a couple of actions, feed the resulting observations into a neural network, and out roll some values — an esoteric policy telling you what to do in any given circumstance.

When traversing…


Photo by Science in HD on Unsplash

As every programmer knows, testing and debugging can be a nuisance. We want to spend our time building, creating, solving — not performing mind-numbing tests and staring at error messages. When adding teams and end-users to the mix, small mistakes can have catastrophic consequences. It is not enough for code…

Wouter van Heeswijk, PhD

Assistant professor in Financial Engineering and Operations Research. Writing about reinforcement learning, optimization problems, and data science.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store