Reinforcement learning in environments that keep changing

It started with games. Playing FIFA, I kept wondering how the opposing player knows when to tackle me. In GTA, how do characters respond like real people instead of running a fixed script? The outcome was never fixed. It changed based on what I did. I could not stop thinking about that.

When I found reinforcement learning, something clicked. An agent learns by acting, failing, and adjusting. No labelled dataset tells you how to play a match. You play, you lose the ball, you adjust. It felt like the most natural thing in the world, because we do exactly that.

This site has an interactive gridworld where you can watch that loop play out: an agent starts out knowing nothing, wanders randomly, and over a few hundred episodes converges to the shortest path. But even in that tiny world, a limit is easy to see. The grid never changes. Once the agent has learned, the problem is over.

When the environment will not hold still

The question that actually interests me starts where that demo ends. What happens when the environment does not stay still? How do learning-based systems adapt when the dynamics shift under them, and how do they maintain accuracy while adapting? Most of what makes RL hard outside of toy problems seems to live in that gap.

Why HVAC makes a good testbed

I wanted a concrete setting where decisions are sequential and the environment genuinely refuses to be static. HVAC energy control fits. Modern infrastructure consumes a large share of global energy, and heating, ventilation, and air conditioning systems are heavily responsible for it. Control here is a sequential decision problem: because of a building's thermal inertia, each action affects future thermal states, not just the present one. It models naturally as a Markov Decision Process. The state captures current conditions such as temperature, time, weather, and occupancy. Actions are control adjustments. The reward trades energy use against comfort. And the environment keeps shifting, with weather, seasons, and occupancy patterns.

Recent research, mostly through simulation-based studies, has shown RL control strategies behaving more adaptively than rule-based approaches. The same studies also show sensitivity to dynamic environmental changes: agents that look strong in one regime degrade when conditions drift. To me that failure mode is the interesting part, not a footnote.

The approach I am working toward

So the direction I am working toward is deliberately modest. Not system optimization, and not chasing state of the art. Simple RL-based methods in a realistic but controlled simulated environment, compared honestly against rule-based baselines, with attention on learning dynamics, failure cases, and practical limitations. I want to understand how the model behaves before trusting what it reports.

I am an undergraduate building formal grounding, and this page will change as my understanding does. Careful small studies produce data that can help future real-world applications, and that is enough reason to do them well.