# Making state management intelligent

Canonical: https://davidk.dev/talks/making-state-management-intelligent-react-alicante-2023/
Video: https://www.youtube.com/watch?v=M70Va8usecc
Speaker: David Khourshid (@DavidKPiano), creator of XState
Type: Conference talk · Event: React Alicante 2023 · Year: 2023 · Duration: 31:54 · Views: 186 (as of 2026-10-19)
Topics: Adaptive & intelligent UIs, AI agents & LLMs, State management, State machines, XState

## Summary
If Pac-Man eats a power pill, then the ghost actually goes to a different state, which is running away from Pac-Man. Reinforcement learning is exciting to me because this represents how, or it forms a way of making state machines able to learn.

## Key takeaways (quoted from the transcript)
- If Pac-Man eats a power pill, then the ghost actually goes to a different state, which is running away from Pac-Man. (8:00)
- The goal of an agent in reinforcement learning is to maximize a future reward, just like hopefully what all of us are doing, so reinforcement learning described in the state machine. (10:00)
- You want to get to the reward state, and your dog is trying to find the most optimal path to get a reward. (16:00)
- First of all, one of the biggest things that you should realize is that LLMs are unpredictable, and that's why we use, you know, state machines to help. (24:00)
- Declarative state management, using events and having finite states really makes your app more, capable. (30:00)

## Chapters
- 0:00 Introduction
- 2:00 When I used to give this talk earlier…
- 4:00 I, I know, solid framework has been mentioned a bunch at…
- 7:00 The goal of an agent in reinforcement learning is to maximize…
- 24:00 Speaking of LLMs is that, hopefully it's here is that we're…
- 29:00 An app for ordering coffee…

## Transcript
[0:00] All right, so my name is David Khourshid and I'm going to be talking about making state Management's intelligent. And first of all, I want a quick buy check. Are we in the Siesta state? are we in the DiSpirito state? Which one are we? RICARDO Okay, good, maybe the CS does. Everyone outside right now and inside, all right. I just want to make sure I want to make this exciting and fun. We're going to be covering a lot you don't need to memorize at all.

[0:00] I'm not assigning homework or a test or anything, we're just going to be exploring some really fun concepts. So, first of all, my name is David Khourshid. Like I said, I work at Stately..ai and we're going to be talking about two things, basically, state and AI. And so we're going to see how these make a really fun combination. Now first, when we talk about states, we are at a React conference, there's many ways of doing states.

[1:00] There's you state useReducer, my favorite useEffect, my least favorite. If you went to my talk last year, uh. Use context and UseyNC external store, which should be used more often. But is really a weird sort of hook. And how many of you actually just mostly use date all the time? Okay, this is your fault. Okay, you did this. You should be ashamed of yourself now. Besides that, there's so many ways to manage.

[1:00] States in React, too. Personally. At Stately, we maintain the XD library and that's what we're going to be going over. But one of the main points that I want you to take away from this is the idea of declarative states and events. And how just thinking in that can move your app towards a way where it could be made more intelligent. And so I'm going to show you exactly how we do that, because nowadays users want more intelligent apps.

[2:00] When I used to give this talk earlier, I was showing demos of ChatGPT.

[2:00] But now we have made so many advancements in what you could do with AI, especially in web development. There is for cell V0, I don't have access yet. I really want it, but basically in B0, you could just type in. I want a component for this, like a slider. Or something like that. And also, ChatGPT just released ChatGPT vision, which is rolling out to new users. I don't tap it yet, but in this video you could see that someone, basically, discovered his first state machine.

[2:00] He drew arrows on a whiteboard and he took a picture of it, sent it to ChatGPT. And it was able to make a simple app flow just from the diagram that he drew on the whiteboard. He probably doesn't know what's called a state machine, but it's great for him to see that. Now. A lot of companies are starting up doing AI. And we know that making API calls that are slow, expensive, sometimes wrong and impossible to debug is sub-optimal and bad coding practice.

[3:00] But if you use LLMs, it's called AI. And now your company is valued at 100 million dollars, and that tends to be the state of things today, and so I want you all to have 100 million dollar companies. So that's why we're going to be adding AI to our code. But as we know, LLNS and using chat Bots and OpenAI like, it's not enough. We need to do more than just chat Bots.

[3:00] More than just talk to a text interface and have it spit out some more text. For us, this is the direction that AI apps should be going in, where it's not just not just ChatGPT. Placed in a nice wrapper. And there's many really good articles on that very subject. Because nowadays, the way that many apps are incorporating AI into their products is more of an input output thing.

[4:00] You ask it a question, it gives you an answer, you tell it something, and it gives you something back, like a summary or description or something like that. The kind of AI that I'm more interested is AI that actually does something for you, where it executes multiple steps and eventually gets you towards a goal. And this is actually what made me start my interest in AI. Because, I, I was looking at things like, which I'll show you in the next slide.

[4:00] But there's. There's things that are sort of incorporating LLMs into these agents type interfaces, such as agent GPT, link chain and auto GPT, and these agents go beyond. You know, just normal. Chat AI and they actually perform tasks to accomplish a goal. So, anyway, I was gonna say that I. I saw, you know, lots of demos and read a bunch of papers on just this concept of basically using agents to actually accomplish things.

[5:00] And this goes beyond just LLMs, like, for example, in this mastering Atari with discrete world models, google research was able to train using deep reinforcement learning agents that are able to play games. Pretty much as good as human players. And the previous state of the art was not very good at this, it didn't do as well as human players. But what these agents did was form a world model in an understanding of how the game mechanics worked.

[5:00] So, just like a human would do, and then there's actual mechanic AI, where, obviously, you shouldn't be pushing robots around once a game sentience. That could be really, really bad for us. But basically, these robots also use deep reinforcement learning to mimic behaviors and to achieve literal goals. So lots of really, really cool examples here, all right. So how do we create intelligent agents, or at least how do we get started with it?

[6:00] In my opinion, and I'm a bit biased, but I think that there's three things that we need now. Obviously, there's many ways to create agents, but I really like these combinations of things. Because, it provides a very solid framework. I, I know, solid framework has been mentioned a bunch at L.A. County, but, you know what I mean, like, a solid framework for, creating these agents. First of all, it's state machines, and these provide determinism and explainability.

[6:00] You have a graph which I'm going to show you, and it's just like a map of the world where you know exactly which paths you can take. Then there's reinforcement learning, and this is what allows state machines to actually learn in a way, using exploration and exploitation in order to optimize for a reward. And then there's large language models, which provides that interpretation of natural language and creativity.

[7:00] And so we're going to start with state machines, we're going to cover these three things state machines I feel like, are the original AI. When you play games such as Pac-Man, things like Enemy behavior or Opponent's computer. Behavior is typically modeled with something very similar to a state machine. So if it's not a state machine, it's either a bunch of IF statements or it's something like a behavior tree or similar. And so one of my favorite examples is the enemy behavior in Pac-Man.

[7:00] The ghosts, actually, this is something I recently learned. Each of the ghosts actually has a slightly different behavior. So Blinky will chase Pac-Man directly, Pinky will try to predict where Pac-Man is going to go and intercepts Pac-Man. Inky sort of works with Pinky, and, or, yeah, sorry, Inky works with Blinky and sort of tries to intercept. Like, you know, just put Pac-Man into a corner. And Clyde's just sort of does his own thing.

[7:00] I don't actually remember, he just wanders around. And so this behavior could be represented, buy a state machine. And so this is like our typical ghost behavior, where we're wandering a maze. And if the ghost loses Pac-Man, then it has a chase Pac-Man. If Pac-Man eats a power pill, then the ghost actually goes to a different state, which is running away from Pac-Man. And when the power pill wears off, then the ghost is going to wander the maze again.

[8:00] And, you know, again. Depending on the ghost's behavior. It's going to eventually decide to chase Pac-Man, and if the ghost is eaten by Pac-Man, then it gets returned to the base. So how do we actually represent state machines like this in code? Well, I'm going to give you a quick, probably two minute, introduction to XState. How many of you have worked with XState before? All right, that is a lot of hands.

[8:00] I really, really like that. But for those who think, okay, maybe XZ2 too much of a learning curve, too confusing. Let's try to do it in a couple minutes. This is XState, you import Crete machine from XState, and, by the way, npm install XState. If you haven't guessed, you specify an initial state in create machine, and then in that state you specify transitions. So, for example, our initial state is wandering Maze, and we have two possible events here on lose Pac-Man, we could go to the chase Pac-Man state, and on Pac-Man needs power pill, we go to the run away from Pac-Man states and then just.

[9:00] Lather, rinse, repeat. All the states are, you know, the same recursive structure. And then, if you wanted to use XD in something like React, we have an integration package, XState, Slash React. You import the useMachine hook from it, and then it's just like useReducer. So you have the state and the send that you destructure from what's returned from useMachine, and you pass the machine in there. And then you could do things like just match against which state you're currently in, and you could also have context and things like that to get further data from your state machine.

[10:00] And you could send events and these events are just like what you would dispatch with useReducer or with Redux if you've used Redux before. Exact same thing. So now, you know, XState, you have no excuse, all right? So now let's talk about reinforcement learning. Reinforcement learning is exciting to me because this represents how, or it forms a way of making state machines able to learn. And so a summary of reinforcement learning is.

[10:00] An intelligent agent learns through trial and error how to take actions inside an environments, and this might be an environment's unknown to the agents. And that's what that trial and error is about. And the goal of an agent in reinforcement learning is to maximize a future reward, just like hopefully what all of us are doing, so reinforcement learning described in the state machine. We start by an agent just wandering around, observing an environment.

[11:00] For example, coming to Spain for the first time and you're not sure. You know how things work, where everything is, why things are closed in the middle of the day, you know? And then you select an action, so basically you decide what you want to do based on the currently available actions. And so, based on that, the environment will React depending on what you do, and then you observe what you do.

[11:00] So when you do something, if you get a reward, you're more likely to do that thing again. If you don't get a reward, then, you know, it depends. You might do it again, or if you're punished for it. So if you get a negative reward, then you're going to really, really shy away from doing that in the future. Now, bringing this back. To the Pac-Man example, our environment is the Maze, and this environment can actually change over time.

[11:00] The environment includes, by the way, all of the ghosts inside, so let's say that you're the agent you're controlling Pac-Man, and the environment is just the Maze and the ghosts. So when you eat one of the cherry things, I guess they're called, then the environment actually changes. The ghosts change, you can now eat the ghosts. You could do different actions. And so the agent is the thing that is actually observing the state of the environments at every time step and deciding which action it could take.

[12:00] Now, of course, Pac-Man can take a few actions it could eat. Pills are, are those pills? I don't know, Pebbles, whatever they are. It could be chased by ghosts, and hopefully you're running away from the ghosts. Or if you're in Scatter mode, Pac-Man can actually eat the ghosts. And so your policy? You as an agent controlling Pac-Man, your job is to decide what to do to maximize the future reward. So, for example, if you're at the very top of the Maze and you keep pressing up, then that's that's not really going to do you any good.

[12:00] But if you decide to take an action where you're running away from ghosts and collecting these pills and these cherries, then you're going to maximize that. Future reward. And so, this future reward might come immediately, or it might come later. And so if you have a delayed future reward, then ideally, you would remember what are the steps I took to achieve that future reward? And you will say, Okay, I'm going to just repeat those steps and assign some sort of discounted reward to those previous steps.

[13:00] And so this is what is known as credit assignment. So in games, it's easy enough to see, okay, when I'm doing something, my score goes up, but sometimes in games you don't know if you're doing something good or not. Until later. So in your mind, you have to think, was that a good move? And so you sort of assign yourself an internal score. This happens a lot in chess, you don't get points for capturing pieces or for checking the king or anything like that.

[13:00] Instead, your goal is to find a way to a Checkmate. And once you do, you could backtrack and think, okay, the moves that I played, but, contributed to me. Delivering a Checkmate and so that's how you could sort of just do that internal credit assignment. And basically, the goal of reinforcement learning is to essentially find the optimal path from one point to another point. So pop Quiz what is the shortest path from A to C?

[14:00] Just that top line, right? But now, what if these lines were weighted? Okay? So, just like you know traffic in Google Maps, you would see some lines are red, you might take an alternate path. So just to do another example of this, I. Something that really taught me about reinforcement learning was having a doc. So teaching a dog a trick, let's say that you tell your dog down, you're trying to teach your dog this trick, and the dog does nothing.

[14:00] Obviously, the dog's not going to get a reward, so it's not going to really associate that with anything. But if a dog does some exploration, like following your hand or following the treat or something, then it is going to get a reward. And it's going to remember that. It's like, when my owner said, down, I got a treat, so that was a good thing. And in the future, the dog's going to exploits that, and that's why if you tell your dog down or sit.

[15:00] If a dog is properly trained, then even though the dog might not get a reward, the dog is still going to do it. Because it knows that in the past, that was a good action to take. Now, let's say that your dog runs away, that's obviously an undesired outcome. You might shout at your dog, and hopefully that's all you do. But yeah, this, this is sort of a negative reward for your dog, and your dog is going to remember I probably shouldn't do that.

[15:00] So, overall, if you think of your dog as a state machine, I know it sounds weird, but basically, you're starting it down. You want to get to the reward state, and your dog is trying to find the most optimal path to get a reward. So when you say down, it's just going to follow these green lines and go to the reward, and that's how. Your dog learns tricks. And so this reinforcement learning is actually really, really useful in fine-tuning large language models, and that's why, chat, GPT, and other large language models use it.

[16:00] They use what's called reinforcement learning with human feedback, in which humans, like yourselves, sign points or credits to whether answers are good or not. And it helps shape the LLM into deciding whether that answer was, what you expect, all right? So now let's talk real briefly about LLANs, also known as large slingwitch models. So LLMs are basically ways to predict next tokens, that's pretty much the simplest way I could explain it.

[16:00] Where you give it an inputs, and given that input, the role of the LLM is to basically act as a prediction engine. And say, given your input, here is probably the next sequence of tokens that you would expect from that input, based again on that reinforcement learning with human feedback. So now, how do we actually add these LLNs to, you know, to your applications? If you come away with anything? This is the one thing that I want you to learn, because this is much, much easier than you may think.

[17:00] So, the first step is to get an OpenAI API key, or if you're using any other model, like from Anthropic, you're using Cloud, same process, you would get that API key. Open.Ai has a, an SDK, you just npm install OpenAI, and then you use their SDK to create completions. And these are just basically wrapped. Fetch requests with optional streaming if you want, and then you just display them to your user. And so I really like this idea of using LLMs with things like reinforcement learning and state machines, because they could really improve and just enable new experiences in our apps.

[18:00] I was inspired by this paper by the way, where it was learning to play Atari with the help of instruction manual. So basically, like I said in the past, reinforcement learning was used to. You know, to have agents try to play games just as good as humans. And recently, they discovered that if they just fed the instruction manual before it tried to learn the game, then it was much, much faster at learning the games.

[18:00] So lesson here read the manual to learn faster, all right. So this is what we're going to do, we're going to make an espresso machine, and, you know, just so that we're all awake. And we are going to do a few things and just combine LLMs with state machines with reinforcement learning and maybe learn a few things in the process. And essentially, we're going to make our espresso machine. Learn new things without explicitly programming them in.

[19:00] So I'll show you what I mean. Hopefully the Wi-Fi works. I'm counting on Eric over here of Solid..js Fame to, you know, help with that. And this is my very simple espresso machine you could see over here. If I press start, then it's preparing the ingredients. The beans are whole, so this is my finite state, so I could grind the beans. And now the beans are grinding and then the water, you know, it'll be Heating.

[19:00] And so basically, these are all the steps for an espresso machine to take, and then when the espresso machine is running, eventually it'll produce an espresso. And just to make sure you could see that, uh. There's different actions or events that you could send, such as adding water, steaming milk, adding cream, adding chocolate. Or you could just restart. So let me just real quick show you this machine. This is with X8.

[19:00] Again, we're using create machine and assigned to update other parts of these states, so this contents over here. So we could say we're adding milk, we're adding cream or chocolate or something like that. And I'm actually going to take this, I'm going to copy it over and I am going to import it again. If the Wi-Fi works, I'm going to import it into Stately..ai, and we're going to see what it looks like.

[20:00] So this is our steam machine, we see that we start in the idle state. I could actually simulate this over here. And you know, then the beans are grinding. And so basically, this is a steam machine that describes the process of making espresso and preparing drinks and all that. But one interesting thing is that if I search over here, you really can't see that, but if I search for something like, I don't know.

[20:00] Cappuccino or cortado, like. None of that exists in the state machine, so what we're going to do is we're. Going to be using OpenAI, and we're going to be adding functionality to this espresso machine without actually changing the code. And, the way we're going to do this is we're going to be reading this espresso machine. Getting the shortest path and feeding it into OpenAI as a prompt, saying which is the goal state that best reaches?

[21:00] This? prompts that the user provides. So this is a quick intro to how you would use OpenAI to do that. First of all. We import Open.ai from Open.Ai, so just like that, and hopefully that font is big enough, assuming if it's not, and then you pass in your API key, so you have new OpenAI API key. Who wants to see it? Nope, and also we have dangerously allow browser true. I'm just doing this in the demo environment, you should really be doing this on the server side.

[21:00] Don't try to do this in the browser side because your API key is going to be exposed. And then people are going to be using your key for all sorts of things and you don't want that all right. So the completion looks like this you await OpenAI.chat.completions.create and the first thing you provide is a system message. Now, I'm usually pretty vague here, but I just say you're a helpful AI assistant that's good at classification and knows how to make a coffee.

[22:00] And then, as a user, I provided this prompt if I want to achieve this goal, and the goal might be make me a cappuccino or something like that. Which of the following end states matches that goal best? I'm using GPT4 to do it. GPT4 is smarter than GPT 3.5, but it is a little bit slower, so your mileage may vary. There's lots of models to choose from. Well, it's either GBT 3.5 or GPT4, with different context.

[22:00] Lanes temperature tells you, basically how stochastic or how random the results is going to be. So if it's high, like 0.8, then it's going to be very random, or if it's low, it's going to be a little bit random. It's never going to be fully deterministic. That's just not how LLMs work, but you know, you could just adjust that to whatever you need. And then Max tokens tells to you how many tokens or.

[23:00] How big of a response am I expecting? And then n of one is, how many choices should it provide to me? So, for example, if you want to pick the best of three choices, you could give it n of three. And then your completion thought choices array will be length of three. So then we're going to grab the content and then I'm doing a little bit of magic. By the way, I should have mention here.

[23:00] I'm using X8 graph to get the shortest paths for my state machine. And basically, the rest of the code is me finding the shortest path from the initial state to the state that the LLM predicted for the user prompt.

[24:00] And then we execute each step by just calling Actor.send and we send it that event. So, without further ado, let's take a look what should I make in my espresso machine? Cortado? Okay, so, okay, let's make it. And so you can see in the console, we have 105 potential states sending that to OpenAI. It's going to predict the drink and then it's going to go and hopefully just automatically make it so it's running.

[24:00] It may be espresso, now, it's steaming milk and then it's combining it and we have a cortado. [Applause]. [Music] Oh, sorry, okay, so learnings all right. So, first of all, one of the biggest things that you should realize is that LLMs are unpredictable, and that's why we use, you know, state machines to help. But, LLMs, by their very nature, they're unexplainable. You don't know what they're going to, you know, predict next, and you can't really see the inner workings of them.

[25:00] You can't really, just they're not going to give you the same answer every time. So, if you want to try end-to-end or unit testing with an LLM, good luck. If you figure it out, please tell me how you do it. Also, event-based logic makes our apps more declarative. So you saw that I was able to basically program this agent to effect this app just by sending it events. And so it makes things a lot easier.

[25:00] I always talk about, like, just how great events are. Rather than directly manipulating state, and so this is just one of those examples. Another learning is that state machines and reinforcement learning make. These LLM agents more intelligent because we're basically providing guard rails for the agents to follow, rather than just trying to have the LLN figure out its own path. Which it probably will get wrong without a lot of fine tuning.

[26:00] So I have a lot of future plans in playing around with this versus state machine synthesis. If you remember Kevin's talk yesterday about mazes, you could basically consider your app a maze, and your app basically. At every time step, the user could do certain actions, like filling out a form, submitting a form, you know, just clicking certain buttons and things like that, and all of that could be mapped out. And essentially, it becomes possible for you to synthesize the state machine just by an agent exploring your application.

[26:00] And this opens the door to a lot of things, like generating end-to-end tests. And also just getting more visibility into what your users can and should or should not be able to do in your apps. Another one is adaptive UIs with reinforcement learning. And so what this means is, let's say that you already have that state machine. You could actually put weights to each of those transitions and see which ones are users most likely going to take.

[27:00] And so that way you could have a measurable way of knowing. Hey, I added this feature. Was this feature actually good? Or you could go the other way around and actually predict what the user is going to do next and create those adaptive UIs with that. There's also predictable LLM output. There's a lot of tools for this already, such as Microsoft has this thing called Lang chat, or sorry type chat, which allows you to constrain the formats of what the LOM returns.

[27:00] And you could also use OpenAI functions for the same thing. And then there's deep learning for state space reduction. This example was a little bit contrived because it's an espresso machine with a few ingredients. And we already saw that it generated over a hundred different possible states. So if you have a bigger app, that might get in the thousands or tens of thousands. And that's obviously going to blow past the maximum context length that you could provide in LLM.

[27:00] So, you know, there might be future ways of just reducing that in a way. I also want to mention, speaking of future plans, that. We've released XHV5 beta, which was used in this demo, and if you want to try it out, then it is available at XState beta. We have made a lot of improvements there. But more related to this demo, there's a couple of things that, you know, I really want to experiment with and share with you and document.

[28:00] I'll talk about daily AI slash agents in the bit, afterwards you could come and talk to me and I'll talk about that. But basically, we want to create wrappers that make it really easy for you to use XState itself to create these, prompt chains and things like that. And I also used X8 Slash graph at beta in this demo. And it's not really documented yet, so I'm sorry about that, but the documentation is coming soon.

[28:00] And one more thing I want to show you. Speaking of LLMs is that, hopefully it's here is that we're actually using LLMs.

[29:00] In Stately itself to add new features such as generating flows. And so, for example, if I wanted to just generate a state machine, like, try anything. An app for ordering coffee, where users can order a cappuccino or cortado or Americano. Just a bunch of things, and it can be delivered or picked up. Then I click generate. So what's happening over here is that we're using OpenAI with NEXJs and React on the back end, and, we're doing a bunch of prompting and it, wow, that was fast.

[29:00] So it actually created this state machine from that, and we could see that. It doesn't always get it perfect, but it gets you to a really good starting point, where, you know, we're in the idle state. It shows that you could select a coffee, select delivery, you could confirm the order and you know it processed the order, the order is completed. And you could also go further than that and actually modify the flow using AI.

[30:00] So, it's an experimental feature, but it just shows you how you could use OpenAI in your existing apps without really changing that much code. So yeah, with that, there's basically two ways to make your state management intelligent. First of all, on the state management side, declarative. UIs such as React and what react to provide to you is really great. Declarative state management, using events and having finite states really makes your app more, capable.

[30:00] To, you know, just being either understood by your entire team. Or using it to generate things like shortest paths and graphs so that things like agents can more easily Traverse it. And the general point is separating your app log or your app logic from your view logic. And, of course, intelligent like AI is actually becoming table Stakes for lots of apps nowadays. Many people want to just use AI in your apps.

[31:00] And so I'm just asking, you don't just add chat Bots or, you know, things that you could chat with in your apps. But actually think beyond that and think, how can we actually make our apps help our users in a more intelligent way? And of course, there's a many different ways to do that, using LLMs reinforcement learning, or, like we showed today, using LOlens as minimally as possible. Such as determining what future states we want our app to be in to help the users, so with that, thank you very much.

[31:00] React Alicante.