Making state management intelligent

If you, I wouldn't call them technical difficulties, they're more like physical difficulties. But anyway, hey, my name is David Khourshid. I'm at @DavidKPiano pretty much everywhere. And I'm so excited to be back at Reactathon this year. Talking about something completely different, making state management intelligent, like, what exactly does that mean? Anyway? I have less than half an hour and I have a ton of information to give to you, so let's just jump right into it. First of all, I work at a company called Stately..ai, which is really apt. Because I'm going to be talking to you about two things first thing is states and the second thing is AI, of course. And so I'm going to have two very important takes, and I don't know if they're hot takes or not, but we'll see. We're gonna have two takes at the end of this talk. And so just try to keep that in mind.

But, first, let's talk about states. I think that as developers, especially React developers, we have a plethora of options on how we manage states in applications. And so we could, you know, have all these different HOOks and state management libraries. And sometimes we think, okay, there's trade-offs like we're going to use one over the other. Because I like the simplicity versus I need to manage complex date. But in reality, I see it more of an evolutionary path, where as our app gets a lot more complex, we have more code, more features. We have to eventually graduate for from simple things like use states, and we might need to start using useReducer because we want to organize our logic a little bit more. And, so, you know, applications do get a lot more complex, and so that's why it's good to just know all of the different options that we have. By the way, don't look too much at this code. It was generated by chat TPT, so I cannot attest to the quality of it.

There's also useEffects when you actually have to start caring about effects. I, I gave a whole talk about that last year, so I'm not going to get into it this time. There's used context when you're really nostalgic for Redux and you want to build it yourself, and you also don't know. When things are going to render and you're totally fine with that, so there's use context for shared state stuff like that. There's USync external store, which is really useful, for just having, well, external stores. And I wish that this was a friendlier name. Because, really, this is like a library author hook, but it's honestly one of my favorite hooks for managing state. And then we have a plethora of options for just different state management libraries. And so the reason that I'm showing you all this is because our app applications have different needs, different use cases, and as developers, we use a variety of these solutions to get an application to get you know what the user wants to do.

And so this has been working pretty well, you know, for many years. There's people successfully using whichever state management library, and I'm not going to convince you to use one or the other.

Play from 0:00

Or whether you should use the state management library or something like that, that's not really my point. My point is that right now, times are changing. And so, you know, you may be familiar with one of the biggest advancements that happen in just in technological history. At least in my opinion. And, not to overhype it, but of course, we have chat. GPT GPT3 3.54. And now they have company. So this is all based on what's called an LLM, or a large language model. And so you have many, I don't know if we want to call them competitors, but like many of these other LLMs have entered the arena so to speak. So we have Claude from Anthropic. We have a variety of open source software solutions. Like Facebook's llama, and even just yesterday or two days ago, they announced open Llama, where they're actually recreating llama from scratch. So that it is actually, like, more viable for commercial usage.

ETC. Sydney from Bing, borrowed from Google, and now we're even having on-device, LlNs. So this space is getting really crowded and of course, it's super, super hyped right now. And the reason is because users, they want to take advantage of AI. And now our expectations for how we build applications and what we provide to users is changing. Users actually want intelligent apps, more or less there are. Some people who say, I don't really want to have AI do things for me, and for good reason, I completely understand that. But you know, just if you're working with notion or things like that, it is really useful to have AI just basically do things for you. And how many of you are using Copilot or chat, GPT, or any of that, like, just in your daily work, and that's pretty much most of you. So, you know, it just goes to show that you do want, and users to want, intelligent, whether you're a developer or a user of an application.

But it's sort of strange right now. Because the nature of LLMs and using generative AI is that we're basic be making API calls that are slow, they're expensive. You know, of course, sometimes wrong, and you know, this is, they're impossible to debug, and so this is sub-optimal and bad coding practice. But all of a sudden we slap an L Alum on it. And now our company is worth 100 million dollars because investors decide to throw money at it. However, despite all of this, I would say that LLMs are not enough. And so these are three recent articles that came out within the last week where, you know, one is saying, chat Bots are not the future, which I agree with. There's also the UI's GPT won't replace, where it argues that user interfaces are still really useful. Again, we're not going to try to transform everything to a chatbot, and also, unpredictable black boxes are terrible interfaces.

Just because we cannot predict, like we don't have a predictive model of how applications are going to work, right? So if a user does something and we don't get the expected output now, we're like, Okay, what's supposed to happen here? Do I have to tweak the prompts, do I have to, like, say the right magical words to make it work? It just it doesn't become that good of a user experience. And so, in my opinion, I think the core of this problem is that LLMs operate at this level. Where you have an input and it does its magic in its black box, and then it produces an output, whether that output is acceptable or not is up to the user. But what? We really want is to be able to express a goal and to have the AI or the application, whether it's using AI or not, actually achieve that goal. Somehow. And so funny enough, there are actually LLM based technologies doing this right now, such as agents GPT, Lane chain, auto GPT, and actually, agency uses link chain.

The idea is that these are using what's called chain of thought reasoning. So, the LLM is able to ask itself questions. And it gets in this loop and it's able to accomplish those set of tasks and reach that goal, hopefully, and immensely. And there's another one here where I'm including it in its own side, just because there's a state machine, which was created by Sean. So, I, I like it a lot, but it actually really clearly shows like, what, you know what's actually going on? Where it's basically asking itself questions. It's storing its context in a in a vector database, and then it's just going until all of its tasks are completed. But I actually want to take a step back, like, way back to, you know, 1980s, when the idea of what's known as reinforcement learning was first introduced.

Play from 3:00

And so if you take anything from this talk, well, I actually really want you to take two things. But one of the things is, you know just what reinforcement learning is, just because I feel it's really important to learn what it is. So, in short, reinforcement learning is an intelligent agent. Learning through trial and error how to take actions inside of some environments, whether it's known or unknown, in order to maximize the future reward. And so there's a lot of really cool things in research that has actually come out of this. One of these is more recent, I think this is from 2021. Google research, did the thing where they basically took the standard models of, you know, just agents being able to play Atari games, and they gave it a discrete world model. So it was able to actually understand what was happening in the world, just more intelligently than doing random actions and things like that.

And we're going to be talking about that a little bit later. And there was also this more recently, where, you have. These little Bots that learned how to play soccer. This is just one of the videos. I really encourage you to, check out all the other videos. Also, have we not learned our lesson from pushing over? Robots like this is going to backfire, seriously, I mean, Boston Dynamics, right, right? But at least this one is cuter than that evil robot dog, so it's still pretty cool, all right. So how does reinforcement learning work well? I made a state machine for it, of course, because that's who I am as a person. So we start by an agent observing an environment, and so in observing the environment. It's selecting an appropriate action based on its policy that it's learned, and so the agent chooses the action. And the environment reacts in some sort of way. So, you know, for example, looks like I can't really think of an example, but I have examples.

So, when the environment reacts, something can happen either. Something good could happen, like a reward, and based on that, the agent will actually say, Hey, that was good. I'm going to remember that, that was good. And I'm going to update my policy based on that. Or something bad could happen. Or, you know, you might get no reward and it just continues the cycle. So if you think about it, this applies to games, this applies to robotics. This replies to a lot of things where agents are basically going through the cycle, choosing actions and at every step, determining was that action good or bad. If it was good, I'll probably repeat it in the future, and it was, if it was bad, I'm probably going to avoid it in the future. And this is how agents can actually learn to perform tasks and reach an eventual goal. So one of my favorite examples of this is actually Pac-Man, so Pac-Man can be considered the agent.

The environment changes, so you have, for example, normal mode, where Pac-Man is just, you know, vibe and eating pellets or whatever they are. There's cherries in there too, and there's also ghosts chasing him, and, there's also scatter mode. So when, Pac-Man eats the special pellet, I forget what it's called. Then the ghosts become edible, and, no, Pac-Man's able to chase them. And so the. Behavior of the environment actually changes, and so Pac-Man is able to do something different, so Pac-Man, in this case, is the agent. And again, at every time step, Pac-Man is going to do one of two things. First, it's going to observe the state of the environment. What's going on? Is a ghost about to chase me, or am I just in front of pellets that I could eat? What should I do? And then, based on the state of the environment, the agent will take an action move up, move down, move left, move right. In other video games, it's much more advanced in robotics.

There's many other things that they could do now. The agents could actually do many things, and so this is part of the policy which we'll talk about. So, for example, the agent can eat pellets, the agent can run away from ghosts, or, if the ghosts are edible, the agent might decide, you know what? I'm actually going to eat those ghosts. Just because I know that's a good thing to do. Now. How does the agent know whether something is good to do or not? That comes down to the policy, the policy informs the agent. If you're in some state, here's all the possible actions you have, and based on either some stochastic policy or what has happened before. Or, you know, just some deep learning or something like that. It's able to say I'm going to choose this action because I feel like I might get a reward if I choose this action. And so the policy informs that this is actually from a site that I saw, where there's actually.

Did you know that there are strategies for Pac-Man? I'm learning, like, for all these old video games, there's actually really, really interesting, strategy. So if you go in the red areas, I think either those are areas to avoid because you're going to get eaten by a ghost, or maybe those are strategic areas. I forget, but it gets really, really interesting. And there's also rewards, so rewards happen based on the environment, so you know, you could get eaten by a ghost, that's a bad thing, or you could, you know, just survive. And even surviving could be a reward. And so that's why the policy is based on something known. As credit assignments, and so this is how we shape what the agent should be doing. For example, we could tell the agent that you know what, eating pellets is good, you should keep doing that. Or you could say you're getting chased by a ghost if the ghost eats you, that's actually a really bad thing.

And so that's why things like score are a great proxy for things like, you know, did I do good? or did I do bad?

Play from 8:00

So why am I talking about all this? In, you know, in the context of what we have right now, which is chat GBT, and, you know, large language models? Well, the truth is that, chat GBT operates on something called RLHF, or reinforcement learning with human feedback. Sorry, and so this is in contrast from something called supervised learning, where basically with supervised learning. We're saying, if I give you a prompt, this is an example answer, here's another prompt, here's an example answer. And obviously, that doesn't work with something like chat GBT, where that gets expensive really, really quickly. And you just don't have the resources to come up with so many possible answers. So instead, the cheaper thing to do are these little, you know, thumbs up, thumbs down buttons at the top right corner. Over there where a human can say, Hey, that was a good answer. And so when you click these buttons, these are actually really important for training.

Chat GBT, because you could say, you know, that was a good answer, and ChatGPT will learn from that and will produce similar answers in the future. Now there's a lot of shortcomings, though, with reinforcement learning, which is part of the reason why it has not been like so popular, lately. First of all, just like with ChatGPT, you need a ton of Trials and also the state space can be huge. So, for example, you're not going to be able to model every single possible combination of positions that Pac-Man and all of the ghosts could be in. That just becomes really, really intractable. Also, there will be consequences. Because, which, that sounds vaguely threatening. But what I mean by that is that if you take something like a simulated environment and you apply it to real life, like the self-driving cars that I. See around here in San Francisco, you know. Chances are that it's going to realize that simulation is not equal to real life. And if it makes a mistake, whereas in a simulation, we could say, Okay, just made a mistake, let's just do a new trial.

Play from 14:00

In real life, something bad could actually happen, so, you know, we have to, be careful about that. There's also, and this, I think, is a really cool example. There's also the exploration versus exploitation problem, so with exploitation, that means, hey, what I did was good, I'm going to keep doing that. And the problem with that is that if you exploit 100 of the time and you don't explore, you learn nothing. And so exploration is sort of the opposite problem. Where you're like, hey, I don't care that what I just did was good or what I did was bad. I'm going to do something else. And so if you do that 100 of the time, you will also learn nothing. So the example I usually give is with a dog, so if you have a dog, or I was going to say a cat. But they're not really trainable, at least not in my experience. But with a dog, if you have a treat and you tell the dog, sit down, the dog's not going to do anything.

At first, if you have a puppy, you know you've tried to train an animal, and it gets no reward. But if it does a little bit of explore creation, like it's following your hand and it decides, you know what? I'm going to sit down. Then it gets a treat, and so that treat reinforces that behavior. So exploitation means that if you say go down, the dog is going to sit down, because regardless of whether or not it gets a treat. It knows that in the past, it's gotten a reward, so it's just going to keep doing that. Behavior exploration is sort of, difference, because, you know, exploration is. A dog is saying, you know, I'm gonna just run away or do something else, even though my owner has told me to sit down. And so this is an undesired outcome. And so either no reward could happen, or the owner could yell at the dog or spray it with.

The ball? I don't know how you discipline your dogs, I don't want to assume anything, but yeah, so the dog will learn eventually. If I am told the command down and I sit down, I am much more likely to get a reward if I obey my owner. And guess what you have trained your dog using reinforcement learning, and whether you believe it or not? Dogs are intelligent creatures. So that's why. Reinforcement learning does not exactly explain how the brain works, but it is a good way of explaining the psychology behind how intelligent creatures decide which actions to take. The sparse rewards problem is that I was talking about, like Pac-Man and dogs. But the truth is, you might not get a reward all of the time, and so this is called the sparse reward problem. Because you have no idea whether the action you're doing is actually good or not. And so that's why there's different techniques in reinforcement learning, like you learning and stuff like that. And just to give an example, let's say that you're trying to solve a really, really hard TypeScript problem.

You're like trying everything and it's still red, it's still yelling at you. And then you just add a generic parameter or something like that, and all of a sudden all the red squigglies go away. You're going to remember that and you're going to say, Hey, all the actions I did to get to that point were probably really good actions. So the next time you encounter a TypeScript error, you're going to repeat those actions anyway. So let's get back to states. The realization that, that I've had, and probably others have had too, are applications, unlike the real world, applications are actually really, really constrained environments. Think about them. Or we're basically building forms, we're building product apps. Of course, we're building things that are more advanced. But it all boils down to we have an application which is constrained by the code base. You know, we're not trying to build the metaverse or anything like that, we're building, glorified forms.

I'm sure we're. You know, some of us are building more than that.

Play from 16:00

But you know, they have buttons, they have inputs, they have all of these sorts of things, and so we have a discrete, finite list of. Things that a user could do, or that could happen throughout the app. So how could we use this to our advantage and actually make state management intelligent? So there's a few different ways. One of them is adaptive UIs, and there's been many experiments like this in the past. One of them is Guest..Js, where basically with adaptive UIs, we're predicting what the next user action is going to be. And so we could do things like preload pages. I know that Next.js actually does something similar to this, where it, like, maybe it's not based on user analytics, but you're actually pre-loading data so that when you click, it feels instant. And so that's an example of an adaptive UI, and you could go really far with this using predictive analytics, there's also generative AI.

There's this really cool project that I saw called E2B, which stands for English to bits. And it basically acts like a, a developer for you. Where you tell it, I want you to work on this with me. So, if you take, you know, anything minor from this talk, there's this really cool tool. Don't tell your boss that you're using it, and you could basically have your own pair programmer with you all the time. There's also gold based agents. And, shout out to Steve from Builder..io, he showed this off, which is really cool. Where you basically tell it a goal and it figures out the steps based on each step of. Like, it's going on a website, it's just scraping, okay? Here's the different links that you could click. Here's the different, you know, fields that you could fill out on the form. Choose the best action that will probably lead me to the state that I want.

Such as, I guess this is how to adopt a pet or something like that. But, definitely talk to Steve if you see him later today. This was a really, really cool project. And, of course, there's AI assistance and the most famous one Clippy way, way ahead of our time. So I've been talking about this a lot. But one of the main ideas that really gets us to intelligent user interfaces and intelligent, state management is the fact that everything is a graph, a directed graph. Specifically, everything from. You know, either your relational databases, your GraphQL relationships, it's a graph, it's a directed graph, and even things like user flows or my, one of my favorite things to show. This is the Slack decision tree for whether Slack should send a notification or not. So, whether it's implicit or explicit, everything in your application is technically a directed graph. And so I actually experimented with this idea and I thought, how could we actually make state management intelligent?

And so, here's the state machine for how I, you know, actually try to accomplish this. I, uh. So basically, we start, this is a state machine, of course, by the way, we start in the idle state and we get the state space. And so what this means is that we're traversing the application and we're determining what are all of the possible states that we could end up in and where? What are all the transitions, that we could lead to? And so from that, we're providing a prompt and we determine what is the ideal end state. So this is the small part where we're actually using OpenAI, and we're asking it, hey, based on all of these possible end states that we could be in, which states best represents the goal that, you know, we could achieve? Or, sorry, what? What best represents the NC that, you know, represents the goal the best? I'll show you the prompts.

I probably butchered that, um.

Play from 20:00

But from there we get the shortest path, which has nothing to do with AI. This is purely algorithmic, this is graph algorithms like stress, shortest path, and we Traverse the path, executing each step in the path. And then we achieve that goal. So I'm going to show a demo, hopefully the Wi-Fi works. In my experience, this demo has worked like 90 of the time, just because that's the nature of chat. TPT. So if it doesn't work, I'm actually glad because it shows that. You know, I'm not doing any tricks or anything. But basically, this is an espresso machine, it's a very basic espresso machine that I made yesterday. And so, you could do a few things, you know, you could start it, you know, grind the beans, you know their ground, we could, you know. Heat up the water and now we're tamping the beans and then it makes an espresso really, really quickly, and then the drink is made.

Then we could, you know, add a few things, or we could just restart it. So I'm going to reload this and so that's our basic machine now. In this, I'm going to press command, shift F, and I'm going to search for the word. Cappuccino. If I could spell it right, you could barely see that. But over here, it says no results found. And so this is important because, this app actually operates on the state machine, so this entire espresso machine, absolutely called, is a state machine. And I have it over here. You can't really see it that well, but you know, we have all the steps here, we have idle. Then, you know, we could grind the beans and then the other steps just show that you could add different ingredients to it. But it's all a step-by-step process, you can't just say, Oh, here's all of the ingredients. This, this, that. Now you have your drink, you actually have to go through all the steps as if it were a real automated espresso machine.

So what I'm going to do is I have this agent...ts file. And by the way, playing around with OpenAI is really, really fun and so if you're scared to try it. Like, it's actually one of the lowest learning curves, actually, I would say the highest learning curve is actually figuring out the prompts. The prompt is, you know, probably the biggest trial and error type of thing. But what I did was, you know, I got the shortest paths. I have the prompt over here and I'm asking ChatGPT first, I'm telling it. You are a helpful AI assistant that is good at classification and knows how to make coffee. I probably didn't need to add that. And so I say, if I want to achieve this goal, and then I include the prompt. Which one of the following end states matches that goal best? Only choose one, at times it would choose two.

It would mess things up, so I had to be really precise over there. And so it basically finds, which one? If there is one, sometimes it will error with no key. And then I execute each step, all right. So, like I said, I'm going to ask it, make me a cappuccino. And hopefully it's going out to ChatGPT and then it will execute the steps. Sometimes it gives funny results, so we'll see what happens here. And, all right, it did. The espresso, it's steaming some milk, the milk is steamed, it's combining it. It added cream for some reason, but there you go, there's your creamy cappuccino. Thank you. And, you know, just to prove we could do other things like Americano again, this is not in the code base at all. But yeah, so it is going to just find all of the steps traversing the state machine in order to do in order to, you know, add the proper ingredients to make the Americano.

And yeah, so I. I was pessimistic. So I had a video demo over here, which we're not going to go through. But I have two minutes left, so let's go through this. There's a lot of future things that we could do from this. One of them is called state machine synthesis, and what this means is that you don't have to write a state machine. You could basically say, here's my legacy code, here's my app, please Traverse it. Figure out the state space, and from there figure out what all of the transitions are. Without me having to do so, that is possible. It's already like there's many papers on it, so it's something that I'm excited to experiment with. And also, how does reinforcement learning play into all of this? Well paired with state machine synthesis, we could say this is the probable state machine of how your application works. But if the agent encounters like, Oh, hey, this was an unexpected state, then we could say, you know what, go back there.

That probably is something that you should adjust the weights to, so that you could go to a different transition. And realize that when the user tries to do a goal again, they're probably going to want to choose a different path. So there is that predictable LLM output. There's this cool tool that actually just came out yesterday, called Jason Former, which says, here's a JSON schema that I want you to adhere to. Please you. Know, try to make that work and also deep learning for state space production, because again, the state space could get huge. I have a minute left, so I'm just going to. Breeze through these slides, I talked about reinforcement learning with state machines and how state machines are really useful for reinforcement learning. This is actually pretty cool because the different ghosts actually exhibit different behaviors. You know, I could talk a lot more about that.

But one more thing, this is another paper that recently came out. Atari is usually used for, you know, in OpenAI's gym and things like that, for evaluating, reinforcement learning agents. And so one of the most recent advancements is that actually pretty funny? One of the most recent advancements that came out is that this research team was able to dramatically reduce the number of trials that an agent can do to learn and to exceed human behavior. In, in Atari games like this one, how did it do it just by reading the instructions? So it took the instructions, which serves us that declarative model of, you know, these, these games. And it was able to actually predictably make moves instead of just guessing and doing random moves. And so, believe it or not, reading the instructions is a very important like progression in, in AI. So I have four learnings for you.

First of all, large language models are unpredictable. Declarative logic is really, really useful. Uh. I believe that reinforcement learning gives us insights for artificial general intelligence, and graphs make many, many things possible. Some people might think that AGI is coming soon, others may think it's coming many years away. In my opinion, right now, think of AGI as a marketing term, we're still many years away from it. And language, even though it plays an important role in intelligence, there is so much more to that goes into intelligence than just language. I mean, just look at robotics, look at these autonomous agents, look at your pets. They don't talk to you, you talk to them, but they don't talk to you, you know? But yeah, and the other big thing is that, you know, in React and in application development, there's basically what I would call two truths that come to state. Or with regard to state? First of all, it's that your UI is a function of your state, and that reality really belongs to your UI framework.

Whether it's React View, next Angular, and the second one is that the state, so whatever that state that is consumed by your UI framework. It is a result of the previous date and the events that change the state. And those two things can really communicate with each other. And the reason that I talk about this is that. In the sense of making your application logic intelligent, keeping those two things separated make it a lot easier for you to build these intelligent applications. And to use things like LLMs and ChatGPT to actually, do things, so, by the way, I will quickly plug. We just released exit version 5 beta, so if you're using it, you know, please give it a try. Yeah. So again, the two takes that I really want to share, and sorry that I'm out of time, is first of all, your state management. I believe that you should separate your app logic from your view logic.

Why? Just because it's going to make it easier to do a lot of really, really cool things like I showed you with, you know, with your state and with intelligence. I believe that the best way to apply AI to your applications is as minimally as possible. You're writing software, you want your software to be declarative, you want it to be predictable, explainable, and you want to give your users a good experience. And yeah, so with all of that said, thank you so much. Reactathon [Applause] thank you.

Play from 24:00