# From AI to AGI: The path to generally intelligent software Canonical: https://davidk.dev/talks/from-ai-to-agi-the-path-to-generally-intelligent-software/ Video: https://www.youtube.com/watch?v=qoNDyOClTtI Speaker: David Khourshid (@DavidKPiano), creator of XState Type: Conference talk · Event: Agent Conf 2023 · Year: 2023 · Duration: 33:48 · Views: 7187 (as of 2026-10-19) Topics: AI agents & LLMs, Adaptive & intelligent UIs ## Summary At AgentConf 2023, David Khourshid delivers a captivating talk titled "From AI to AGI: The Path to Generally Intelligent Software." With his expertise in artificial intelligence and software development, Khourshid provides valuable insights into the journey towards achieving generally intelligent software. ## Key takeaways (quoted from the transcript) - You don't have the same environments every time, but it's still a constrained space and also things can change depending on what happens in the environment. (11:00) - The agent should not hard code the model of a specific environment, instead, it should be smart enough to generalize what environment it's in. (14:00) - I found a way like I found a specific path in the Maze where I get a good reward every time. (17:00) - Instead of our normal understanding that, you know, when we're in the prompt state and we go from feedback bad to form, instead, we are. (30:00) - Training a reinforcement learning agent on or a reinforcement learning model on it, and being able to adapt that graph to what's actually happening in the real world. (33:00) ## Chapters - 0:00 Introduction - 4:00 Symbolic AI, I think, you know, around the 1950s, 1970s… - 6:00 If Pac-Man eats the pill, then, you know… - 8:00 We are slowly or quickly, depending on your perspective… - 11:00 In Pac-Man we have an environment - 17:00 Let's say that you tell your dog down and the dog… - 32:00 Training a reinforcement learning agent on or a reinforcement learning model… ## Transcript [0:00] All right, let me just plug in, find the thing over here. Okay, all right, so yeah, as Teja said, I work at a company called Stately..ai. And so the AI is going to be important in the future. And yes, right now we're a diagramming tool for visualizing your application logic, but in the future, that AI is going to become really important. And so that's why I am very excited in this talk to talk about, just what I think the future of AI is, and where I think that we could go from here. [0:00] So my name is David Khourshid, I'm at @DavidKPiano. And I want to know what is your favorite UI component or UI pattern? Maybe like buttons, drop downs, anyone? Carousel carousels are cool, what else? Modals? Okay, I hate them, but you know, more power to you. My favorite is the command palette, like ever since I started using it in Sublime text, it's been pretty much showing up everywhere. So it's even in your Dev tools. [1:00] If you open command P or command Shifts P or control shift to P in Windows, then you're going to see your command palette. And here's the reason why I love it so much. There's even companies around it, you have Raycast, linear uses it in their product and VS code. You have a command palette, and so everything is just a few keystrokes away, which is amazing. So think about user interfaces without command palettes. [1:00] We have to first find the place in the UI that we have to hit, and then we have to. You know, maybe click a few things and then eventually type something, perhaps, and eventually we get to some definition of success where we've accomplished our goal. With command palettes, it's different, you open the command palette, you type, and straight from there you accomplish your goal. You find what you needed to find, you do what you needed to do, and it's much fewer steps than just doing it in a normal UI. [2:00] So what is like that today? That's been all over the place. Well, chat, GPT, large language models like we talked about, and, you know, GBT3 OpenAI. This is like your command palette on steroids, right? You have a single text box or input, and you enter a prompt and. Then you get a response from that. So right now we have optimized the whole idea of providing input and get an output, and we're getting really, really good at that. [2:00] In our industry as a whole, where you provide a prompt and you get a useful answer. But where I want to go, and where I see AI and user interfaces in our applications going is the ability to start somewhere. And you could consider that an input and have something to all the steps necessary to reach a specific goal. So this is something that is beyond the capability of a command palette, unless there is a specific function already programmed inside the command palette. [3:00] To say, Hey, just do this for me, write my program for me, or accomplish a specific task that might take multiple steps. Same with ChatGPT, you could ask it questions and tell it to produce you poems and songs and things all day, but if you tell ChatGPT, go make me a coffee I. I actually have no idea what it will do, but chances are it won't do it if it does be really scary. [3:00] So, so let's talk a little bit about history of AI. And so I'm going to, pretty much breeze through this because I don't have too much time. But, AI is actually a very, very old field. All of the techniques and the things that you have saw come out in the last, you know, few months or even years, have actually been researched, decades and decades before. [4:00] So, symbolic AI, I think, you know, around the 1950s, 1970s, I don't know the exact date, but, symbolic AI is the idea that we could take these prompts, and so we could translate them into essentially computer programs. And so when you translate these two computer programs and things that computers can understand and turn them into symbols, then a computer can reason about it and it could create a solution. And so the beauty of this is that with this kind of AI, you don't have any large data sets to deal with, and there is no guesswork. [4:00] You could actually have the computer prove that what, what it produced is actually logically correct. And so this is why this was a popular area. Of research many, many decades ago. Because we didn't have the not the technology, but like the power to like the GPUS and the memory in order to compute large data sets. The problem with this is that, you know, first of all, there's no semantics. Once you translate things into a computer language the same way that a minifier does, then you lose all semantic meaning. [5:00] So, instead of, like, David talks at agent Conf, it would be a B's at C. Like, we're condensing that information and we make it pretty much useless, so there's no natural language. Also, the real world is complicated, there's ambiguity, there's Randomness. And so symbolic AI can't really represent just the true nature of the world, where things might be unknown. And then there's core state machines. You knew I was going to talk about this. [5:00] State machines are actually one of the most classic things they're. They're actually one of the most classic ways of implementing artificial intelligence. Almost any video game you've played, whether it comes to enemies, or to screens where you have to click through, or even to NPCs non-playable characters. [6:00] It probably is powered either by state machines or a higher level of abstraction, on top of steam machines known as behavior trees. But one of the classic state machines is Pac-Man, so this is the state machine for a ghost. Where initially it's wandering a maze, and so when it sees Pac-Man, it decides, you know what, I'm going to chase Pac-Man? And so if it loses it, it will go back to wander the maze. [6:00] If Pac-Man eats the pill, then, you know, Pac-Man can eat the ghost. So it runs away from Pac-Man. And so when the power pill, wears off, it goes back to wandering the main. So this is essentially the state machine that describes the ghosts now. So interestingly, when I was I, I went down this Rabbit Hole of Pac-Man, and I realized that there's actually, many different behaviors for the ghosts. For example. And correct me if you're a Pac-Man Enthusiast, and I got this wrong. [7:00] But Blinky does. The simplest behavior it follows Pac-Man Pinky will actually try to anticipate where Pac-Man is going, so it's a bit trickier over there. Inky, I believe. According to this, it, pairs up with Blinky, and it sort of tries to intercept where, Blinky is going, so that it sort of traps you in. And, you know, catches Pac-Man. And I think Clyde just wanders around randomly, you know, Clyde's special, all right, but you know? [7:00] Over the last few decades, we've gone further into neural networks and deep learning. And so these are large data structures that try to mimic, the brain, and try to mimic the neurons in the brain. And so you have hidden layers, in input layer, hidden layers in an output layer. And so the way this works is once you feel that lots of, lots of training data, whether it's supervised or unsupervised, it changes the weights of all of these. [8:00] So that eventually you have an output at the end. [8:00] And I'm not going to talk too deeply about deep learning. Okay, too early in the morning for that, but yeah. So this is just an important advancement in AI. And again, this was researched decades before, so it's only coming to light, you know, within the last couple of decades. Because now we have the computing power to actually do this, so what's coming in the future? Well, there's a lot of contention around this and a lot of skepticism. [8:00] But we are slowly or quickly, depending on your perspective, moving towards artificial general intelligence. So what is artificial general intelligence? Again? Depending on who you ask, it's the ability to mimic humans and animals in a way that we could accomplish goals. Or, sorry, that AGI, some agents that has AGI can accomplish a goal in any environment. And so, in my view, it has to have three things first, it has to have a mental model. [9:00] And so the mental model is an understanding of the universe. So, or the world, so the ability to reason about certain things and understand relations and associations between things. It has to have a perception of time. And so what this means is that it has to realize that when it does something, that thing has consequences. And so it should be able to observe its environments at every time step and perform actions and understand if I do this action, this thing will happen, and also it has to. [9:00] Have an imagination. And so this is a really important thing. Because, right now, large language models LLMs like chat, CPT, and GPT3, and those class of language models, they do have this imagination. Why? Because we're feeding it everything from, you know, from the internet, and from the Corpus of basically what we talked about, the Corpus of human Literature ETC. And so we're getting closer and closer to AGI, but we I. I would say that it is still many, many, many, many years away. [10:00] And so that's why in this talk, we're going to be pointing towards the future of AGI. In the same day, in the same way that we point a telescope, to the sky, and so, uh. Many researchers believe that one of the most important fields in this actually doesn't really have much to do with chat, GBT or OpenAI or anything like that. But more, it has something to do with an area of AI called reinforcement learning. [10:00] And so I'm going to, basically, if you take one thing away from this talk, I want to teach you about what reinforcement learning is. And just how important it's going to be in the future, even if it's very limited right now, so we have a few parts. We have an agent which is like you and me, and in environments, which, for example, is the stage all of you over here. [11:00] And there's actions in aging can do. And depending on its goal, it could get a reward for what it does in the environment. So let's say Pac-Man as an example. In Pac-Man we have an environment. The environment is the Maze, and so there's many things in the environment. There's these little pellets of over here, there's a power, whatever they're called. I think there's like cherries and stuff, and there's ghosts and so this maze, it could vary. [11:00] You don't have the same environments every time, but it's still a constrained space and also things can change depending on what happens in the environment. So you could consider the positions of ghosts, the environment, and also just the mode that you're in. So when Pac-Man eats a pill, then you get into scatter mode, and so now the ghosts are running away from you because you. Could eat them. So, the agent is the thing that is interacting in the environments, and the agents has a few goals. [12:00] So the agent has to observe the state of the environment and it has to take action, so right now, I'm observing the state of the environment. You are all sitting here and I have to take action, and so the action is talking to you about this topic. Now, what if I just sat here on the stage and did nothing? You know, I, you know, it would be pretty awkward. So it like, that's not my goal in this environment. [12:00] And so the agent can do many things depending on its environment. So, for example, it could run away from the ghosts, or it could chase the ghosts again. It could decide what to do, and this happens at every time step. It observes the environment and decides based on my environments I should do this or I should do something else. And so that decision is called a policy, the policy decides what should the agents do next. [12:00] So, for example, Pac-Man is running around. And depending on whatever policy is implemented, it would say, Hey, you should probably move up at this time step. And so the policy is really important because that determines how smart the agent actually is. So with the policy. You could run around randomly, or you could do strategies. Which again, I went down this Rabbit Hole. There's actually a lot of strategies for Pac-Man. Like, you shouldn't stay too long in the tunnels and in the red areas over here. [13:00] You should actually avoid them because it's really easy for the ghosts to trap you in there. So it if you're playing Pac-Man, just avoid those areas. And so that would be an example of a policy. And then, of course, there's a reward. Now, the reward could be the score, it could be the fact that you ate a ghost. Basically, the reward is a way of indicating to you how good was that action. [13:00] So how do we indicate that? We indicated using a value function? And so the value function says, if you do this action, here is the perceived reward. For example, eating a cherry that's good, here is your reward, or getting eaten by ghost that's bad. And so this tells the agent. Hey, maybe you should avoid that next time, and you shouldn't get eaten by a ghost. So the agent will choose actions that actively avoid that. [14:00] Now, the thing with rewards is that they can be sparse. Which means you might not know whether you're doing the right thing or the wrong thing until later. And we'll talk about that well later. And then we have the model, so the model is the agent's understanding of the environment, again, the model. Or, sorry. The agent should not hard code the model of a specific environment, instead, it should be smart enough to generalize what environment it's in. [14:00] Like, there's many different Pac-Man games, and so if you've played Pac-Man and you're the agents playing Pac-Man, you're not going to memorize this map of Pac-Man. But if you play another game, you're going to say, you know what? I could operate in this environment, even if it's slightly different, okay? So with reinforcement learning, there's a lot of shortcomings, and so first of all, learning, you need a lot of Trials. So for an agent like Pac-Man to go through the Maze and figure everything out, it needs to do it a lot of times and eventually it will develop a good enough policy to navigate through it and to play the game itself. [15:00] So there's, there's reinforcement learning models that, I forget what it's called, but it's trained on all of the Atari games. You can just have it play and exceed at these Atari games. But they're, you know, they required lots and lots of computing power and trials in order to do that. There's also the common notorial state explosion. So the reason that you see, reinforcement learning works so well in games is that games are a constrained environment. [15:00] Try to do things in the real world. And you have the problem of multi-dimensionality, so you have so many different parameters and so many different variables. And even things that are unknown and hidden to you. That it's very infeasible to just represent that in a single model. And also I have this threatening thing over here, there will be consequences. What this means, it's not really threatening. It just means that when an agent does something in real life, you know it's not like you could just try everything right, just like. [16:00] We can't just try to do things push buttons. In the real world, we might accidentally destroy something, break something, change something. And this is in a way where the environment permanently changes and we can't go back. So it's actually really difficult to model or to use reinforcement learning in the real world. Also with planning, you need again lots of Trials. There's a problem of sparse rewards, where it's like you don't know whether you're doing a good job or not until later. [16:00] So how do you quantify that? There's the exploration versus exploitation problem, and also the value function is arbitrary, so, speaking of exploration versus exploitation, exploitation means. [17:00] I found a way like I found a specific path in the Maze where I get a good reward every time. And so this happens with us humans, too, where we exploits that path, because we know that is always the good path, but if we don't explore, then we don't know whether that's the best path. So exploitation has a problem where it doesn't learn anything because it continues on the same path every time exploration. [17:00] On the other hand, if we do it 100 of the time, we're also going to learn nothing, because we're going to do random actions every single time. And so, pointing towards AGI. One of my favorite examples is a dog. If you have a dog, or a cat, or another animal that you could train, you're going to understand this pretty, intimately, and so this idea is training your dog. So let's say that you tell your dog down and the dog does nothing, well, there's going to be no reward, nothing changes, so the total reward of this is zero. [18:00] But let's say that you somehow coax your dog to actually go down, and then you give the dog a treat, so this dog gets a reward. And it remembers, Hey, I went down when my human, you know, told me to go down, so I'm going to get a reward. And so what this means is that when the owner tells the dog to go down and the dog goes down again. It anticipates a future reward, so it's going to do that every time, or at least hopefully every time, because it anticipates that. [18:00] Hey, I might get a reward from this, this. Is called exploitation. It knows the path to go. Exploration is when the dog says, you know what? I'm feeling a bit rebellious. I'm going to do something different. When my owner tells me, go down, I'm going to run away. So what are you going to do? You're going to scream at the dog and this is going to be an undesired outcome for the dog. [18:00] It doesn't get a reward. So eventually the dog builds this mental model in the dog's head. Where when my owner tells me down, I am much more likely to get a reward if I go down. So that's how dogs learn, in case, you know, hopefully now you will get a dog and you will see, oh wow, I'm doing reinforcement learning with the dog, all right? So there's also the problem of sparse rewards. [19:00] Policy drives actions, and it says, Hey, if you want to accomplish a goal, you should do this, this, this and that. And so the problem is just like, in real life, you don't know whether what you're doing is actually going to work in the end. So whether you're going to get a reward in the long run. And so there's many different ways of quote unquote addressing this problem or solving it in a way. [19:00] In reinforcement learning, one of the ways is key learning where. If you eventually get a reward, you remember all the actions that you did that led to that. And so with a discount rate, which means that, the actions that you did closest to getting the reward are rewarded more heavily than previous actions. This is a way of assigning values to those actions, and so that could be useful in certain scenarios. I all right. [20:00] So moving on, how can we improve reinforcement learning? Just doing a time check, all right. So I think that we could use one of the oldest AI techniques, which probably guessed state machines, of course, and so here's how we could use state machines. To, you know, have to clap, Okay, you see, so that was reinforcement learning in action. I learned that when I pick up my cup, I get the reward of clapping. [20:00] I'm not going to do it every time, but now, you know what reinforcement learning is. So, we're going to model, or, sorry, we're. Going to use state machines with reinforcement learning, and so we're going to do it by modeling our environment as a state machine. Where the state is actually finite. States grouped by common attributes. And this is where deep learning can help a lot, because we could classify states that may look infinite, and we could use regression models. [21:00] To say, like, okay, this date actually belongs in this general area, that state belongs in a different area. And so our action is decided like our policy is going to be. What is the shortest path to whatever goal I want to accomplish? And that's how we choose do this next, and then our reward is actually going to be a little different. It's not an actual, just arbitrary value, but it's instead. If we have an idea of I know how to navigate this Maze and do these steps to reach a goal. [21:00] How well does the environment reflect my understanding of the world? So, for example, and I've had lots of problems with this. Here, I have a model of how a door works, right, you open it, you pull, it opens. In Austria, that's not the case. You have to push a lot. And so I, you know, I, I get like, punished for that, so I get no reward from, you know, opening the door a certain way. [21:00] So I have to relearn the model and know that I have to push. Or at least a lot of the time, your doors are very confusing over here, okay? So making apps intelligent with large language models. Again, we have an input and an output, but it still can't tell us the desired goal. We were talking in the panel about Lang chain and things like that. But there is a chance, because it still tries to generate an output and say, here's all the steps. [22:00] This could still be wrong because LLMs do not have the concept of time. They don't actually do these things in an environment and observe what changes in the environment. So instead with reinforcement learning, we could use just plain reinforcement learning augmented with state machines. And we're going to use LLM for one small part. And that is determining what our desired goal is going to be. And so we're going to be using a very simple app. [22:00] That I made. It's a feedback form, and so this app. We have a prompt, and, we have a form where you could fill out the prompt and it says, thank you for your feedback. So it's not as simple as just producing an output, it actually has to do things. It has a click a button, fill out the form, submit that form, ETC. And so here's the strategy we're going to use. We're going to model the environment, so we're going to observe the state, the action that we want to execute, and then we're going to model it. [23:00] Saying, if I do this action, here's the most probable next state. And so from these observations, we could actually derive a state machine. Now, for the sake of time, we're not going to be doing this part of the experiments, instead, we're going to assume, all right, we have already a state machine. Understanding of what our environment, which is the form, is, we have an understanding of what that's going to be. We're also going to use LLM, so chat GBT, actually GPT3, not chat CPT, because it doesn't have an API yet. [23:00] And we are going to translate what we want from plain language into a state. So we're not asking GPT3 tell me all the steps to do it, instead we're using GPT3 to tell me what should the final outcome be? And so it's very good at that. So we're translating these parts into an actual states, and we're also going to generate the probable event data which is needed in order to perform these actions. [24:00] So from that, we're going to use, you know, just a simulated reinforcement. Learning with a little bit of graph theory. And basically find the shortest path, and so with analytics, you could find the shortest weighted path. In order to get to that final state step. And we're going to execute it. But, if my demo fails, I'm going to tell you what I wanted to do. And so we're going to actually change the UI and see can the model actually adapt to this new UI. [24:00] All right, so ready to see a demo? All right, so let's do this. All right. So again, I have my feedback app, I'm using Puppets here, and I'm also using XState and X8 is describing. The steep machine of my feedback app. And, yeah, so let me just zoom in here a little bit. I have over here a weight feedback prompts prompt. And so what this is going to do is we are going to pass it to, let's see, we're going to pass it to ChatGPT. [25:00] And so this is that feedback prompt over here. And so what we're doing over here is we have some data over here. That says, give me the desired states and also give me the example events that I need from this. So I'm reaching out to GPT3. And this is actually pretty simple to do, with the OpenAI API, so I have this over here, you could download it from, OpenAI, so npm, install Open.ai. [25:00] And so, if we go back to our prompt, I'm going to basically say, you know what, I? I attended a small part of Kent C. Dodd's workshop the other day. And so I'm going to say, let's see. There was not enough Coffeescript send that feedback, there should have been more coffee scripts. So I'm giving the plain language description. And we're going to press play, this is going to open a browser, hands off. [26:00] It's going to figure out the steps that it needs to do in order to do that, and it's going to fail, and let's see why it fails. I'm glad it failed, because now, you know that I'm not, you know, bluffing. So here's the problem with OpenAI. It's actually pretty, unpredictable sometimes. So I'm just going to try it again, right? So your whole idea of, are we gonna, you know, try it again? [26:00] And are things going to happen? So okay, let's try this again? Sorry about that. I knew that this was going to fail, so actually I was hoping that it wasn't. So, send feedback that there was not enough. Coffee script. Let's take a look at this too. So prompt submitted form. Okay, come on, let's try this. I have a backup just in case this doesn't work. Oh, sorry, this is actually okay. There was not enough coffee script and it submitted that feedback. [27:00] So [Applause]. But okay, so B Pi. So the GPT3 model it was probably producing, like a wrong state. Even though I tried to, I tried to fit the problem so that it did the correct. State. All right. So now, even though we broke things already, let's actually break things a little bit more. Which is in a really, really, really scary. So instead, I am going to just change this real quick, quick and I'm going to make a splash screen. [28:00] This is going to be a very, very silly splash screen. And so I'm going to, someone else give me some different feedback. Who attended the workshop? I, I really can't see any hints, but like, actually, let's talk about the conference? Let's say that the conference, was, well, that's, that's Chad. CBT I didn't see that chat error. Sorry, not chappy. TBT co-pilot said that, okay, please submit that feedback to the organizers. I don't know. [28:00] Don't worry, this is gonna fail. So it's not actually going to, all right. So it's going to figure out the path to do it, hopefully. And, okay, so we have this. Let's get more information, prepare yourself. This is a screen that I threw just to throw the model off, which it did. So it's like, Okay, we, we have an unexpected state over here, so let's pretend again, we're not going to get into this too much. [29:00] I'm just going to tell you it is possible. Maybe in the future demo, I'll do it. But let's say that using reinforcement learning, the model learns that. Okay, we have this idea of what our of, what our model is, where we have a prompt and on feedback.bad. It goes to the form. But we know that is not sufficient because it doesn't really reflect, what's actually happening. And instead, we have some sort of surprise date we'll call it, IDK, splash. [29:00] And so what I'm basically doing is I'm not changing the model, I'm just adding to the mouse. So, pretend that this reinforcement learning agent is rediscovering the environments and learning. Hey, uh. When something happens, I realize that I actually go into a different state and so when I continue, I go to the forum. So instead of our normal understanding that, you know, when we're in the prompt state and we go from feedback bad to form, instead, we are. [30:00] Going to this weird splash screen and we have to go through there to go to the form. Now there's a good reason that I'm not actually deleting this, and it's because we have the previous understanding that works before. So now what I'm going to do is, I'm going to go down here and we are going to get rid of this error. So we are actually going to say, Hey, try doing this original shortest path, but if that doesn't work, relearn the environment and try a different, shortest path from the state that you think you're in. [30:00] So hopefully again, same feedback. The food was actually pretty good. I'm not, you know, I'm not advocating for this specific feedback. All right, so if we go back here, let's see if it could do the right thing. Okay, no pass found, you know? Let's try it again. Because again it is stochastic. So all right, let's see, all right, okay, cool. So with new to press that button and it new to submit that feedback, and so, wait, wait, wait, wait. [31:00] So now let's say we change this back. We're like, you know what? People say that the splash screen was dumb, so let's get rid of it. I'm not changing anything in the model and so hopefully it adapts to. Okay, there we go. [Applause] all right, so what just happened? What did we learn? First of all, GPT3 is unpredictable, I told you, speaking of unpredictable, I would. I predicted that it would fail, at least at some level. [31:00] Also, we've seen that state machines are actually really useful, not just modeling your application logic, but using it in AI to actually do things where, we reduce the extents of GPT3. [32:00] To say, you know what, just tell me what the goal state is going to be. And instead, theoretically, we learn the state machine, which could get very complex of an environments. And we could do this with controlled environments. Such as applications, where we basically have the browser, we don't have to interact with the real world. So, this is why, in my opinion, a lot of people say that reinforcement learning is wrong. [32:00] It is actually wrong in so many ways, but it's useful because it's wrong. So, even like theories of quantum physics. And relativity are wrong in certain ways. They provide useful insights for the future in for how things actually behave in the real world. And reinforcement learning is the same way. I believe that it gives us a lot of very useful insights towards artificial general intelligence and also just in general. The representation of things such as application logic as graphs makes many things possible, such as finding the shortest path. [33:00] Training a reinforcement learning agent on or a reinforcement learning model on it, and being able to adapt that graph to what's actually happening in the real world. To do things like, let's say that we have a command palette in your app where you just tell it. Do this for me. And it will understand every single step in order to do what you wanted to do. So with that, we could, really. I believe we can make our apps intelligent today with the technologies that we have at our very fingertips. [33:00] And so with that, thank you so much each and Conf.