# Mind-Reading with Intelligent & Adaptive UIs Canonical: https://davidk.dev/talks/mind-reading-with-intelligent-adaptive-uis/ Video: https://www.youtube.com/watch?v=na1-RumWtxE Speaker: David Khourshid (@DavidKPiano), creator of XState Type: Conference talk · Event: ReactiveConf 2019 (Prague) · Year: 2019 · Duration: 23:11 · Views: 1424 (as of 2026-10-19) Topics: Adaptive & intelligent UIs, Statecharts, XState, State machines, Forms & UX ## Summary ------------------------------------------------------------------- What if you could predict user behavior with smart UIs? In this talk, we will explore how we can make adaptive and intelligent user interfaces that learn from how individual users use your apps, and personalize the interface and features just fo ## Key takeaways (quoted from the transcript) - Today, on Halloween, which is why I'm wearing this costume, I'm gonna be talking to you about mind-reading with intelligence and adaptive user interfaces. (0:00) - The adaptive user interface can detect that and actually provide that guidance and help them achieve the same goals. (5:00) - We want to create an adaptive user interface, so here's a game plan. (9:00) - We could feed that into our environments, which is the abstract model that we've created, and use that to sort of adapt the user interface in real time. (15:00) ## Chapters - 0:00 Introduction - 13:00 We have this on transition callback… - 15:00 You can see that this is just a normal tic-tac-toe game… - 18:00 Have tests automatically generated, have even documentation automatically generated… - 22:00 I try to think X8 and Microsoft products ## Transcript [0:00] Hello everyone, my name is David course Shade, I work in Microsoft and I am so excited to be here again in Prague. The last time I was here, I talked to you all about statecharts and state machines, and how they could help influence the way that we do developments in a positive way. And today, on Halloween, which is why I'm wearing this costume, I'm gonna be talking to you about mind-reading with intelligence and adaptive user interfaces. [0:00] And so the thing is, with a Halloween, a lot of people, you might, you know, might celebrate here, I don't know, I saw a few people in costumes, so especially at this conference, and we saw that. A lot of people like to dress as something that they're not, and part of it is to express their own creativity, or just something that they like. And that's the important thing is that everyone is different. [1:00] And this is something that we have to keep in mind when we're developing applications and user interfaces. Because we all speak different languages, we all come from different backgrounds. So in my opinion, when we do things like a be tests or final charts or final diagrams, I feel like that sort of disingenuous and ineffective, because those only target certain people and we make assumptions about the people that were developing our apps for. [1:00] And also the thing is that the user flows that are represented in our app are not always linear. In fact, users could do so many different things and we have to represent those in different ways. So, one example of an adaptive user interface is guest J/S, which was made by I believe, Sam Saucony at Google. And this uses predictive analytics to intelligently prefetch resources on the page, depending on where the user is likely to end up next. [2:00] And so this contributes to a positive user experience. Because when the OR visits a webpage, then they are going to be able to see the next webpage loaded a lot faster than if they had to wait to load the resource. So this gets to the core of what adaptive user interfaces are. You might have heard of responsive user interfaces, which adapts to devices. Whether you have a laptop, a desktop with your crazy 60-inch monitors, or a cell phone, or even the Apple. [2:00] Watch or Google, if they're still doing watches. But the key is that adaptive interfaces adapts to us, we are the device in which is interacting with the application. And so one of the earliest examples of this, you might remember, is Clippy, from Microsoft. I'm legally obligated to include him in this in this slide. So, take, for example, email. A user comes into this interface and is presented with an inbox in which they could choose in action to do. [3:00] They could view the latest messages, or they could compose a new email. And so, while we might try to make a one-size-fits-all user interface that solves all the problems with different users using the app, we might see that users use it differently. In this case, this user might always try to view the latest email. Or when the user composes a new email, then they might be more likely to send the email rather than save it as drafts. [3:00] So those are user flows that we could optimize for. Another example is in the Google inbox interface, where instead of asking, are you sure when you're deleting a message? or when you're sending a message? Instead, we could reduce that and create a more adaptive user interface by saying it's deleted and giving them the option to undo that. And this completely eliminates a step in the user flow, which is very good for the user because they're able to do their actions a lot more efficiently. [4:00] So, another example in Google inbox. I don't know if you've seen this feature, but I use this all the time is when you're composing a new message, you see that it will actually dicks. The next thing that you're going to type, and sometimes it's not exactly what I want to type. I'm lazy, so I choose Google's suggestion anyway. And you know, the same thing happens in messages. And you know, maybe iPhone will do this one day too, or I was gonna say Microsoft, but that doesn't really exist. [4:00] So also, adaptive user interfaces can adapt to whether your user is a power user and knows all the function. Now they are ready, so you want to provide these power shortcuts for them to achieve their tasks as quickly as possible. Or maybe the user needs a little bit more guidance, and so when? The adaptive user interface can detect that and actually provide that guidance and help them achieve the same goals. However, developing adaptive user interfaces definitely has its challenges with adaptive user interfaces. [5:00] There's a lot of factors we need to take into consideration, including user privacy. Whether we wants to adapt at the local level or in a global level, such as adapting via routes versus, you know, adapting singular components, or you know whether we want to, you know. Provide the user an option to change it via settings. So, do we want to give the user manual control, or do we want this to happen behind the scenes? [5:00] So one way that we could tackle this problem and actually introduce adapt abuser interfaces in our everyday development is by adopting what's called model driven development, which is not a new concept at all. So the idea is that you represent your application as an abstract model. Now, if you've been doing design or working with designers, you might recognize something called user flows. And so these are. They're technically a form of abstract models which tell you which state goes to which next state in the application. [6:00] And it's a very visually clear way to represent how your user is going to interact with the application now? Last year, I talked about finite state machines and statecharts, and this is a way that we could introduce these abstract models into our application. In case you forgot, I'll just do a little bit of a reminder a finite state machine has an initial state, such as a sign-in page. It has a finite number of states, which represents all of the possible states that the user can be in when they're interacting with your app. [6:00] It has a finite number of events, which represents everything that could happen within the app, whether that action is initiated by the user or within the app itself. It has a mapping of state transitions which are triggered by these events, and so these are deterministic. And they say, when were in one state, such as the sign-in states, and an event happens, such as sign in. Now we're on the signing in state and so these arrows are going to be very important. [7:00] And there's also a finite number of final states which say the user has completed this flow. Or we're done, or this app is at its terminal points and you would have to restart or do something else to start the app over again. Now, why do we use finite state machines? Well, if you look at my talk last year, it's to prevent impossible states such as this one. Now. Statecharts are an extension of finite states, and statecharts have a lot more features that we're not going to get into right now because I don't have time. [7:00] But the idea is that using these concepts, such as finite state machines and statecharts, help prevents impossible states. And help you make sense of your application. I created XState, which, which is a way of coding these statecharts into your application, and it has all these sorts of features. And if you don't need these features, then there's other options as well, such as X8 FSM, which is just for the finite state machine. [8:00] But long story short, go check out the XState, because it's going to be a great way for you to represent your state's. Also, there's XState for React as well. I haven't released version 1 yet, but a lot of people are using the useMachine hook in production right now. And so the idea is you would have this used machine hook. You would, you would use it just like a used reducer hook. And use it directly within your application. [8:00] Now, execute also be used with you. Angular's felt, all these other things. I don't have official plugins for these quite yet, just because don't tell them reacting, but it just works so with React. We sort of had to make it happy with hooks. And so the idea is that we're trying to prevent this leak of faulty logic, edge cases and race conditions, instead of just lapping a disabled button everywhere and having that be the end of that. [9:00] So let's take an example application, let's say that we have this. This feedback form where you're asked, how was your experience? The user could click good in which it says things for your feedback, or the user could click bad in which it says, tell me why? And then there's an option for the user to fill out the form. So this is what that looks like. You have a good. Says, thanks for your feedback again. [9:00] It's a very, very simple form, you can say no, it's submit. Thanks for your feedback. Nothing too difficult to code, but we're moving beyond just coding. Now. We want to create an adaptive user interface, so here's a game plan. First, we want to represent this app as an abstract model, of course, that's using state machines and statecharts. Then we're going to use transition analytics to determine the just frequency of how often the user goes from one state to the next state. [10:00] And then we identify the paths that could be adaptive and improve the user experience. And use the analysis from step two in that adaptation, feeding it back in real time. So we need a few concepts to learn in order to do this. First one is weighted graphs. You might have learned this in school, but if you haven't, it's actually a pretty simple concept. Let's say that when you're signing in, you could be in a success state or a failure state, in which case you're not able to sign in. [10:00] So what you could do is you can measure how frequently do users actually go from one state to the next? States? And so by doing that, now you have a weight on that edge and you can represent it visually like this. So our applications can be represented by these weighted graphs in a way that we could visually see. What are the most frequently traversed paths throughout the application that users would go through? [11:00] And so a path? A diagram that looks like this might end up looking like this in reality, in which we could see that some of the states aren't really traversed quite the same now. In visual Studio live share, I actually used this technique in the state machine that was used within VS code. And we were able to measure just how frequently users were either signing in, sharing a session, adjoining a session, whether we had a high frequency of errors or not. [11:00] In a very visually clear way, that connected all the dots, you know, based on how the application worked. Okay, so let's make this adaptive. Let's say that when the user clicks bad, we want to sort of give them the option to either fill out the form or not, so let's. Take a look at how that might work. What we could do is we can measure each of the paths taken and hopefully prints of the demo Gods here. [12:00] So, for example, when you click good and reload, we measure the count, so this is just a very simple example. But let's say I click bad and I see this form, I don't want to fill out this form, so I just exit. And so, let's say that I keep having a bad experience, but I really don't want to fill out the form. And so this isn't good because this is an incomplete user flow, but you see, when we have an adaptive user interface, we could actually, you see what just happened there. [12:00] We could actually redirect the user to a different state. And so this is easily done with these state machines and statecharts, because we have these hats represented as directed graphs. And now we could just measure in the debt in real time based on what the users actions are. [13:00] So this is a way of analyzing by transitions, and you could probably think of many more applications in which you could do this. For example, if you're on Instagram and you see when the user logs in that. The most frequent state that they go to next is the camera, instead of the gallery of the profile. You could shortcut the fact that when they log in, they go directly to the camera. Instead of whether they're whatever other default state that they might be in now. [13:00] Decision trees can also be useful in determining which paths we wants to take. And so I'm not going to go through this. But there's a lot of useful info in towards data science, which I'm going to be mentioning later in this talk as well. Now, prediction can also be done via shortest paths, which I'm also not going to get into. But we could use this analytics tracking within our services. So, for example, the next state? [14:00] We have this on transition callback, which this is the only place in which we're tracking analytics. Because that's the singular place in which all state transitions are represented. In other words, what this means is you don't need to litter your entire application with analytics or telemetry calls, you have one single place to. Do it. And so you would provide a target event, a source, and possibly the time stamp for that event to happen. [14:00] So let's sort of think about some future ideas for analysis as well. Higher order Markov models are a very interesting topic because what they represent is. Let's say you have a weighted graph in which you're in state A, and you're measuring whether your go to state B or C. So it might seem that overall, the user goes from A to B most often, but if they're coming from D, then they might go to see more often. [15:00] And so that sort of changes the dynamic of how the user actually behaves with the application. [15:00] Also store of a crazy concept. But using deep reinforcement learning within your applications. You could actually measure and train an agent to determine which path is the most likely path a user is going to take. And so that's done by the agents just experimenting and saying, I think they're gonna go here. And then what the user actually does is fed back into the agent, and they're either rewarded or punished based on whether the user actually takes that action. [15:00] And so we could feed that into our environments, which is the abstract model that we've created, and use that to sort of adapt the user interface in real time. So you might be thinking, wait, seriously, like, can I? Can we really use reinforcement learning in the browser while the user is interacting with the app? And yes, you can. You're just gonna have to sort of take my word for it. All right, actually found. [16:00] Let's see a demo. So if you remember from last year, I made this tic-tac-toe app using finite state machines because that's sort of my brand. But you can see that this is just a normal tic-tac-toe game in which player X player Oh, can play. But what you might not know, I'm a terrible tic-tac-toe player. All right, there we go, we drew. But what you might not know is that we could capture analytics on what the user is most likely to do. [16:00] So, if I turn on the frequency here, I could see that the user is most likely to click, for example, the center one. And then from here, we played a bunch of games here. And so, you know, I'll just play here. We could see that, even in bored states, that we don't really, we haven't really represented or played yet, that those are all grayed out. But when we reset, we could see that things that we played again. [17:00] It shows up, you know, in the frequency of each of the squares. So let's take this to the next level. Let's actually introduce a reinforcement learning agent and have it predict all of those unknown game states. So what it's going to do is it's going to behave like a human, so it might not be the best AI, but it will. It will try its best. So what it's going to do is it's going to play like a human. [17:00] And you could see those learning values over here on each of the squares, and I could play against it. Let's say I play here, and you see, it shows the most frequently traversed path, so now let's say I do something new, I go over here. It's still going to try to adapt based on how the users have actually played. [18:00] And so, in other words, we have created a reinforcement learning agent right in the browser. And I'm going to share this code after the demo. So, and of course, if it failed, I have a video to show you. So, the good news is that this does work, and the bad news is that when I was trying this out a hundred times last night, I got this. Fail to compile air and React, of course, but I got it because I had no space left on my device. [18:00] I'm not sure whether that's the fault of my analytics tool or my actual computer. So again, the idea is that we have an application, we represent it as an executable model, and then we use this metric tracking in real-time within our model. And potentially even feed it to something like Google analytics, KISSMetrics segment, do MixPanel, or even as your analytics. And then we feed that back into the application and feed it into these adaptive transitions within our app, and then we have an adaptive user interface that works in real-time. [19:00] So this is great in the process of having a great dev experience, but also using your dev experience to create a great user experience as. Well, because when we go from having the determinism of state machines to having the way to simulate how users are going to behave with our app through simulation and analytics. Then we ultimately can create a great user experience and go beyond the code that we just wrote to make the app work, and use that exact same code to actually improve user experiences. [19:00] So XState does have a lot of utilities and tools that I've created, and I actually just published XState analytics. But do not use it yet. Because I do want to figure out that memory leak problem, and so I would caution you against that. And also, the next version of XState 4.7 is coming soon. Thank you to Matias for really helping out with that. And so if you want to play around with that right now, you could just npm install XState at next. [20:00] And also the first talk to stay Gatsby you would be surprised to know that if you use Gatsby in the background, you are using XState, because Gatsby uses this for. It's very complex build processes, which are, of course, very, very important to making the Dev experience as good as possible, which it is working with Gatsby. So there's a lot of resources. If you want to learn more about working with statecharts and also the documentation, Czar a good resource to. [20:00] I actually included embedded visualizations right within the XState documentation, so go ahead and check that out. And we have a vibrant spectrum community in which we talk about these abstract modeling concepts such as state machine, statecharts. Petri Nets all this sort of stuff, even if it doesn't have anything to do with XState. It's just these old ideas of state machines and statecharts, which have been around for half a century, which are just coming to light now and being more and more relevant in the front-end community. [21:00] So ultimately, I want to make your code do more, I want your code to actually be able to analyze. Have tests automatically generated, have even documentation automatically generated, be able to simulate how a user behaves in real life and possibly even predict how users behave. So thank you very much, reactive Compton. [Applause]. David Cor she thank you so much. We have some questions for. You talking about DAF steps leading to various usage paths. [21:00] Do you think it would be useful to adjust the UI even if an app based on the previous user activity? [22:00] Have you ever interacted with an app that's just so complex and you find yourself doing the same steps over and over again? What if those steps could be automated? So I would say that yes. It is very, very useful to have these adaptive paths because ultimately the goal is to make the user do what they do faster and better. Can I write a test that is going through all possible UI states with AK state? [22:00] How does this approach work with manual and automated tests? Yes, you can use XState to do that. There is a library called X8 Slash test. And I actually gave a presentation that reacts Rally a few months ago talking about that. And it's called model-based testing, which has also been around for quite a few years. So, yes, short answer and the last question, is there a plan to adapt this predictive design through Microsoft products? [22:00] So I try to think X8 and Microsoft products. But there's only so much I could do. Maybe one day I could talk to them and hopefully do that. Thank you so much, David Gore sheet [Music].