# Mind Reading with Adaptive and Intelligent UIs in React

Canonical: https://davidk.dev/talks/mind-reading-with-adaptive-and-intelligent-uis-in-react/
Video: https://www.youtube.com/watch?v=ppvi09LIUnU
Speaker: David Khourshid (@DavidKPiano), creator of XState
Type: Conference talk · Event: React Finland 2019 · Year: 2019 · Duration: 25:02 · Views: 1231 (as of 2026-10-19)
Topics: Adaptive & intelligent UIs, React, State machines, Statecharts

## Summary
What if you could predict user behavior with smart UIs? In this talk, we will explore how we can make adaptive and intelligent UIs in React that learn from how individual users use your apps, and personalize the interface and features just for them, in real-time. With probability-driven statecharts, decision trees, reinforcement learning and more, UIs can be developed in such a way that it automat

## Key takeaways (quoted from the transcript)
- Use interfaces in reading the mind of the user in a way, and so this idea is called adaptive user interfaces. (4:00)
- If we think about our goals here, our goal is to collect information from the user to better provide a user experience. (11:00)
- I used a state machine to both model the user interface of how users logged in and out, and also collect analytics. (15:00)
- Our overall goal is to make applications that have this adaptive user interface have an executable abstract model of that system. (23:00)

## Chapters
- 0:00 Introduction
- 2:00 Clippy was one of the first adaptive user interfaces
- 9:00 If we think about our goals here…
- 12:00 I used a state machine to both model the user interface…
- 16:00 A user going from logon to gallery to camera

## Transcript
[0:00] Hello everyone, my name is David Khourshid. I work at Microsoft and this is my toothbrush, and there's a reason that I'm showing you my toothbrush. I know it's really random, but everybody is different, everyone has different abilities, different personalities, different uses, and these types of behaviors. Just like Luca was talking about human behavior, bees change over time. Alright, I mean, everyone here is different. You're, I mean, you're all pretty whites, but that's okay.

[0:00] But the point is that when we think about how we develop software and user interfaces, we need to consider everyone, no matter where they come from. And that's why ability and accessibility and internationalization is so important, and so the way we develop user interfaces in software. Right now is sort of like a toothbrush where we all share the same toothbrush, and that's not good because nobody wants to share the same toothbrush.

[1:00] We all like our own toothbrush, and in the same way, we all like our own user interfaces. So let's talk about the current state of the world when it comes to how we could, how we could affect change with regard to user interfaces and software, one common technique is a be testing. But that effectively splits the population of our users into two groups. Those who, you know, those who stick with the same software, and those who might like the new version of whatever app are working on.

[1:00] But the problem is that we don't consults those users when we're developing this. And so the way that we usually develop software is in closed meetings. And, you know, just by a handful of stakeholders that don't represent the population at large. And so this is why things like A B tests are ineffective. Because, essentially, you know, at worst case scenario, you're going to piss off half the people and make half the people be like, Okay, this is sort of cool.

[2:00] But whenever you change something, you are trying to enforce that same change on absolutely everyone.

[2:00] So all, there's something called funnel charts, and so if you've worked with things like KISSMetrics, segments, IO mix panel, you'll understand that. There's these funnel charts. Where you see how user conversion happens from step to step, and how users either continue through all the steps in the user interface or they fall off at. Certain steps. And the problem with that thinking is that user flows are not linear, users could do different things, users can explore the app in different ways.

[2:00] And so it doesn't make sense to track the metrics of just one simple flow through the app. And instead, you have to thread through all the possible ways that a user could use your application. So one of the recent projects that came out, well, I believe, last the area was highlighted at Chrome Dev Summit, I believe, or Google I/O, whatever it's called, is SJS. And so guest OS is based on predictive analytics inside the browser, working with your web pack apps to smartly chunk your bundles.

[3:00] In a way that, basically, when a user visits a page, then it uses Google analytics or whatever other metrics to determine what is the best. Next page to free to prefetch and if you don't know if prefetching is. That means that the browser silently starts loading it in the background, so that when the user ultimately visits that page, it becomes instantly loaded. And so that sort of the idea behind Google amp projects - I talked about Google, I work at Microsoft, so sorry, Microsoft if you're watching.

[3:00] But so, this is based on predictive analytics, where not only do we measure how often users go from one page to the next page, but we also look at a whole bunch of different properties on the user. Like, which users are more predisposed to go from this page to page A, or to page B, or two page C? Using those analytics, we've prefetch and then we load the page. So this is a pretty interesting project and it definitely plays a role into just the overall picture of making better.

[4:00] Use interfaces in reading the mind of the user in a way, and so this idea is called adaptive user interfaces. One not one misconception is that when we think of adaptive user interfaces, we think of adaptive design or layouts. And adaptive design and layout is related to how we could modify the app based on the viewport or based on the device where it should act and behave differently, whether it's on an Apple watch, or a phone, or a computer, or a big projector.

[4:00] But this is not exactly what we want. To think of when we think of adaptive user interfaces, instead, we want to create software that adapts to the behavior of people, whether it's individual or groups, and not just devices. And so, to give you an example of an adaptive UI, the probably the most famous example is this little guy right here. Y'all remember Clippy? And Clippy was one of the first adaptive user interfaces.

[5:00] In fact, early versions of Microsoft word had many of these instances of adaptive user interfaces. Especially with the toolbar showing and hiding certain items depending on which ones the users were using the most. Now, clippy used use predictive analytics and sort of machine learning at the time to want to determine what the user was doing and make suggestions based on that. So it was. It's an assistive UI, and it's also an adaptive UI as well.

[6:00] So here's another example of how we could use of death of the eyes in changing the way that we develop. Like a common user interface, patterns, so I'm sure you've seen this pattern before. When you delete something in an application, you could click yes, you could click no. What? Because it confirms, like, do you really wants to delete that? But adaptive UI is different of death of you. I would take a screen like this, measure the analytics of the user and say, how often do they click?

[6:00] Yes? And so we could. We could say that most of the users are probably going to click yes. Because it's a big red delete button and you don't usually accidentally click it, of course, there are those users who sometimes accidentally click it and they're like, Nope, I didn't want to do that. So that's why a better user interface would be one where we predict that the user is going to choose. The most common action, which is delete it.

[7:00] And of course, we give them a chance to undo it, which is this nice faded button over there and after a second or two, we just we just delete it. So in this deleted phase, were sort of not deleting the file yet, or not deleting whatever they want to delete. But instead, we're giving the user an option to undo their choice. So one of the more recent adaptive you eyes have seen and predicted.

[7:00] You eyes is Google and not Google inbox. I'm sorry too soon. Rest in peace, Gmail, and so Gmail. I don't know if you've tried this, but it's really cool. It predicts what you're going to type based on the context of what you're typing. And it does this because Google has so much data on all of the email sense. Well, I guess that means they're reading our emails, I don't know anyway. So it'll predicts the next thing that you're about to type, and usually it's correct.

[8:00] It's been scary, accurate in, you know, me typing emails that takes me superfast, and I'm guilty of being that person who. When I get email and Google inbox or, sorry, Gmail, I will press one of the predictive choices there. Because I'm like, number one, I'm too lazy to type it out on my phone. Number two, it's it's easier to just, you know, send one of those default choices. However, developing adaptive user interfaces is definitely challenging, and so there's a lot of things involved with adaptive user interfaces.

[8:00] Such as the intersection between human-computer interaction, artificial intelligence, and also user modeling, and you could see all of these things sort of combined. Here definitely is a lot more than taking a mock up, putting it in, React or view or whatever, and just implementing it. So how could we do this with our current development?

[9:00] So we have to shift our mindset and think about model driven developments. This is something I've been talking about for a couple years now and something that I definitely talked about last year. If you were here at Reax Finland, I gave a talk about state machines and statecharts and how they could really help us. You know, model our user interfaces in the very declarative way to prevent bugs and to visualize. And all of these other benefits.

[9:00] What we're going to see, that model driven development really has a lot more benefits than just helping us developers. So when you think of a model, think of something like a user flow that the designer might provide you. But these schematics, these specifications for how your application should work are essentially a way that we could develop apps. And, and well, I'll just show you so finite state machines. This is something I talked about last year, but.

[10:00] In case you need a review. Basically, what I was trying to say is that we could, we could develop our apps so that they are these finite state machines and so on. You could imagine an app where you sign in, that would be an initial state, and then we would have our finite number of possible states, such as signing in, going to the home or going to the profile. We have a finite number of events as well, such as being able to submit the sign in, and then whether it's a success or your unauthorized, it takes you to a different state and then from the home screen, you could go to the profile screen.

[10:00] So just like the user flow over here, we could create state machines that describe our app in sort of the same way and actually make them executable. And this is something I was talking about last year. So yesterday at my workshop, there were about 20 25 of you at the workshop. We created this screen right here. This is a very simple flow where you can think of it as a feedback app.

[11:00] Like little components. That sort of pops up and says, how was your experience? The user could answer good or bad. if user answer is good, it says thank you, if you user answers bad. It goes to a forum where it tries to gather more information from the user and then it's submitted. Now, if we think about our goals here, our goal is to collect information from the user to better provide a user experience.

[11:00] And so don't think of this is just a mock-up, but think of this as a goal to accomplish with our software. So, if you're not familiar with X 8, X 8 is a way of creating finite state machines and statecharts. That that could model the user application and the user interface, and all of the requirements involved.

[12:00] And so it's essentially just an object, just a JSON object. Where we look at the state, in this case, let's say we're on the questioned state. We look up the events that just happened, such as click good, and then we navigates to that screen. Now. Something fairly recent that I'm I basically just released a couple weeks ago, is XState React. And XA React provides a used machine hook where it's one line of code, so you take this useMachine hook.

[12:00] You take your machine, you put it in useMachine. And just like the useReducer hook. You have the current state and you have a send, I like using send instead of this patch because it's less to type. And there you go. This is all you need to manage your states, so take that Redux. And so I talked about many of the benefits of using state machines and statecharts in the past, such as determinism.

[13:00] We could know exactly which state will follow, given the current state and an event. But we could also take that abstract model of our user application. And we could visualize it, and we could also communicate between developers, designers, stakeholders and more with those visualizations. But those are all very developer centric. And so the more benefits we see, the more we see that we could actually get to benefiting the user experience such as testability, making sure that we test all possible edge cases simulation in actually executing those edge cases and seeing how it actually works in the live system.

[13:00] And even analytics, which we're going to be talking about. So here's a game plan. In order to make adaptive user interfaces, we first need an abstract and all of our system. Now, statecharts and state machines are the popular way of doing this. I've also seen people using Petri Nets, that's fine, it doesn't really have a strict standard, but yeah, it's just another thing that you could consider. So then we capture transition analytics, and so that means of what happens and when it happens and where it happens as well.

[14:00] And so these transition analytics include things like user information, the state that the user is in and the events that occurred and what time and occurred. So then we also take our abstract model, which again is modeled as a state machine. And we see which parts of it can become adaptive and change. And then we use the analysis that we collected in step two, or that adaptation. So the first concept to understand when it comes to all this is weighted graphs.

[15:00] So let's say that we have our part of a state machine, and we have. A signing in state and two things can happen, either a success or an error. So we could collect analytics on these two transitions, whether it goes from signing into success or signing into error, and determine the rate of each one of these. And so these are called edge weights, and so this makes a weighted graph, and some of you might know this from University, so bringing back those memories for you now.

[15:00] These weighted graphs can also be visualized as well, and we could see which paths are more likely to happen based on these analytics. And so I did this about a year and a half ago, and Microsoft using when I work on the visual Studio live share team. I used a state machine to both model the user interface of how users logged in and out, and also collect analytics. And so, using D3, I was able to visualize this chart and see the flow of people, how many people share the session, how many people joined a session, and what was the error rates as well.

[16:00] So let's take a look back at our feedback app and let's decide that.

[16:00] One of our business goals is that we do want to capture information from the user, so we want to capture whether they clicks good or bad, we want them to tell us why, and so one way we could do this. Of course, we could take them to the thanks screen, which they're likely to close it. But we also have a give feedback button where they could actually go and give us feedback, and of course, clicking bad makes them give us feedback anyway.

[16:00] So, by measuring the analytics of the transitions and determining which path the users are more likely to take. We could actually adapt this UI, and I'll just show you, and we could, we could make it so that it adapts to the user. So in this case, I'm not using any analytics platform, I'm just using local storage, so it's a very local solution. And so how it works is, let's say, as a user, I click good.

[17:00] My experience was good. And then I see this form, I'm like, Ah, I don't want to fill out the form. It's like I had a good experience. Don't ruin it for me, so I press escape. And I get out of there, so the next time this shows up, I click it again. Of course, it shows me because I'm like, I'm getting really tired of this. I press escape, reload, then I press good, OK, it's asking me again.

[17:00] So now, after a while, the system adapts and it says, thanks for your feedback. And this is what we want because it adapted to the behavior of the individual user. In in that, it gives me the thanks screen, but it also gives me an option to provide feedback. And so now, every time I go on here, it will do the same thing. But if I, if I decide to provide feedback and I'm like, you know what, my experience is so good, I actually want to tell you about it.

[18:00] Then the application will adapt and it will give me back my screen. And of course, if I, you know, go and X it again, and eventually it will, it will go back to that screen. So that's just an example of an adaptive user interface. Alright, going back here, well, that should be adaptive. So the tricky part about the adaptive user interfaces is that it sort of gets into the realm of probabilistic or non-deterministic state machines.

[18:00] Because we could have two possible transitions, and those two transitions happening on the same events need to be based on something. Because, remember, one of the rules of state machines is that you can't be in more than one state at the same time. And so that's where analytics come into play there. So there's two ways that we could do this analysis. Like, once we have our abstract model of the system, we could first determine the analytics of how often.

[19:00] These paths are taken either by the individual user or a group of users, or certain, you know, demographics of users. So, for example, we see over here, let's say that we're making some sort of Instagram application. We log in and it takes us to the gallery by default, and then from the gallery, the user is. Either the user could either scroll through the gallery, or they could go to the camera, or they could go to the profile.

[19:00] So we measure those transitions and we see that going from login and sting in the gallery we have. A 90% chance of a user going to the gallery, and then a 31% chance of the user staying in the gallery. Now, a user going from logon to gallery to camera. We have a higher chance because we see that users have a tendency to go directly to the camera. And then over here, logging gallery profile, we see users are less likely to go to the profile.

[20:00] So what we could determine here in the way we could adapt our UI to fit the users needs is we could see that. Since the user is more likely to go from the login screen to the gallery and then immediately to the camera. We could identify that desire path, which is, you know, what it's typically called, and go directly from login to camera as part of a an adaptive user interface. And so, just like the feedback app you saw, you know, we could, we could do all that.

[20:00] Oh, we could also analyze actions as well, for example, trying to decide when ads are supposed to be displayed. We could score those actions and depending on certain factors, we could see whether ads should be interspersed or on top. Sorry, I'll show you that over there. And so, you know, we could measure these based on what's known as decision trees, and so these decision trees are just data from the user where we determine things based on that data.

[21:00] And so this is very important in reinforcement learning and machine learning as well. Because we take a lot of data and then we determine based on certain factors, what should happen based on the data that we have. And so it's a very statistical model of deciding what to do, you know, based on the user. So using state machines, we could, we could do graph analysis on these, since these are, you know, directed graphs, and they do fit that model.

[22:00] So we could use shortest path algorithms to determine where users are going to go, and we could also use analysis analytics to track this. Is XT it again, where transition, transition statistics. So, for example, we could give it the target, the source, events, and timestamp. And we could determine based on how many users go from one state to the other, the likelihood that they will travel from for. Example A to C.

[22:00] And so we could see here that the shortest path is not ABC, but instead it goes through a DEC. And so that's the intuition that we have to keep in mind is that the path weights matter. And so there's there's a whole bunch of other AI ideas that come along with this, including higher-order markup models. Where we could get more accurate predictions. If we don't just determine what the user might do, but what the user previously did as well.

[23:00] And so you could see that the that the prediction changes based on that. So, reinforcement learning can also be used where in agents, whether that's a bot or just you know something. We program, can interact with an environment and get rewarded based on accurately predicting what the user did or did not do. And so our overall goal is to make applications that have this adaptive user interface have an executable abstract model of that system.

[23:00] Track metrics in just, well, four lines of code, feed those metrics into your analytics platform of choice and determine which transitions can be made adapted. And that way we could have these adaptive UI, and so there's there's many benefits to adaptive user interfaces. And I'm running out of time, so I'm just going to listen here, you could talk to me if you want to know them. But there's also a lot of modeling considerations, such as such as whether you should fit this to a user or fit this to a group of people.

[24:00] And so in XState, I've been working a lot on these sort of ideas because it's not just a state management library, it's a lot more than that. It makes these abstract models for your applications, and I just released XD React and X8 graph, which gives you those shortest paths in simple path algorithms. And there's a lot more coming down the pike, such as X8 View, a visualizer, analytic, simulation, and testing utilities as well.

[24:00] So I'm going to leave you with saying that. Our goal should be to make your code do more then it's already doing, and to also improve the developer experience and the user experience. So thank you very much. React Finland [Applause].