Mind-Reading with Adaptive and Intelligent User Interfaces

Hello everyone. I'm very excited to be here in Alante, Spain. Beautiful city. And I'm excited to see half of you here because it is 9:00 a.m. in the morning. And you know, but the entire point is that users are unpredictable. You know, users, developers. I don't know whether you're going to wake up and see my talk or maybe see it later. That's totally fine. My talk is titled mind greetings adaptive and intelligent user interfaces and my name is David Khourshid I work at Microsoft my Twitter handle is not on there but it's @DavidKPiano also I want to thank all the sponsors for this conference because without them this slide would have been blank and really awkward.

[Music] so let's talk about the different ways that users behave. Let's say that you have a button on your user interface such as fetch user data and let's say that my computer doesn't behave exactly the way I want it to. There we go. So you click this button, you are waiting for some sort of API request to resolve and then eventually you get some data. All right. But what if we want to actually respond to how the user expects the application to behave? And what if for example it's taking a very long time to load? You might want to show a loading spinner or something like that and eventually it all comes to you and you know you have the data. However, like I said at the beginning, users are very unpredictable. And so, I know personally I have done this many times. You all have probably done this many times. You just keep clicking that button hoping that for some reason the data loads faster the more you click it.

It doesn't really work out that way. So as a developer, you see this behavior and you think, you know what, I should probably disable that button so that eventually the data comes back and you think, okay, I have outsmarted the user. They will not click the button too many times and make too many API requests. However, I'm a developer as well. I could go into the dev tools, und disable that button, and then make as many requests as I want. So yes, everyone is different. We all come from different backgrounds. We speak different languages. We we have different needs and we might not even all be using the same devices when it comes to interacting with web applications or websites. And so this is why in my opinion I feel like AB testing is ineffective because building the UI is one thing. Doing it to a specification is one thing but we also have to analyze how that specification really fits the needs of users and whether we need to go back and change features, add features, just change the behavior of the app to adapt to our users.

And right now we do this manually and we do it using AB tests. And it doesn't matter whether you're making a website or a web application. Sometimes you will get these huge tests where it's like, you know what, we might want to add this feature. We don't know if it's going to work for the user. And so the other problem with AB tests is that it's very much a blanket statement. It's a it's a way of saying, "Okay, this is something that maybe 60% of our users like, so we're going to do that." But guess what? That makes 40% of your users pretty angry. Also, user flows are seldom linear. So, we can't really use things like funnel charts and things like that in order to say, "Okay, how many users are going from this date to this date in our application to the other state?" because there's so many different ways that users could use your application and it's helpful to understand this.

Play from 0:00

So there is a there is a library called guest.js which has been around for a while. It's I think it's made by Sam Sakone at Google and guest.js is a way for predictively just selecting different pages and preloading pages on your application. So what you do is you set it up within your Webpack app and it could be React or View or any anything like that and it will allow you to based on Google analytics and based on what the user is used to doing like whether they're used to going to some profile page after going to the dashboard or just navigating a certain pattern of pages to predictively prefetch those pages and make loading a lot faster.

So this is a really solid first step into what is known as adaptive user interfaces, also known as AUI. So what is an adaptive user interface? Just like a responsive user interface adapts to your screen size and your device width and height capabilities, an adaptive user interface instead adapts to the user. So that might be showing some things based on based on what the what the user prefers or doing different behavior based on based on the user. So in short it adapts to people and not just devices. And so Clippy is a very good example of an early version of this. as you would type in Microsoft word or Excel or something, Clippy would pop up, recognize what you're doing, and provide sometimes helpful actions based on based on what you're doing. So, another example of this is let's say that you have a simple UI where you have a delete button, you click delete, and you want to make sure that the user really wants to delete whatever they're deleting.

So, you would show some sort of prompt that says, are you sure you want to delete this? yes or no and you know depending on what they click it will delete it or not. However, let's assume that the user always clicks yes because it's like, okay, I am intentionally clicking this delete button. Trust me, I know what I'm doing. Then we could create a user interface which adapts to that and says instead whenever you click delete, it shows deleted and it might provide an undo button so that you could still have that cancel behavior. But this is easier because it provides one less step for the user to interact with the system. And so this is also an example of an adaptive user interface. You might have noticed a feature in the new Google, not Google inbox, but I guess it's Gmail right now where it adapts even further. It takes it takes these predictive statements and allows you to just autofill based on you know based on what you're typing and it is really smart.

Play from 4:00

I've used it a lot of times because I'm really lazy and I don't like writing emails. So I will just do this all day. I don't even know what it says. I just keep swiping right and hope that it says something incoherent. So developing adaptive user interfaces is very challenging though because this is even though this has been researched for years, it's still very much cutting edge way of developing user interfaces and applications because there are so many things to consider. You're no longer developing an app where it's just you have this screen, you click this button, now you have this screen. Now you have to think of other things such as you know such as artificial intelligence such as how we could adapted what are certain points of adaptability. So it's not an easy thing to do and that's why we had to sort of change our perspective and change the paradigm of the way we go to what's known as model driven development. So model driven development is a way of developing your application. So that instead of just coding things, you don't code first.

Before you might have seen things like flowcharts and user flows. These are the types of things that you want to create first. So you could have an entire schematic and diagram of the entire system that you're trying to develop. And so this is what's known as an abstract model. It's abstract because it doesn't really work. It's just a visual of how your app is supposed to be presented. And one way that you could make these abstract models and turn them into something real and something you could use within your app is something I've been talking about for a while, which is finite state machines. I mean, seriously, if you type state machine on Twitter, I will find you. I will respond to you. That's just my thing. So, a finite state machine, in case you're not familiar, has five parts. The first one is an initial state. Every finite state machine has a state where it starts. It also has a finite number of states which is the possible different states that you know you could go through an application or maybe the possible different states of a component.

So again this is abstracted events which level of abstraction you're working with. There's also a finite number of events and these events could come from the user or they could come from the from the application itself or could come from external things such as promises, async requests, etc. And then there's this thing called transitions and these transitions map states and events to the next states that they go to. And this is this is deterministic which means that when we click when we're in the signin states for instance and we click the submit button we know we're always going to be in this signing in state and it's never going to change states randomly and it's entirely predictable.

And this is a very crucial part to being able to develop adaptive user interfaces because we want to be able to predict all the different ways that the user could use the application. Now the fifth part is something that's not entirely applicable here but it we have a finite number of final states which means this is where the app is finished or the system is finished and can no longer transition to anywhere else. So there's also the notion of statecharts because if you've ever developed with finite state machines you might think okay this is cool this really helps but there becomes a point where the number of states becomes exponentially large and you want some way of grouping some way of nesting some way of controlling the chaos that all of these states bring. It's a it's a visual formalism for complex systems created by the computer scientist by the name of David Harel in 1987 not 97.

And it was actually created to mitigate these exponentially large combination of states that he was seeing with finite state machines. And so this is a very scalable way of creating these models of your application. So statecharts have a few parts as well. First is actions. You could have entry, exit, and transition actions. Entry means that anytime that you enter a states, that action is performed. And you could think of actions as side effects. Transition states are whenever you go between states like between idle and searching, you could attach a side effect to that transition. Guards also prevent unwanted transitions from happening. So for example, if we wanted to perform a search, we might want the search query to be to actually be a valid query that we could use. And so if that condition or guard evaluates to false, the transition will not be taken. So to simplify, there's also hierarchy, which means we could nest dates together and group them and treat the parent state as one big state and just nest infinitely.

And this really controls the complexity of our applications and our models. In in a similar vein, there's orthogonality, which means that we could be in more than one region at the same time. And there's history or remembered states. So these are just features of statecharts, which aren't exactly important to what I'm talking about, but it's it's important if you want to get deeper into statecharts. So, one interesting thing about statecharts that I find myself using often is hierarchical states where, for example, we could group states together that have very similar behavior. And this makes modeling your application a lot easier because you could instead of creating many transitions with every single state, you could group similar behavior. And for example, if we want search to occur again after we're in the success or failure state, we could just group both of these states in the search state and be able to do that. So in code, actually I didn't really explain that yet, but let's take a look at an example of a finite state machine in our application.

So let's say that we wanted to create this very simple feedback form. It asks how was your experience? You could click good or bad. And if you click bad, we want to collect information from the user telling us why it was bad. And they could choose to submit it. And it says, "thanks for your feedback." And let's assume you get this from your designer or project manager or someone whoever gives you this type of thing.

Play from 7:00

And it's something that you have to develop. And so you're like, "Okay, I'll develop it." Think about how you would develop it. You might you might have a bunch of if then statements. You might have like something that says, "Okay, which screen are we on?" But to do this in the more formal way, you could use a finite state machine. The typical way of doing this is with switch case statements. So, for instance, you would you would just determine which state you're on first, such as let's say we're on a searching state, and then you would say which events just occurred, such as resolve, and you would return with what the next state is going to be.

Or if there is no transition defined for that event, then you just default to whatever the current state is. Now you might have recognized this if you've done Redux before where you would initially start instead with this. So this is sort of a little shift in thinking too. In Redux the natural pattern I guess the most common pattern is to start with switching on what your action is. Here I substitute the word events for action because it's the actually right word. Action refers to some sort of side effect. But over here it like we can't really map actions directly to what the next data is going to be. And that's because it could be many different things. And so what you end up doing is in your reducers you have if then statements just littered all over the place just to figure out the next state. So that's why I recommend always basing it on what the state is first.

So an example of this with our feedback form, we see that it works like so. And I'm using a hook over here. It's called useReducer. And so this useReducer hook. This is extremely helpful because instead of having a use dates hook with useReducer, you could have this notion of a finite state machine within your React app directly. You could have like this the current state and the send function. And the send function allows you to send events to your state machine, which is just those nested switch statements and allow you to easily just go through the app and all that. I'll probably have to rerun that. There we go. And so this reducer is exactly as I explained where you have your states such as whether you're on a question state, what events happens, and then it tells you what the next state is. And so we could express all the behavior that we want in our application just based on this sort of pseudo state machine that we have with switch case statements. And we could say bad, tell us why, no, and then you could go on.

So, so there's another way of doing these finite state machines and this is the way that I prefer much more. It's with object mapping. I do this because it's a lot more tur. It's a lot easier to look at and you're going to see in a little bit that we could actually take this, analyze it, and visualize it too. So the difference is that instead of using nested switch case statements, we just use objects mapping and we're mapping certain values or certain keys to values and those values might also be objects as well. And so we can have a simple transition function that says let's take the state that we're on such as the form state and then let's see what events just happened. And so we're looking on the on property to see if there is a mapping between the current event and the state that we're on to give us the next state. For example, thanks. And if there is no mapping, then we just default to the normal state.

I find this so much simpler than switch statements and that's why I like using that. Now XState is a library that I created that basically fits this exact same paradigm with objects. The only difference is that you would wrap it in a machine factory function and you pass in that object into it and you get a whole bunch of cool features. So this you know looks exactly the same where you know we would have our mapping of states to the event that happened and the next the next state that it's going to be in. Now with React we have this hook too. It's called the used machine hooked and that's from the XState/React package and this allows you to basically use state machines in XState just like you would useReducer in normal React. So the output is exactly the same. Instead of useReducer you would useMachine pass in machine and you get two things. You get the current state and you get the send function. And so you could match based on what the current state is just by current matches.

And you could also send events directly to the machine again just like useReducer. But you have the benefit of using full state machines and statecharts which allows you to do things like side effects orthogonality nesting etc. All right. So, now that we've described what state machines and statecharts are, and hopefully you have an idea of how you use it in your apps currently, let's go over the game plan here. First, we're going to create an abstract model, which we're going to use state machines and statecharts to do this. We're going to analyze the transitions and this is done by assigning weights to the different transitions, which I'll talk about in the next slide. And we're going to identify which parts of these of the application we can make adapted, which parts we could change based on user behavior. And then we use analysis to actually execute on that adapt adaptation. And this means that we don't have to go back and manually change the behavior based on what the user is doing.

Instead, the app could do it automatically. So first weighted graphs because we're designing our state machines and statecharts as directed graphs which you know we have our nodes which are states and our edges which are the transitions between states based on events. We could actually assign what's known as weights to them. All right please weights. Speaking of weights, well, that goes all right. And so we could say that for instance, when we're in the sign-in state, we could say that 90% of our users eventually reach the success state while 10% of our users reach the error state. Now, because we've assigned weights to these edges, we have what's known as a weighted graph. And so we could determine which paths are the most commonly taken. And so this is an important realization to make because when you are designing your applications and developing them there's many different ways the user could use the app and it's important to understand what are the most common ways that those are used.

Play from 14:00

So your application might look more like this. And then you have a very clear picture of how your app is actually being used. And even just seeing something like this, you can understand how you can modify your application to fit the user needs such as optimizing for this path right here. And so I did this a while ago in visual Studio live share. Raise a show of hands. How many of you have used visual Studio live share? All right, that's about like three people, but thank you so much. And so in visual Studio live here it's a way of collaboratively coding between you know between many people and so I what I did was I have I had a state machine in the application which is in visual Studio code and I analyzed the just the transitions between the different states that it could be and I was able to see how many people were for example successfully signing in and what they did after they signed in.

So we saw that about an equal number of people both shared a session or joined the session and we could also see that sometimes there were errors but it wasn't too common and so these types of analytics are extremely helpful when developing your application. But besides that we could use them for actually adapting to the application. So an example of this, let's take our feedback form that we that we originally had and let's actually make it a little bit adaptive. So let's say that when the user clicks bad and goes to this tell us why form screen that they don't really like filling out this form and they're really annoyed by it and they only want to fill it out sometimes. So what we want to do is we want to determine whether we should show the form or whe whether we should show them a thanks for your feedback screen and instead give them an option to fill out the form when they want to.

So let's take a quick look at that. So you could see over here. So what I did here was I took that exact same feedback form. I used state machines and statecharts in order to model it and I use those statecharts in order to actually run this application. So let's say as a user when I click add it says here tell us why and you could complain but I'm like you know what I don't want to fill out that form. So, as I keep using the app, I, you know, I, for whatever reason, I'm having a really hard time with this app. And it just it, you know, I just keep saying bad, like I'm having a bad experience. So, care. No, I don't. I'm just angry. So, all right. Nope. All right. So, eventually what's going to happen is it's going to take me to this screen. Now notice I didn't do anything.

I didn't have an option to say please stop asking me why. I feel bad. You know, if any of you are in a relationship, you might relate. It's like I just I just feel bad. I don't want to talk to you about it. But now it takes me to the screen where it says I can provide feedback instead. And then I would provide feedback like so. And so let's say that for whatever reason I'm like, you know what? Now I want to keep I want to keep telling them, you know, why it's bad. So now when I click bad and I provide feedback and I actually say something, eventually it's going to, and I can just fill out nothing here, eventually it's going to take me back to that screen.

Play from 21:00

So, as you can see, it adapts to my behavior as well, which is which is really good. And I could say good and yeah, see, now I have a better experience with the app because it adapted to me. So, that's that's one way that we could use adaptive user interfaces in applications in order to model just these different ways. So, there's there's many different ways that we could actually let me just go full screen here for a minute. There's many different ways that we could actually analyze the different the different ways that we could use the app. So let's say that you're designing an Instagram type app. You could assign weights to the different events and the different transitions that occur between the states so that when the user logs in, we could see that the user the user has a much greater likelihood of going directly to the camera. And remember this depends on the user.

So, you can't just make a blanket statement and say every time a user logs in successfully, we want the user to go to the camera view. And so you might have noticed this with Snapchat or something. Once you open the app, for a lot of you, I think most of you, it just shows the camera right away, which if it's on selfie mode, it's a little unnerving, but you know, and so we can do this by analyzing these transitions and seeing that because we have a weight of 0.9 and 0.57, the user is much more likely to want to go to the camera. And so we could optimize for that. We can also optimize based on the actions done. In this instance we want users to click on ads you know just taking the same Instagram example. And so we can see that when scrolling through the feed if the user does not want to really click on the ad then they will scroll. And so by scrolling we would assign some sort of score such as negative -1 or if they tap the ad we would assign a positive score such as positive 10.

And so based on what the user behavior is, we want to optimize for which type of behavior is going to allow the user to click ads the most. And so we see over here that when ads are interspersed and the user scrolls, then you know, we're less likely to show the ad or we're less likely to have the ad clicked. So instead, we might want to decide, do we want to show the ads at the very top or do we want to show them in between the screen? And so again, not one sizefits-all. It depends on the user. So we could also use utilize what's known as decision trees to help us make decisions based on these analytics. And so this could be contextual data and we could say that it really depends on the type of user, whether they have a lot of friends or a lot of connections, whether they're posting frequently, where they're located in the world, because all of these different things come into play when you are designing adaptive user interfaces.

Play from 25:00

So we could use prediction also via shortest paths. And because we have this model of our system, we could analyze based on the different weights and we could determine which paths through the application are going to be the most commonly taken. And the way you could do this, especially with XD or if you're using any other state machine abstraction, is that because everything works inside of the state machine where you just have a transition function, you could you could do analytics tracking just in one single spot. You don't have to litter your entire application with, you know, sending analytics or sending telemetry. Instead, you could do it right in the transition function over here, giving it things such as a target, which is which state you arrived on, which state you came from, and you could also give it parts of the events where you remove personal information, and also a time stamp because you want to know at what time they made that request or also you want to know how long did it take to go between states and where the user was spending the most time.

So these shortest paths can really help us determine because it it's not always the shortest path as in the least number of seconds that the user will take but we have to determine it based on weights as well. So we see here that to go from A to C it you know it might seem shorter but actually it seems like users more commonly take the A to D to E to C path. So that would be the shorter path. And there's other concepts such as higher order markup models which complicate this a little bit more. And I'm showing you this just for illustrative purposes. It's something to think about. But let's say that we're in state A and we want to know whether we should optimize for going to state B or state C. We could see that in general going from state A to state B is the most common path.

But this might change based on which state the user came from. So now we could say if user came from state D then they're more common to or they're more likely to go to state C. So this is another important consideration. Look it up. Variable order mark of models. It's a really long Wikipedia article. So only if you're really interested in this. But also there's there's the introduction of reinforcement learning and machine learning to help guide how we could model the user behavior in a more automated way. Instead of manually checking and being like okay what types of users do I have you could use reinforcement learning agents in order to do this based on all that analytics data that you have. And so the idea is that when we create an executable abstract model of our application, we could track the analytics of how users are transitioning between states and we could capture a whole bunch of useful information from that and send it to something like Google analytics segment.io IO or mix panel and feed that data which is raw data directly back into the application and creates adaptive user interfaces.

And so this is a very powerful and a very exciting sort of new way of thinking about developments because we're not just creating a one-sizefits-all user interface. Instead, we're creating user interfaces that are tailored for the user experience itself. And so this is why I really recommend you learning state machines and statecharts is because not only do you have a better developer experience but you have a better user experience too through the same mechanisms in which we built these models. So adaptive user interfaces allow you to create these personalized systems for your users. It also prevents information overload because you could adapt to how your users think about your application. And you could also provide intuitive guidance as well because again not every single user is the same and some other users might need help doing something while others might be a power user and know how to do it a lot better. And it also helps the user understand the tasks and goals based on based on how the application is and so you could tailor the application to what the user wants to use and you know do in your app.

There are modeling considerations such as whether you should whether you should create these adaptive models based on smaller components or the entire app at large. Whether you should do it based on prediction or feedback in which feedback is directly asking the user how do you want this application to adapt based on based on you. And so also you could use realistic data or you could use those machine learning models, reinforcement learning agents like I talked about in simulating that data so that you could have sort of an automatic way of testing how adaptive your app is and predicting how users are going to behave in your application. So it might seem like I'm talking about like just another way of using the future of state management, but none of this is anything new. This is based on something called SCXML, which has been around since 2015. And it's an XML way of describing statecharts and state machines, but none of us like working in XML anymore, except Newslash. JSX is basically XML. So I guess we still like XML a little bit.

And there's tools such as QT and Yakindu which use SCXML in creating applications for desktops, for cars, for embedded devices, so many other things and it uses this exact same SCXML language.

Play from 28:00

So by doing something such as using XState, you're using basically the exact same language. And I have used or XC was used in many places such as React training. It's all over Gatsby which raised $15 million. That's not because of me. That's just you know they decided to use that. Also in the service work game X data is also used. And there's also one other important advantage of using these statecharts and state machines and that's visualization. So with visualization, we could take that exact same state machine and we could visualize it in our applications and simulate them just by clicking through them and being able to see events. And so eventually we could we could use these exact same visualizations to show the different weights between the states as we go through and as we simulate our state machines. Just like so. So, you know, look for that coming in the future.

There's a lot of advantages to using statecharts as well. One of my favorite examples is NASA, which used statecharts for auto coding the Curiosity Rover, which by the way, this is a $100 billion project. So, imagine if you had one shot at deployments for a $100 billion project. You're going to want to not mess it up. So they use statecharts for visualized modeling precise diagrams so that not only their engineers but any other stakeholder even if they're non-technical can see the behavior of the Curiosity rover and it allowed for automatic code generation comprehensive test coverage and accommodation of late breaking requirements and changes because they could see when a feature needed to be added or something needed to be changed or removed they could see exactly which part of the statechart needed to be changed because you know it's just arrows and boxes and when one of those arrows is removed it's like okay this box needs to be removed we need to reconnect this arrow somewhere else and then we can regenerate things such as test and code so that we still have a very robust way of working with our logic and making sure that no matter what situation this rover is in everything is going to work absolutely fine.

There are disadvantages though and even though this is a cuttingedge sort of way of developing user interfaces, there's a huge learning curve. I'm not going to lie to you. It's a huge learning curve because you aren't starting with code. You're starting with a model. And this modeling requires planning ahead. We can't just move fast and break things. Now we have to stop, diagram things, write things on, you know, pencil and paper or whatever modeling tool you want and think about all the different states in our app. And of course, not everything can be modeled yet. So you can't just apply this as a blanket solution to everything. It's also that there's also a complexity trade-off too. When we do normal code, the more code the more requirements we have, code grows linearly. With finite state machines, it grows more exponentially, which means that the more states and the more requirements we have, the exponentially more states we might have in our application.

Statecharts are instead logarithmic, which means that the more requirements we have because we can manage them better and group them, it's it's going to be a lot less complex as your app gets more complex. So there's a lot of resources such as the world of statecharts which is an introduction to how statecharts work in real life applications. There's an entire community where we talk about applications of statecharts and state machines in our application in real life apps whether it's React apps view Angular or even things such as embedded devices and really cool things like that. And of course I have documentation for my library because I would be a bad maintainer if I didn't. So XState there's a lot of tools that I'm developing for it. I just released XState test and I will be releasing an analytics tool eventually. So there's a whole bunch of tools and in short I'm telling you all this because I want you to make your code do more than just developing the application logic but instead being able to visualize, analyze, simulate, test and create adaptive user interfaces.

So, thank you so much for your call today.

Play from 34:00