Write Fewer Tests! Model-Based Testing in React

All right. Thank you. So, they say that a good software engineer always looks both ways before crossing a oneway street. Now, I don't think I'm the best software engineer because I always look one way even if it's a twoway street. And I don't know if you know this about America, we drive on the right side of the road, and over here, you drive on the wrong side. Needless to say, I've almost gotten hit by a car almost twice. And I'm excited to be here alive, unscathed and talking to you about probably what is one of the most exciting aspects of frontend development today: Testing. Which is just like right below forms, you know? [ Laughter ] all right. So, testing. Making assumptions, like which way to look in the road, making assumptions is a dangerous thing. Because users are unpredictable. We have no way of knowing how they're going to use our app.

Even if we want them to use it in a specific way. A few months ago, I put out a survey asking what people dislike most about testing apps. To be honest, it's not the most fun thing in the world. And the majority of the people said writing them was the most difficult part. There's updating and determining tests, figuring out why tests fail and discovering all of the possible edge cases. So, today we're going to fix this problem. But at the same time, we're going to fix all of these other problems as well. Now, show of hands. Who think about your last project that you've done? How many of you have written at least one test for it? All right. How many have written at least ten? Or a hundred? Or a thousand? Okay. All the hands how many of you have written no tests for the last project you've done?

Play from 0:00

Shame on you. [ Laughter ] so, despite how many tests we write, how can we be confident that our application is going to have absolutely no bugs? The easy answer is that we can't. Think about how much code we write. The number of tests that we have to write in order to ensure our code is working is, like, it's exponential proportional or however you say that. So, as requirements go up, the amount of code we write goes you were and the amount of tests that we have to write go up even further. And so, this is a problem because this explosion of code that's necessary to fully test your app and the code that's necessary to write your requirements is so high, that we miss out on all of the requirements that we really want for our app. So, this could include thing like missed features and lack of user research in making our product better. And so, this is a big problem. Now, let me cover a few base terms for you right now.

Endtoend tests are tests that make sure that the app is working properly. Integration tests make sure that the code is working properly. And unit tests make sure that the developer not asleep or anything. And so, that really points to the fact that the most important tests we're going to make are the tests that directly model what our users are doing in our application. So, endtoend tests are going to be at the top of that testing trophy. This is by Ken C. Dodds. The easiest are unit tests that come with linting or TypeScripting or flow if you use flow or whatever reason. You know, unit tests are easy to write, but there's really no trophy for writing your unit tests. If your user complains about your app, you can't just say, you're wrong because all of the unit tests pass, so, you know. So, that's why this is what the testing trophy for unit tests looks more like.

All right. So, the whole premise of my talk is, what if we wrote fewer tests? Like what if we didn't write tests at all anymore? You know? So, what if tests can be generated? And so, what if we could generate these tests without those tests being written? And so, on top of that, what if these tests can be regenerated when requirements change and the model changes without having to rewrite the tests? Is this possible? Yes. And it's called modelbased testing and it's not something crazy I came up with. Like everything else that I talk about, this stuff has been around for at least a decade if not more. And modelbased testing is sort of the way of autogenerating endtoend tests. And it has a sibling, propertybased testing which allows you to generate unit tests essentially.

Play from 2:00

So, let me run you through an example to show you how modelbased testing sort of works. Some of you might know this problem, it's called the Diehard riddle. You have a 3 gallon and a 5 gallon jug, you can fill them up with water, but there's no measurement lines. In order to diffuse a bomb, Bruce Willis had to get 4 gallons from 3 and 5 gallons. I'm sure you have heard of this, but what do we do first? Shoutout. All right, we'll go with 3 because that's what's on the slides. [ Laughter ] they're not as dynamic as I would have hoped. But, all right. So, let's move 3 to 5, right? What can we do next? Fill three again and then we could like fill three and put it in five. Now we have exactly one gallon in three. And we could empty five and then move that one gallon to 5, right?

And then we could fill the 3 gallon jug. And then we have 4 gallons. And so, you might think that this is the only solution, but it's not necessarily the only solution. And you might arrive at this solution just by trial and error and just guessing, okay, we know we have to put this in the 5 gallon jug and just trying to figure it out. But what does this look like to you? And what have I been talking about for the last three years? Finite state machines. It's sort of my brand. There you go. Finite state machines, as a reminder, is basically whenever you see boxes and arrows, you can attribute that to, okay, that's some sort of finite state machine. All apps start in an initial state such as idle. When you're loading an app, nothing is happening yet. And you might have different states that your app can be in.

So, if you're loading data, you can be in a loading state or a success state or an error state and these states are finite. Which means that you can't be you've established what your specific states are. There's not going to be any unexpected states. And also, there's a finite number of transitions between the states. And I pressed a button. That was a weird transition. These states can transition to each other. But how they transition to each other is by events. For example, if I'm on the idle state and the event occurs, now the loading state. And we're essentially following arrows. And that's all you need to understand about state machines. If we represent this Diehard puzzle as a state machine, we can generate the combinations for how to fill the jugs. Zero, zero, zero, 5. If we fill the 5 gallon jug, 0, 5, and so on and so forth. And so, just by looking at this, we know we need to arrive at this 34 state because that's where we're going to have the 4 gallons.

Play from 5:00

We can see there's a clear path from the initial state of none of the jugs being filled to the 34 where we have our 4 gallons. So, this, of course, is a contrived puzzle. But we could think of our applications as these puzzles, these riddles where we're going from one screen to another. And essentially what we're really testing in our applications is we want to make sure that our users can reach certain states. How they reach it doesn't matter. We just need to make sure that it's possible for them to reach it. For example, logging into a form. If it's impossible for them to do it, then that's a bug. And we have to make sure that we find a way to automatically test that. And that's where we introduce abstract models. So, if you know a thing about testing, behavioural testing, there's this thing called, I think it's called cucumber testing or this other weird word where you have three thing. You have a precondition, an action and a postcondition. These are your given when then tests.

For example, given, I'm hungry. When I eat fairy bread, I guess? I don't know what that is. Then I'm no longer hungry if I eat enough fairy bread. So, we can turn this into an abstract model where our precondition is a state. Our action is an event that causes a transition to the post condition. And so, we could just take this abstract model which does nothing on its own and we could tie both assertions and executions to them to make them executable tests. And so, here's what I mean. When you're creating an app, you want to create a model. So, you want to say, this is how my app works. I have a login page. You click this button. We go from the login page to a loading page to a success page. You could represent that as a directed graph just like this. Then the second thing you would do is you would generate the abstract tests.

And this is just a function that basically takes your graph and it goes through all of the possible paths from one state to any other state. And then, you make those tests real. Because the abstract model doesn't actually do anything, you have to tell it, this is what it means to be in this state and this is what it means to execute this action. And so, again, you're representing just these chains of preconditions, actions and postconditions. And those make up your entire testing path. And then it's up to you to execute the tests using, you know, a test runner of your choice. You would go through it, you would assert when you reach a state, you are truly in that state. You assert when you execute an action, it actually performs that action without error. And sometimes your test will fail. But you will know exactly where it fails and exactly how it failed because you could trace awful actions and states that it went through to get to that failure.

And then finally, you really profit because once you fix those tests, and let's say someone comes in later and goes, all right, I know these are great. I know you have all of these tests written. You didn't write them, you generated them. But they will say, we need to add this new functionality. By doing that, all you have to do is change the model. Don't need to add or remove any tests. These new tests will be generated because it's going to find a new path to those different states. So, let's take an example. Over here, I have a feedback form where first initial state. It just says, how was your experience and then you could click good, you could click bad. If you click bad, it will ask why. It's a feedback form and you can submit it. If it's good, thanks for your feedback because any good news is fine with us.

And then you can close the app at any time by clicking the close button or by clicking any of these Xs. And so, let's let this load a bit. And see, so, basically you clicked good. It says thanks for your feedback. Really simple app, create React app, 30 minutes, not less. Submit. And yeah, that's basically how this very simple app works. Let's test it. We're going to be using three things. Using React's testing library, jest and XState because XState is going to allow us to represent this as an abstract model. Now, notice, this app doesn't necessarily need to use XState. It doesn't need to use finite state machines at all. We're simply using XState and state machines to model an app as a user would experience it. So, testing library, it's by Ken C. Dodds. It's amazing and simpler than Enzyme. And there's integrations with Puppeteer there's the DOM and view and all of that. So, writing a normal test is pretty straightforward. When we render a form, we get all of these like get by test ID.

And we can assert that something is visible. So, this is our given, right? And then we click the good button. That's our when. So, that's our action that we do. For example, you know, what we're on the question screen and we click the good button, then we should be on the thanks screen. This is a behaviour that the user observes and the behaviour that the user has come to expect. And so, that's how we should write our tests. So, in about 32 lines or so, I've written two tests. And I am proud of myself. But this is not sufficient. Because if we really take a look and we turn this app into a finite state machine, we will see that there's actually a lot of different paths that the user could take.

Play from 8:00

You know, we could go on question. We could close in any screen. We could click bad, which takes us to the form and then we could submit. Requirements might change. We might have a back button if the user clicks bad and then decide you know what? This app is actually pretty good so I'm going to give it good feedback. And have so, to do that, we're gonna take that state machine, which by the way this is XState, this is basically Redux with extra rules that actually make sense. You pass this in as an object. You give it your states like question. And then you say on certain events, here is the next state you should go to. Just as simple as that. All right. So, here's what we're going to do. We're going to take that state machine and generate paths. And paths, again, are from one state to every other state. So, shortest path is simply the shortest way to get from one state to another state.

For example, if we start on A and go through E. We can go through A to C to E. But this isn't necessarily the shortest path, because if we consider weights, there might be a shorter path. Taking a highway might be a few extra miles. Or kilometers or mile or whatever they use here. When you reach E, it might be faster. Or might be more common for users to traverse this A, B, D, E path rather than ABC. And then there's the not shortest path, from one node to another node. And there's many ways as long as there are no cycles. ABCE, ABDE, or ABCDE. We're not revisiting any states. We could generate one of those two types of paths. And in this case, there's a few different paths that we could take. Question, obviously, there's one path because you're always gonna be on that state. Form, you could go from question to form by clicking bad.

You could go to the thanks screen in two different ways. You could click bad, submit the form and you're on the thanks screen. Or click good and you're on the thanks screen. Can then there's many ways to get to the closed state. All right. So, here's what we're gonna do. We take the state machine that we created, and we put test assertions in each of the states. For example, in the question state, we're using React testing library to assert that we do reach that question screen. And then when we create a model. And this is using the XState test library which consumes that machine. And you could pass it in what those events actually mean. For instance, the click good events, we could tell it this abstract events means you're going to find a button on the page that says good and you're going to click it. And then you could actually submit cases as well to say, this action or, yeah, this event can be represented in many different ways.

We could have a value of something. We could test edge cases. Have a value of an empty string, et cetera. This is going to allow you to be really thorough with your tests. And then comes the fun part. We generate the plans, whether it's the simplest or shortest path plans. And then we are going to go through each of the paths that's generated, and we test it. Just by calling it path.test. And we pass it in whatever sort of test objects that we want to pass in so that we could actually assert our tests within the state machine. So, the results of this, and, of course, we could test coverage as well. Just to make sure that every state has been visited and every event has been executed. And, the results of this is that we have tests generated for us automatically. Demos are a little scary.

I'm just going to show you the screenshot that we have here. It just shows that we went to each state and we traversed each path in different ways in order to reach that state.

Play from 14:00

So, let s take this a little bit further. What if we were to introduce I keep pressing that. All right. Puppeteer with just an XState. If you're not particular, Puppeteer allows you to do browser automation right on your computer and scripted. Very similar to how you would write your tests. And so, we don't have to change the model at all. This is an example of us actually taking the exact same abstract model that we created and just using Puppeteer instead of React testing library. And you can see it will actually execute and run through all the tests. But let's actually try a demo. And this is a little bit precarious because there's many things happening at the same time. I have a web socket going to the state machine, and something that relies on a WiFi connexion. During a conference, that's really bad. But this is not a live coding, it's a live testing thing. So, if it fails, yeah, that was part of the so, let's try this, over here I have an abstract model of to do NBC.

All of you are familiar with it. It's a to do form and written in so many different frameworks. So, I have this machine here. I don't know if you could see it that well. But I could explain it. I modeled it as being in two states. Empty and then nonempty. And you know what? I hate scrolling through code. Let's do this real quick. I'm going to I'm gonna actually copy/paste this into the visualizer and then you can see it over here. We have an empty state. And you add a to do, it goes to the nonempty state. There's a few things we could do, toggle all. Toggle all the to dos. And then it goes from active to completed, et cetera. All right. So, going back to here, that's our test model. And so, what we did was we wrote some Puppeteer tests that just make sure that when we're in a state, we are indeed in the state that we expected to be.

And also, for each of the events, we have add to do. We want to type the event message into wherever it says what needs to be done. And then we press enter on the keyboard. And other here, I'm just putting the message, ReactConf AU. So, let's hope for the best. See what happens. Also, don't blink because these tests run pretty fast. That's a good sign. All right. [ applause ] there we go.

Play from 18:00

And we have 22 tests generated just from that model. And so, we could just increase that model and show it doing different things. Maybe there's different features. Maybe we want to add more than one to do. And really push this to the limit of edge cases. So, yeah. Let's go back to the slides. All right. So, basically, this is time travel debugging in the future. Because with an abstract model, we know what's going to happen ahead of time instead of what happened in the past. So, it's sort of like what Redux does, but more useful. So, what does this have to do with React? And the quick answer to that is: Absolutely nothing. You know? You could use Vue, you could use Angular, you could use you could go, I really want to try this on an old Backbone app. But I felt like that might cause test failures, so I didn't want to do that with to do NBC.

But the idea of abstract modelling and modelbased testing, it doesn't matter what the implementation details are. This is the crux of testing library as well. It shouldn't matter which framework you use. It shouldn't matter if you're using JavaScript or you're testing on a browser or on a mobile device, these tests should describe how your user experiences the app. So, there's a lot of similar libraries. XState test is the one I created for modelbased testing. There's one called Simulato by the USA today network. Which does a similar thing. And GraphWalker, made by Java. I made the classic recruiter mistake, it's for Java. But it's still really interesting to look at just to see how that works. Now, I'm going to be honest with you, going into modelbased testing is harder. There is a learning curve because you have to learn state machines and if you want, statecharts. But you also have to learn how to model your application correctly.

Thankfully, this is an intuitive process because you can just grab a whiteboard, grab a pencil and paper and model exactly how your app is supposed to work from a user's perspective. The app, I models by a user, it's empty at first. And when I fill the to do, it's nonempty. But it's nonempty it I where none of the to dos can be completed, all of them can be completed and then I can test individual things. And so, those are different states within the model. And also, there's no guarantees that modelbased testing or any type of testing is going to absolutely cover every edge case. But this is going to be exponentially more coverage than any of your other tests. I know with normal testing a lot of people strive for 100%, 90% test coverage. But that's just covering lines of code. That's not covering use case coverage which I feel is a lot more important.

And that's why I feel like there's a lot of advantages to modelbased testing. Because not only it's gonna be more efficient because you have a single model and you're generating all of these different tests just from that model, but you have requirements integrity. Which means when a latebreaking change comes in, a functionality change you want to add or remove something, all you have to do is change the model. You have that visual model, boxes and arrows. You say this needs to change. This needs to move. And then all of your tests will be regenerated. You no longer have to say, well, I just spent hours on all of these endtoend tests and they want to change something. I'm just going to skip or comment out or ignore all of those tests. No more of that. And this is a much more flexible approach that reduces cost and time as well.

You're only writing a model, not writing the tests anymore. And, of course, like I said, it's implementation agnostic and it's going to find a lot more edge cases. Especially the more types of events that you provide to it.

Play from 21:00

So, XState test is part of the XState growing libraries. I released XState Vue, none of you care about it, that's fine. Vue is really nice, you should take a look. But XState React has been around for a while as well. It provides hooks. By the way, if you know how to use the useReducer hook, you know how to use the useMachine hook. It's the exact same signature and you can plug it in just like the useReducer. I have a few things on the horizon, analytics, tools, and a visualizer which you have sort of seen a preview of already. So, I'm also really excited because XState has reached over 10,000 stars which I never thought would happen. Because state machines are actually really boring if you have been through a university class and they taught about it. Really boring stuff. But turns out lots of companies are using it, Gatsby, React training and Microsoft and other companies who say, hey, please don't tell anyone.

We're using your library. I don't know why they're being secret about it. But that's them, not me. And long story short, I want to encourage you to make your code work with the abstract models and be modeldriven. That way you can automatically generate the tests and you can go even further into generating docs or take a look at Playroom, you can generate prototypes in the future with this as well.

Play from 25:00

In short, I want you to make your code do more. Thank you very much, Sydney. [ applause ]

Play from 27:00