Travel Through Time to Debug JavaScript

Captions provided by White Coat Captioning (https://whitecoatcaptioning.com/). Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.

JASON: Hello everyone and welcome to another episode of Learn With Jason. We have mark Erikson on the show. How are you doing?

MARK: Doing well. Super excited to be here.

JASON: It is a blast having you on the show and I feel like I always learn a lot being in your general vicinity. Do you want oo give us a little background on who you are and what you do?

MARK: Currently, job wise, I am a senior front end engineer at Replay where we are building a time traveling debugger for JavaScript and we will be talking about that plenty over the course of this show. Most folks probably know me as either the guy who shows up any time the word redux is mentioned online or that guy with the Simpson’s Avatar.

JASON: And that is, I mean, I think you have probably the most consistent Avatar game of anybody I know. And comments on you being from space?

MARK: It is a mass a fact spaceship background. — effect.

JASON: Everyone is very excited about this. Yeah, so briefly, the last time you were on the show we talked about Redux and it was great because I think that episode was a very good one, I think. It just helped me at least to spell some believes that I had about what Redux was or is and what it is actually, you know, kind of capable of, with modern stuff which I thought was wonderful. Are we doing anything with Redux today?

MARK: There will be just a little bit. Part of what I want to get through is talking about like how do you approach debugging JavaScript in the real world and that does include with a little React and Redux specific concepts but for once redux is not the focus today and I am actually pretty happy about that.

JASON: I won’t talk about it any more. Let’s talk about Replay. Well, before Replay, let’s talk about the high level. When I reached out to you to see what you wanted to talk about you said time travel debugging. This is the sort of thing that, you know, I feel like that’s a concept that I have heard thrown around and something that we have seen in esoteric demos and I know that was one of the core promises way back in the day with even things like Redux. Was this enables time travel. And I have been doing this since what, 10 years since all of this was started and I don’t know what it means still. Can you explain time travel to someone like me who is currently stuck in the present?

MARK: I do want to get into talking about debugging in general but let’s talk about the time travel thing for a second. One of the basic truths about life and debugging is things only move forward. You can’t go back and change what’s happened. It is that bit from Space Balls about when? Now. If you are debugging something, maybe you have a break point somewhere, you are paused, you hit step next, and see a value and whoops, I just went too far. Now you probably have to go back and, you know, reload the entire program. Maybe do fiddling with the break points. Get back to the area that you were looking at, step, step, step and thou you are back at the point in time you actually care about and it is very easy to miss the piece that’s interesting or have to do a lot of work to set that back up. The basic broad picture of time travel debugging is if we have some way to rewind time or capture everything that’s happened, you don’t have to go through all that setup and getting back to where you were. You can go to any point in time and you can see what’s happened and it is, you know, again, like depending on whether you are using redux or tools that have other features, you can go to any point tinal that’s interesting and see what’s going on and you can see how things change over time rather than just their current value wherever you are paused. Like with Redux, we have the Redux DevTools and you can see both the history of the actions that got dispatched and for each one, you can see what the action object contained, you can see what the state looked like after it was done processing, but you can also see how the state changed as a result of that action being dispatched. In the case of Replay because it is a DVR recording of your application for a couple minutes at a time, you can go to any line of code, you can see how many times did that line of code run, you can see what the variables were every time that line of code ran, you can add print statements that print out what it would have logged if you had a log statement in the code originally, and you can get this sense of the progression of the program that you wouldn’t have if you were trying to step through things yourself.

JASON: That is extremely cool. Honestly, it sounds like — it sounds too good to be true. [Laughter] It the sort of thing I hear that and I am like great, if you showed me that in a sci-fi book I would suspend my disbelief enough to say that’s a great idea. It feels like that’s such an impressive thing to be doing thinking about the idea that you said a DVR recording of what’s happening. That’s very cool. My curiosity then is around in debugging in general, as you were painting your picture of what it is like to debug without Replay, even that was maybe a little more advanced than someone like me because you were mentioning things like break points — I don’t use break points. I am a console log ride or die. That has been a pretty significant source of maybe pain is the right word. Making things just more complicated for me than they need to be. Let’s talk in the abstract about debugging because for someone like me and in the roles I have worked in I have always figured out a way to get back and I think it is the equivalent of hitting the rock with a stick in terms of my debugging and I never learned the tools or the craft of it. What’s your world view on this?

MARK: Happily. I have opinions on this. Big picture, debugging can be boiled down to like why is this broken and how do I fix it? I think you actually brought up a very good point which is most people frankly don’t know how to debug or if they do know how too debug it is only because they have years and years of general programming experience and it just so happens they have spent time debugging and they learned it the hard way. I think this is a really bad reflection on our industry as a whole — spend lots of time teaching people how to program, we spend lots of time teaching people how too use library X or tool Y, and from what I have seen there is little emphasis put within companies or courses or whatever about how to debug. If you think about it, we spend a ton of time debugging and we are sort of flailing in the dark without actually having like a scientific approach to what we are doing. I think one of the best things that almost anyone can do for their own career is to spend time trying to learn how to debug effectively. It is kind of like learning to code. I often say when you are learning to code you are learning how to look at a problem statement and break it down, how to describe a series of steps to solve those problems, and then how to do express those in terms of a programming language. The first two are just about thinking. Not even about writing the code at all. So debugging is much the same way. Most of debugging is learning about how to think through the problems and a scientific approach to tackling them and then you have get into the nitty-gritty details of specific tools but most of it in your head. I do think it is a skill most people can intentionally learn and practice and apply and they really should.

JASON: Yeah, I think that’s a really valid point too about, you know, I think we focus sometimes really heavily on the specifics of the tools. I think that that has led to a little bit of a — like a gold rush mentalty. We are looking for a tool that will magically solve all of our problems and the biggest problem with software is that people don’t build the right mental models around it. They don’t learn to think together as a team. Most of the tech debt I have seen isn’t because somebody was right or wrong but because two people were building the same tool from different models and they were not compatible and they didn’t take the time to learn how to think together so no amount of tooling in the world would save them. I think that’s lead to the heavy handed tooling we see like if you look at Angular that’s almost entirely driven by code gen, I think that’s enforcing a mental model saying use the CLI and it will think for you and you have to build within there the tight boundaries of these is the generating model and don’t use the API or everything else falls down. That’s the early days of Redux and what I saw it becoming. I know I said I wasn’t going to talk about Redux —

MARK: A little digression is OK.

JASON: You would throw these very prescriptive it must be exactly like this and you need funk and saga and this and that and it wasn’t thinking about programming. It was trying to like put these very prescriptive things in place and when you dug into why, there was the programmer who was trying to force everybody to think like them and everybody else who was handed a system and they just used it because that was the system. In debugging, it is sort of the same thing. The better your mental model is of the problem space, the more you can reason about what’s happening and have an idea of where to go. And then I think what you are talking about with tooling and like tooling is sort of the thing that happens after you know what you trying to understand and then you can get better or worse visibility based on the tools you choose. I am whacking on it with a hammer in my case and in your case you have surgical tools to dig in and see what exactly is happening. You are saving significantly more cycles because, you know, for me, my approach to debugging has been move the console log up one step. In complex systems, I am just guessing. I am really excited to see more about how you are doing this. When you are talking about building a mental model of debugging, how have you found that to be different from software. These are two different skill sets clearly. I can build code and never debug it.

MARK: The mental model is key. If we say debugging is something is wrong that implies you know what it should be doing in the first place. Really, in order to debug effectively at all, you have to have a decent idea of what is intended correct behavior. It beed — it could be as simple as I press the button and sparkles light up instead of the entire program exploding or there is 15 different architectural layers to this application and they interact in this way and the data is being passed from here to there and this piece is supposed to run before that piece but you have to have an understanding of what the system is supposed to do in the first place and how the pieces are supposed to fit together to then look at the actual behavior and say this piece is wrong because it doesn’t match what it is supposed to be doing. That’s kind of like a step 0 — you have to have a decent enough idea of what the intended behavior is to kind of isolate down and say well, this piece doesn’t match up with what I think it supposed to being doing. That’s kind of your prerequisite.

JASON: OK. That definitely makes sense. So I think the other pieces that are really interesting when you talk about time travel debugging is that what we are doing as engineers, when we build a piece of software, is we are oftentimes telling the robot do something, hundreds or thousands of times, and we have like as you said, a model of what’s supposed to be happening where I know I am going to pull something from a REST API and then I am going to run operations against it to change that data into a shape that’s what I need. That makes some assumptions about what the REST API is going to send me. I am assuming that the data comes back in this shape with these values and I think this one is going to be set always and I assume they will always have the same data if I call the API over and over again. When you get into these huge datasets, my code is running with all those assumptions built in and until, I explicitly build in all of my edge cases and stuff which, you know, that can take years.

MARK: Nobody does.

JASON: Right? The first time you assume it will work perfectly all of the time and then the first thing break and you have to figure out when in those thousands of cycles did something actually go wrong and which cycle was it, what went wrong, why did it break and what’s the thing I can do make it not break next time or is it even my software? Is it that this API is literally sending different data for the same call sometimes and I need to have a whole different problem to solve. What’s interesting about what you are talking about with this time travel debugging is if we have a DVR of the last time I ran this program and I can say, OK, instead of me having to console log 10,000 entries, and then dig back through my terminal history, assuming I can even scroll that far back before my history runs out, right? And try to find the one thing that is malformed, or writing a bunch of specific debugging logic to guess at what might be wrong, or whatever the thing is, I can use something like replay and just rollback the time to spot check what’s going on — is that correct?

MARK: Pretty much. Let’s take a step back. I have a presentation where I have talked about approaches to debugging and stuff and in that that I listed out some high level steps to debug any problem. Your basic algorithm for debugging. The high level steps are number one, understand the problem description, like what am I even trying to look at/what’s the system. Step two, you have to be able to reproduce the issue because, like, OK, someone reported it is broken. Does it only happen on their system? Does it happen all of the time? Does it only happen on Tuesdays under a blue moon when the time zone is set to GMT-3. So you have to be able to produce it and see, yes, it does happen and that’s the entry POIPT to know where to look. If I know part of the introduction step is click this checkbox and tab and button that tells me what areas of the code are involved and it means I can start to look at it, and if I do step too far, I can go back and make the same thing happening again. It also means if I think I have a fix I can try those same steps and see if the problem actually goes away this time. Understand what the problem is, reproduce the issue, then you get into the scientific method of trying to figure out why it is actually broken. Like, think about the problem, understand the code, say I think this thing is why it is going wrong and that’s that narrowing down process, and then you use the tools, try to identify the relevant lines, maybe change one thing, compare the behavior, et cetera, but you have to do it in an intentional way rather than randomly changing bits of code and hoping it goes away.

JASON: Just change the one thing? We are not supposed to rewrite the whole program? I am learning a lot about myself and how I am wasting time right now.

MARK: Even once you have identified what appears to be the problem it is usually worth trying to go a step further and see like, maybe this was just a symptom, and the actual root cause is somewhere deeper. Like maybe the bug is, you know, could not read property X of undefined here, but the root cause is well, yeah, my REST data was changing and we didn’t validate it. Once you have that figured out you apply a fix and verify it works and that’s also where you should be adding more tests, updating documentation and assumptions, and trying to make sure maybe you fixed not just this one bug but maybe an entire category of bugs going forward.

JASON: Right.

MARK: To bring the back to what you’re saying, reproducing this issue is a very, very critical step early on and part of the advantage of a time travel debugging tool like Replay is that you only have to reproduce the issue once and now you can investigate infinitely as much as you want after the fact instead of having to reproduce it every time and maybe it only happens under certain weird multi threaded conditions and you are spending half the time trying to make it happen again before you can even investigate.

JASON: That’s actually, especially when you were talking about time zones, time-based bugs are a huge source of frustration. Or if you are doing something with chronjobs. Being able to have a single reproduction you can debug from is huge. I have written a fake code to manually set a date where it needs to be and mocked and it is absolutely awful to go through and do that type of work or to have to wait 12 hours until you are in the right time period where the 24-hour clock would break it or whatever the thing is. That is something you can see that I have wasted days on. Like I have to go debug this.

MARK: It happens on someone’s machine and not yours or someone reproduced the bug and reported and if you have that recording available, especially in the case of Replay, they have done the work to reproduce it and I as the developer can dive in and start investigating without having to take the time to reproduce it.

JASON: Hold on. You just said something that I had not considered was possible that has not been considered. It isn’t me the developer that has to reproduce the bug but you are saying if we have got it instrumented properly someone can send me their own reproduction and say this is my session here and then I don’t even have to figure out how to reproduce it. I take theirs and replay it on my computer.

MARK: Exactly. QA teams, external users, other developers. We do this at Replay all of the time and use Replay to solve bugs in our own program and most of our own internal bug reports are accompanied by a replay where someone on the team saw it was broken and file it and I can pick it up later and see what they were looking at.

JASON: OK. That is really cool. I feel like one of the things that’s hard about this show is that I love these abstract discussions about programming and also I really want to see in action so I want to talk about this forever but I think in the of time I want to switch over —

MARK: Let’s take about 15 minutes and talk general debugging tools like not even Replay yet but just like what tools are available right now that users should probably be familiar with. If you can pull up, let’s say, let’s go with the React and Redux code sandbox link I sent you. Let me get over to which one? This one here.

JASON: We have live captioning happening. Thank you so much to White Coat Captioning and I forgot who is here with us. It is Maggie. Thank you so much for being here today. That is made possible through the sponsors, we have Netlify, NX, New Relic and Pluralsight, all kicking in to make this show more accessible to more people which I deeply appreciate. We are talking to Mark today so go give him a follow on the old Twitter. We are talking about Replay which we will get into in a minute and I am working on these examples here that we will go play with. Did you want me to pull these up in the Replay browser?

MARK: Standard browser will do here. Pop open the React/Redux sandbox. Next session. Sorry. Back.

JASON: This one?

MARK: Yes, please. So this is just the bare basic React plus Redux counter example act as a code sandbox and the point here is to talk about what tools does even like the browser offer for general purpose debugging. All browsers today have a large suite of developer tools built in. There is various ways to access it. For me, I usually right click on the page and inspect element to open them up. I end up popping this out as a separate window but that’s just me.

JASON: Let’s do it. Somewhere off —

MARK: All the developer tools have a bunch of different tabs built in at this point. Usually there is an elements tab which shows you the HTML structure of the page. Trying to read the tabs. Second tab is…

JASON: Console.

MARK: Yeah, then you have con out puputting the console logs and is an interpreter so you can run expressions or type in the names of variables and see the contents. You have the sources tab which is the in-page debugger letting you inspect the JavaScript files and do all that break point type stuff.

JASON: A show of hand in the chats, anybody —

MARK: How many have used that before?

JASON: I have never used this page. Put a W in the chat if you have used it and an L in the chat if you have not.

MARK: You have the network tab showing you how many network requests went on to what URLs and what kind of files and how long they took. Then there is performance tab where you can make a report of the page to see the JavaScript functions running and taking up time. These are tools built into every browser at this point. What I want to focus on for the moment is the sources tab and the JavaScript debugging experience specifically. Over on the left —

JASON: This is making me feel silly. They have all used this tab before. [Laughter]

MARK: Over on the left, we have a file tree. You will see a bunch of different stuff in here. That’s because this passenger has loaded a lot of different things. There is the actual application code, there is code sandbox scaffolding for running this, and there is various other transformed versions of the code in here as well. In this case, we can see that if you look at that top section that ends in.csb.app and that’s the real source code. We have folders for node modules and source as we can see. If we open up source/features/counter/counterslice.ts and part of the renes reason you are seeing a couple versions is JavaScript programs go through a build step and everything gets manageled. — mangled. A typical build step has to generate source maps which contain the original versions of the code so browsers can read those and so show us the original code and we can debug that even though the minify messed up code is what is actually running in the page. Hopefully, in mose cases, you have source maps and can see the original structure of the code and from there we can use fairly typical debugging tools. Now, most graphical debuggers have basically all of the same features and capabilities, you know, it just a question of where are those buttons located. For example, one of the things you can typically do is you can add a break point and a break point tells the JavaScript engine when the code is running and gets to this line, stop, pause, halt. Do not proceed further until we tell it to. For example, let’s scroll down to about line 39-ish. There is a line that says state.value plus equals one. The way you would add a break point in most debuggers is you click right there on the line number, or just to the left of it, and add the little blue marker saying there is a break point on this line. Now it is just sitting there not doing anything. Let’s interact with the page and make something happen. Over on the page, if you click the plus sign, the click handler ran and dispatched an action and it is running and because we got to this line the browser is now paused. Now we have the ability to look at the variables that exist in this portion of the page and we can do the various stepping commands. Now as it turns out this is probably not the greatest example because the Redux reducer is only one line. Due to the specific details of how it is implemented even looking at that state object is not all that helpful in this case. Let’s actually try something else. Click on that breakpoint online 39 to remove it. Click the blue play icon in the lower left just below the file tree.

JASON: Resume script execution.

MARK: Stop pausing, keep running so everything else kept going and the page updated. Let’s pause somewhere else. Try opening up the file counter.tsx and again, we have got two of them and I think one of them is maybe italicized and I can’t remember which is source mapped and which is the original version.

JASON: Italicized is the version from the source map it looks like.

JASON: Let’s try putting a breakpoint on line 16. So kind of like just inside the page. Go ahead and click the plus button again. At some point, everything updated and our React component is rerendering and now we are paused inside here. There is a bunch of things we can do at this point. One is because we are paused you can hover over variables in the page or in the editor and see what their current values are. We are just past the line that has count. Try hovering over count online 15 and we can see that the current value is 2. You can tell the JavaScript engine to advance further by using the various step buttons just below the file tree. Every debugger basically has these buttons built in. So the usual commands are play or resume to keep going. There is a step over version which will advance to the next line and if the next line is a function call it goes past it and goes to the next line in this file. There is also a step-in button to where if the next line is a function call it will jump from this file into the source code of the function that we are about to run. There is a step out button where if we are inside a function will exit the function and go back up one level. Depending on the debugger there may be one or two options but these are the fine grained control flow. We are paused somewhere and I want to move forward bit by bit without just like hitting play.

JASON: Using these doesn’t require me to setup lots of different breakpoints. Once we hit the breakpoint these will allow me to navigate through the program from where the breakpoint starts.

MARK: These are active when you are paused somewhere and until we hit the run button we continue to pause at a line and we can step over, step over, step in, step out, et cetera, and again, like every debugger has these. Visual Studio Code has them, full fledge visual studio, IDEs, these are standard controls every debugger for every language has built in. There is also tools to let you see what the variables are in the local scope that we are looking at. If you look in the lower right there is a scopes panel and we can see that right now we are kind of nested about three levels deep. The current scope is described as local and it is telling us for example, what is the value of the JavaScript, this variable, at the point in time? Well flight right foe it is — well right now it is pointing to window which isn’t helpful but we know when it is. Count is 2. Dispatch sin scope but it doesn’t exist yet because we haven’t gotten past the line that assigns it. We have got several other variables that are declared. The JavaScript engine knows they will exist but because they are declared with const keyword they don’t exist yet. Try hitting the step over button on the left and we should see dispatch is now a function and if you step again we can see that because we just read the values from the React hook we have in crement amount and set increment amount. Those show us the variables that exist in this scope. If we go expand the word block down there, it will show us the variables at the next level out of the JavaScript scope. We are inside the file which is maybe inside the if statement and that shows you your levels of nesting in terms of available JavaScript decorations. Basically every debugger has a way to show you what variables exist at the code that you are currently paused at and then there is also a watch tab next to that where like maybe you are going back and forth between different functions or like you really only care about some certain variable names so you can type in variable names or even expressions like object 1.A.B.C and every time you are paused it will try to evaluate that and it will show you what the value is if it exists at that point in time. These are very common pieces of graphical debuggers that every GUI debugger has.

JASON: Very cool. This is built into Chrome and Firefox and whatever browsers you may have. But if you open up Visual Studio Code or something else yields you would see these same controls in different places but the same capabilities available to you. Another thing you can do is we have the breakpoint there online 15 and you can generally modify and maybe you have it there and don’t want to pause next time it runs so you can right click and disable it. You can also make breakpoints conditional like maybe I only want to pause if the count is currently, I don’t know, two or something like that. You can edit the breakpoint and add a condition value so like, count triple equals 2 or something like that.

JASON: Let me set it to 3.

MARK: Sure.

JASON: Then it will run a couple times first, right?

MARK: You also need to reenable it.

JASON: Reenable it. It is enabled and conditional now.

MARK: Hit run because we are currently paused. Hit the plus sign again.

JASON: And remember, I set it to be at 4 so it shouldn’t do anything this time.

MARK: Yup.

JASON: And it did want. That’s — OK. This is huge because one of the reasons that I haven’t used breakpoints is because every time I think they would be useful it is like in some loop operation where I don’t want to hit play a thousand times so this is a game changer. I feel like you just changed my life, Mark.

MARK: In fact, there are times when you might have like a half dozen breakpoints set. You don’t care about breakpoints 2, 3, 4, 5 until the first one gets hit, so I might go in and disable the later ones, make the first one conditional, once that gets hit, then I go and reenable the rest of them because now I actually care about pausing at those things. This can still be a bit of a pain and that’s one of the pain points that Replay solves which we can get into in just a minute. In fact, on that note, maybe we ought to switch over to Replay in just a second.

JASON: OK. I do have it installed. Let me just pop it open, right?

MARK: Sure.

JASON: Let’s do it.

MARK: The basic sales pitch for Replay is it is like a DVR for recording your application. It does require you use special modified version of primarily Firefox but we are working on Chrome and have Chrome for Linux and Mac and Windows coming shortly. You open up the website, press the record button, use the website for a couple minutes, hit save, it uploads it to the cloud, and then you go to our website, own n — open up the recording and start doing the time traveling debugging thing.

MARK: Worse case if we have to we will stip the recording step and go straight ahead to the debugging recording step. Looks like you are signed in.

JASON: The classic game of hot dog not hot dog I am a developer.

MARK: OK. There is a bunch of possible programs we can use here. Let’s open up the sandbox link in here. In the upper right press the record bus n and it will reload the page. Now just click a few buttons. Click plus and minus once and click add amount and at a sink — async and decrement once. Hit stop in the upper right let’s click the public access. That way it is shared and other people could open up if they want to. We will skip that for now. OK. We made a recording of the program and we can now do the time travel debugging thing. The first thing we are looking at is viewer mode which is a video playback. If you click the little blue play icon in the lower left it should show you like your mouse and all the actions that you took. Basically the video recording of what you did in the browser. That part is cool but the really interesting stuff to me as the person who works n othis and cares about it is the debugging part. There is a tab in the upper right called the switch tab. Click on DevTools. What we see now looks and feels like the Firefox browser developer tools as a program because it basically is the Firefox browser developer tools as a program. Our client side codebase actually started as a copy/paste of the Firefox DevTools source 2.5 years ago. In the upper right, we have the video play back so wherever we are paused in the recording we can see what the page looks like. Let’s do this. Over on the left side, the middle icon that looks like the file icon should bring up the list of sources. Let’s open up the source and open up counterslice.ts. OK. Now we are looking at the source code for one of the files, you know, this is the redux logic that’s in here. The first thing that I want you to notice is that we have got not just line numbers on the left side but we have a bunch of little blue numbers in there. These are hit counts. This tells you how many times each line ran during the recording.

JASON: Down here where I am doing stuff I can see we called in crement three times. All right. OK. — increment.

MARK: Just opening up this file I get a sense of what code ran and what didn’t. That is a big deal because it fits in the mental model question. Maybe I know we were supposed to go inside an if statement and that line doesn’t have any indication it ran. Maybe I thought this loop was supposed to execute five times and as it ran 5,000 times. That’s not supposed to happen. The meaning will be very dependent on what you are trying to investigate and just being able to visualize how many times is useful.

JASON: It is very course signal. One of the first things we are doing is disqualify things that are not the problem. The first thing is like something is broken. It could be any line of code in the app. The first thing we have to do is disqualify a bunch of things we don’t need to check and having a quick visual indicator like this that the code that was supposed to run didn’t run. I don’t need to debug the code. I need to debug why it didn’t run. That’s a huge problem too. For me, at least, a huge number of the bugs that made me want to throw my laptop out a window were because I thought I was debugging something that wasn’t actually the thing that was running. This would have probably saved me hours of life. I might still have hair.

MARK: [Laughter] OK. So earlier we talked about GUI debugging and the other main technique as you pointed out is print debugging. JavaScript you are adding console log statements, Python you are adding the keyword print, Java system out.print line. But in my application code, I write real code that prints some tech and got to this line plus one less variable too. That serves a couple purposes and gives way finding so we know what code did or didn’t run. It gives a sense of timeline so I know this line of code ran before that line of code. We are looking at the values of variables as they existed at certain points in time. The biggest problem, is it an incredibly valuable technique. You know, you pointed out earlier, you have basically relied on print debugging most of your career, I find that both print debugging and GUI debuggers are equally useful tools that are valuable for different situations. Having both of those tools in my toolbox is incredibly important. Just within the last couple days. I have been spending this last week working on a pull request that will include those source map files in the build system for the React library because React is not currently shipped with source maps which means when you get production errors it is always that minified gibberish for the error message or if you step out of your component you are in the middle of React’s code and it doesn’t make much sense even if you have source maps but it makes slightly less nonsense at least. I have gotten React’s build system updated to include source maps but tools like Vite and Next were not picking them up correctly. I was trying to figure out why and where went into node modules in a Vite project and was adding console.log statements to the code for other libraries to figure out what it was doing, what file pact it was looking at to try to find the source map files on disk, and what it was or was not finding. OK. Two points. One, you can totally edit code in node modules and add log statements. That is a real thing you can do and second, that was a time where print statements were useful and a GUI debugger was probably not so much. These are both valuable technique do is have. — techniques to have. Problem is you have to have them in the code before it runs unless you are using Replay. There online 39-ish where we have there state.value plus equals 1. If you hover over that line, there is the little red plus hover icon. Click that. This is what we call a print statement. It is basically your console logotype debugging except we just added it to the recording. If you look over on the right side, we have got our version of the browser console and you can see that it printed out three different lines because this line of code ran three different times. Now by default it showing us the name of the function and the line number that it is on. But the line number isn’t all that interesting. What if we replace the line number with state.value so that we can see like what was the number before we added 1 to it? You came back and it was 6 and now it is 7.

JASON: Right. This as it me pushing like different buttons, right?

MARK: Let’s try adding another one. Try adding one inside the decrement handler. We can do the same thing like state.value here and OK. I can see that we did that after these other things. What happens if we put one inside of the increment by amount handler? There is multiple things to take away. We are adding that print logging after the fact. None of this is in the application but I have my recording and I can start to go in and add those print statements. If you look at the console, it is starting to form that visual timeline of I did thing A, I did thing B, I did thing A again. We can customize the print statement do is have whatever message or variable evaluations we want and we are getting that sense of progression. Let’s try this.

JASON: Before we move on, do you ever see a tool and you are just like, irritated? I am irritated at how good this is. [Laughter] Like this is something — of course this should exist. Of course this is going to be immensely valuable and the fact that if I can — I am assuming if I hit this share and I do anyone with the link —

MARK: It is already set to public because we hit that checkbox earlier.

JASON: So now I am throwing this back to everybody on the chat. Go check this out. Other folks can come in here. People are coming in now. I can see them in here. Let me maybe not show the comments just in case people decide to be naughty. But this is just — this is the sort of thing that if I knew how to build it, I would have built it. But I didn’t so I assumed it was impossible. Now it exists, I am irritated I didn’t use it as soon as it was available.

MARK: I have a bunch of thoughts on this topic too. One is there is serious deep magic type of engineering that is happening in the cloud to make this possible. We have articles linked from the documentation about the technical levels of the recording process and playback stuff works. There is like serious engineering going on to make it possible. It is real and it works. Another point is yes, I am an employee of the company and talking about this thing, companies pay us money to use this, it is free for individuals, free for open source projects. But on top of that, like, I generally firmly believe that this is like a revolutionary advance in being able to debug. I have full faith in the project and the company as a useful tool. That is why I joined this company in the first place because I have burned hundreds and thousands of hours of my career debugging things. I know how valuable this tool would have been if I had it years ago. I want to build this thing that will save developers time and I am excited at how much like — there are so many things we can build that we barely even scratched the surface of. I can show you a couple of those in a minute.

JASON: This is the sort of thing that it is like the first time I saw wheels on a suitcase, you know? I was carrying my heavy bag and I saw somebody go by with their suitcase on wheels and I was just pissed. Why didn’t I think of that? That’s such a good idea. [Laughter] OK. This is great. We have thus far to recap and get back on track. We used the app in the replay browser while recording which allowed me to get this snippet which I have shared, we can see some folks in the chat have joined and are looking at this themselves. I am looking at the source in this source window and I have added not breakpoints but console logs to something that happened in the past which is basically allowing me to go back in time and change how the program ran which let’s me console log information on a previous run. Already extraordinarily cool. What else can we do?

MARK: We have so many things we don’t have time for. I will try to hit a couple highlights. Item one, if you look at the print statement we have inside the increment case you will see there timeline at the bottom of the print statement that says that line of code ran three times. Right now we are paused at the end of the recording and you can see that by looking at the bottom timeline and the little red dot is all of the way over to the end. Item one, we can jump to any point in time in the recording just by clicking on that big timeline at the bottom. Just try clicking somewhere in the middle of the timeline. If you look at the video preview, you can see that, you know, at that time the counter was six, if you click somewhere else in the timeline, it will show you, you know, the counter was, you know, 0 whatever. We can see the video preview change, the little red line in our console that indicates where you are in that sequence changes. You can jump to any point in time and spect — inspect things. You will notice we have the same tabs as the browser developer tools. If you click on the elements tab, for example, this is the same HTML elements thing except it is showing you what was in the page at that point in time and yes, you can pick elements and drill down and see here is what my HTML was at that point in time.

JASON: That is huge.

MARK: We also have the React DevTools integrated. So if you click on the React tab, and this is something we didn’t get to early. I guess we are going to skip past the React and Redux DevTools portion of this demo. There are React and Redux browser extensions and you install them in the browser and when you open up a page that has React or Redux inside of it, they will show you stuff that’s happening in the page. Not even Replay. Just install them in the browser. They add tabs to the DevTools panel. Like with React, the extension shows you what is my React component tree in the page. With Redux it shows here is my actions over time, state over time, Etc. From there we have integrated the React DevTools into Replay and what you are looking at here is the React component tree at that point in time the same way we were looking at the HTML element tree at that point in time. This is the same thing if you were looking at the browser except it updates as we jump back and forth. I also have a small roof proof of concept for Redux. We have three little blue dots indicating that line of code ran three times. Try clicking on the blue dots or the left and right arrows. OK. Right now we are paused on the third time this line of code ran. Because we are paused on that spot, we now have our standard browser step debugger controls available and we can hover over things. Slightly better example, let’s go down to increment by amount case and click on those dots. Try hovering over the word “action “in the code. Like online 46-ish. Hover over the word action. In the same way you could do that in a browser DevTools we are seeing the contents of that variable in scope at that point in time. So cool. OK. Over on the left side, the next to the last icon on the left sidebar has a pause icon with a little red dot. Click that: The top row we have the same step in, step out, step over. There is also a step backwards button.

JASON: Come on! This is so good. We can try. Notice we went to 7 when I clicked it. Now I am going to go back.

MARK: We are inside Redux toolkit because that’s what called this function.

JASON: Cool. OK. OK. This is very very cool stuff. It is like — yeah. I am pretty — so if I want to step out this will take me back up to my code, right?

MARK: Yeah, we are probably somewhere else in the call stack at this point. Go back to the console tab. And hover over one of those entries and click the little blue pop fly out button and it will jump us back to the line of code where we had that print statement and the time that that message got printed. You know, that’s another way to jump back and forth. The other stuff over on the left sidebar shows us the list of the print statements we are and what files they are in. There is the scope channel and standard debugger controls. Here is another thing we can do. One of the great things about replay. I talked about sharing replay back and forth and maybe the QA person recorded it or someone has done prevous investigating. On the right side of the print statement panel, try clicking the red plus comment icon. Just right where you have state.value. Click that. We are inside increment by amount. That comment is now part of the replay. Frequently my teammates ge and I start to investigate a probe — a problem and say we went to this line and found that. Someone can epon up the Replay and see what that person has done and trace their thoughts.

JASON: Is this comment tied to time? Because we made it on a print statement when paused, it is tied to the line of code and the time that code got hit.

JASON: If I was doing my looping thing and have a million things I can comment and say and say this is where it breaks and jump to that line of execution?

MARK: Yup. I want to show off some features I have built and I am excited about. Still in the Replay UI, on the upper left side, the upper left icon with the I symbol, this is the general information panel. In the upper left who recorded it and what page are you looking at and then we have this list of events. We know the time are happened and what code ran in response. We know a click happened but what happened with that click? I don’t know. Nobody knows. I have added extra analysis code that looks at the click and sees what piece of application code ran when you clicked that button. Fingers crossed, try clicking the little blue fly out button on the right side. Come on, magic. Oh, hey, look. It is the unclick handler for the plus button.

JASON: So cool.

MARK: If you were to do that for the others, it should jump you to the places where those ran. This is just very cool. The way I did this is our back end exposes an API and our front end uses that API for everything you have seen thus far. That’s how we built like the stack frames panel and the consoles and all that stuff. But over the last few months we have been able to start building deeper analysis by being able to say like OK, let’s ask questions about the execution of the code and find out interesting information and use that to build nifty new features. There is another brand new one that I just added. It is still very, very experimental and in fact, I just merged a PR this morning that it had accidentally gotten removed so all fingers are cross. In the upper right, there is a triple dot hamburger next to DevTools and go to settings and click experimental on the left and scroll all of the way to the button and click enable React panel. Close this. There is a new React icon on the left side. Fingers crossed — OK. What this is trying to do is — we did a bunch of interacts with the page. Clicked buttons, those dispatched redux actions and those caused React to rerender. Frequently like I am paused with a breakpoint inside a React component and I know that my component is rendering but I have no idea what even started the render process in the first place. Like maybe I am — I am inside a component that’s like all of the way down my component tree and this component rendered because its parent rendered because its parent because something called set state up here. Now we don’t have any of the correlation capability yet. But what I tried to figure out is can I determine when your code called something resembling React’s set station somewhere. Hover over that first on-click and click the jump button. It should actually end up taking us to the same place. You know what? This example isn’t necessarily all that great because this is a simple application but I have proven this works with a plain React set state call. It usually works with dispatching a Redux action as well. This is very experimental. The analysis logic relies on digging into a lot of React’s internals which could potentially change at some point in the future. It relies on me knowing certain things about the way React works and working backwards from that. We don’t even know like — what is the right way to present this information or maybe it useful yet. This is like the first proof of concept that like can we even build something like this right now. But I was able to whip up the first working version of that in like four hours.

JASON: I mean that’s cool that you’ve got the pieces here, right? That you are able to do this much. These are all things that initially I would be kind of — I don’t know how I would even start because like as you said, there is pretty hard core engineering that goes into not just like introspecting and out but observing it as it runs and modifying in a non breaking way as it runs so I as a developer can do actual debugging that’s not like imaginary. We are not hallucinating what’s happening in the app. We are just running the code but you have kind of given us like what? Middleware for code execution if that makes sense. I mean it is very, very cool that this is possible. I know you have a lot to show and I don’t want to keep you from doing it but I do have a couple questions about — we have mainly looked at React. So my two points of curiosity are one, do you need to have the Replay browser to use one of these recordings and inspect?

MARK: So there is two different answers to that question. The way our recording technology works is we fork in the software development process sense the actual browsers. We have our own version of the Chrome repo, our own version of the Firefox repo, and we have added thousands of lines of C++ and JavaScript to the actual browsers to enable making these recordings because what we are actually capturing is at the operating system level. So every time the browser opens up a socket,, every time it receives data, makes a call to a C library random function, every bit of external interaction gets recorded and that’s what is saved and then we can rerun the browser process in the cloud and feed it back all the information that got recorded. If you made the recording at 3:00 on Firefox 97 in Japan, when it is rerunning it thinks it is 3:00 on Firefox 97 in Japan.

JASON: Sure. Now let’s talk about the hard stuff.

MARK: Yeah, that’s the deep magic engineering I am talking about. Once you have made the recording, once you have made the recording, what we are looking at right now is just a normal next JS redux React application and you can do the debugging in any browser — Firefox, Chrome, doesn’t matter what was used to record it. This just a website and I am looking at the website and using it. It does get back to right now our primary recording browser is Firefox because both founding engineers were Firefox DevTools developers. We are working on chromium support. We have Chrome for Linux working. It is still missing pieces and the runtime engineers are starting to build out support for Chrome for Mac and Windows over the next few months. Our plan is that within the next few months, Chrome will be our primary recording browser and that’s what most people would use.

JASON: Very cool.

MARK: Another point I will make with that is we have a feature that we are working on. What we did here was we made a manual recording. We opened up the recording browser, hit the record button, used the app, Etc. We are working on integration with Playwright and cypress end to end test suites. That’s especially where the Chrome for Linux stuff comes in. The idea is you configure your end to end test suite job to run those tests using our recording browser and then you push a PR, it kicks off the end to end tests, 50 tests go run, and we make recording of all of those and if you look at replay’s dashboard. Like you have the list of the recordings you personally have made but there is also a test suites view that shows you that OK, we pushed this PR and you won’t see that view because you are not part of a test suites team but there is a view that shows here is the PR, it was pushed, the branch name, that ran 50 tests and 47 passed, 3 failed and here is the recording and you can pop open the video and debug the end to end test even though it happened in a GitHub action somewhere.

JASON: It just builds on the promise of what tools like cypress have been getting us closer and closer to and the idea that Cypress will show us what went wrong and now Replay will use Cypress to not only show us but let us go in and tinker with it. That is very — I mean it just gets us closer and closer to like being able to effectively go into user’s machine and just try stuff there instead of asking them to, what browser was it. It feels like we are building closer and closer to these things being just possible. OK. We have about 5-7 minutes left. I have one other question before we continue. We focused a lot on React but I want to make sure. Is this for any app debugging?

MARK: Yes. So because we do the recording at the browser level, we record everything that happens in the browser. A lot of our developer experience, our features are kind of focused on React because we use React. React is very popular. But you can use this debug any code that runs in the browser. It doesn’t matter if you are using Vue, Angular, Solid, vanilla JS, backbone. Our CEO Jason went off and recorded random websites. We found a bug in StubHub’s back end code a while back just by recording their own check out flow.

JASON: Very cool. Very cool. With our last 10 minutes, what do you to show?

MARK: We talked about all these features. I could talk about them a lot more but the last thing I want to show off is the potential of Replay that we have barely even begun to scratch. The next to the last link under other resources. There is a replay protocol example repo. Go clone that repo locally real fast.

JASON: OK. Let’s get — oh, boy. You know —

MARK: Entirely too many project folders.

JASON: Yeah. Let’s go with git clone — is it going to let me do? No, no, no. Trying to be fast and it is making me slow.

MARK: I think you can just clone the http. We will see. I mentioned a few minutes ago, Replay’s back end as an highway — API — and that’s how we built this and did the imageal magical features on the front end. This demo will show non-front-end examples of talking to the API and doing other things to demonstrate the basic concepts and it as a like look what you could do. Let your imagination run wild. Go ahead and open up that folder in VS code. In the terminal go into the coverage extractor folder and run Yarn.

JASON: Oh, do I have Yarn? I do. Cool.

MARK: I wrote this middle of last year. Earlier we saw the hit count feature. It will show you how many times each line of code got hit. I am sure you have seen test runners like jest have a dash-dash option telling you 87% of the lines of code in this file got hit. You should probably increase test coverage Etc. The coverage reporting can be done in several ways. One is printed out at the console as part of the test output. You can often have them spit out an HTML file containing like the structure of the source code tree and the contents of each file and how many times did each line of code run information in the HTML information file so you have an artifact you can look at later. It occurred it me what would happen if I smashed these two ideas together? I have hit count information from the API. I know what libraries get used to generate the HTML files. What I tried doing both of those? I wrote a small example script that does this and I hopefully have it configured correctly. Do a Yarn start. Fingers crossed. Oh, no. Oh drat. I don’t think I — did I not push the change? Give me just a second here. I am going to try to — drats. Let’s do this. I don’t know the right way to send you the ID at the moment. In your browser, let’s go back to replay where we are looking at the code sandbox example. If you look at the URL right after app—is a good all of the way up to the question mark. From the C9 to the question mark. Copy that. Go back to VS code terminal yarn start and paste in that recording ID and I am hoping this is going to work. Please. Please. Yeah, there we go. It is doing things. It is actually doing too much. OK. We are going to fix this fast. Hit control C. Open up the source code. I fixed this locally and forgot to push it. Open up main.ts, scroll down about 20 lines and keep going. There is a spot where we are doing some string checks. OK. Line 55 add an exclamation mark in front of the double equal Webpack. Hit save. The problem is because this was in a sandbox it was picking up some code san box files and I want to exclude those. Try this again. That looks bet SKWR — better and like a list of just source code files. Pull up a file explorer because we will open an HTML file browser off disk. There is a coverage folder in there. Open up index.HTML in the browser. There is our HTML code coverage report. If you click on say source features counter, counter slice, there is the source code contents with the hit counts.

JASON: We never hit the failed state.

MARK: Yup.

JASON: Cool.

MARK: Again, to clarify, what I did was we used the same API from our back end that our front end uses to build all the features. I asked it what was the list of source files in this recording. For every one of those give me the source code contents and the hit count information and then I took the same HTML code coverage report library that Jest uses and figured out how to format that hit count data in the same data structures it needs and generated code coverage. That shows us what code ran in this recording. Like even that by itself maybe sort of useful.

JASON: I mean it is super useful. If I expected that some data manipulation was going to run, it could be that it is running and not working or it could be that I screwed up and forgot to call it in the first place and being able to quickly gut check that I did in fact call the code I thought I was calling, is a huge timesaver. Even if it is not a huge timesaver it saves me cycles and those cycles add up. With that, Mark, we are out of time. I am going to send everybody to your Twitter. I am going to send everyone to the Replay website. And what other resources should people be looking for if they want to get started?

MARK: The best page is https//replay.io/record-bugs. We have a short link to the page. Replay.io/record-bugs. It will redirect to that particular page but that works. That’s the biggest thing there. I linked it but I have a talk with slides I have done about debugging which expands on a lot of these principles of thinking through the problem.

JASON: We have had live captioning. Maggie has been here all day from White Coat Captioning and thank you so much, Maggie. That is made possible through the support of our sponsors Netlify, NX, New Relic and Pluralsight. While you are checking out things on the Citgo and check out the schedule. We have a banger of a schedule coming up. We are knowing to do real user measurement stuff. We have Sunil coming on to talk about his Partykit which is real-time collaborative and they are always good for talking about the future. And Mark Debossy is coming on to talk about shaders and cool stuff that will make the stream more fun for everybody. And episodes I haven’t put on the schedule yet so mock sure you head over and check them out. Get the thing, subscribe on YouTube, do whatever it is you got to do. This has been a blast. Mark, any parting words before we call this one done?

MARK: My main advice is you can learn how to debug, you should learn how to debug it will be a huge boost for your career. There is all the general purpose debugging stuff that you can and should learn and in my very obvious bias opinion everyone doing web development should try out replay and see how much time time traveling debugging can save you.

MARK: Thank you for hanging out and taking the time with us today. We will see you all next