Create a Timewarp Filter in TensorFlow.js
with Gant Laborde
Let's recreate the popular TikTok timewarp filter using a webcam and TensorFlow.js. Machine learning's favorite mad scientist, Gant Laborde, will teach us how!
Resources & Links
Captions provided by White Coat Captioning (https://whitecoatcaptioning.com/). Communication Access Realtime Translation (CART) is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.
Jason: Hello, everybody, and welcome to another episode of Learn With Jason. Today, we're bringing back everybody's favorite mad scientist. How you doing?
Gant: Fantastic. It's been too long since we've seen each other.
Jason: I know. I still think about, you remember the last time you were on, we did that Halloween mask. That was so much fun. I'm going to find a link to that episode that we can show.
Gant: That was a really good one. We actually did a twofer, if I remember correctly. We did Halloween, so we had Halloween got like the masks kind of on and then we did another follow up episode where we were doing it like, where they were turning their heads and we're still putting it on people. I think that would be a cool Chrome plug in or website feature.
Jason: It's something that's been on my back burner. You know what would be a lot of fun is to have a button on your website that would find your face and put something on it. It's fun. It's goofy, surprisingly approachable. I just dropped the link in the chat for anybody who wants to see it. For those who haven't seen you, do you want to give us a background on yourself?
Gant: Absolutely. Gant Laborde, been coding for 20 years. Really good at it for three.
Jason: You were like really good at it? Dang. Beautifully done.
Gant: I love working with intelligent, smart, fun people. I'm a CIO of a company where we basically we get to create cool stuff for people all the time. You know, we build mostly react native mobile apps constantly. Pretty big names come through our door and we get to show their teams how to do cool stuff and have fun. I tell people we work with puppies a lot. You know? We get to see all these cool projects. It's a lot of fun and one of my favorite things is honestly like teaching, educating, working with people. Sort of have to have that for consulting, right? And going to conferences, which is where you and I met.
Gant: And quite a few other conferences after that.
Jason: Yeah, to the point that I just kind of expect to see you whenever I show up in a crowded space.
Gant: I'm not even speaking at them. I just show up. In the speaker's lounge. I'm with Jason.
Jason: I see some familiar faces. I see Justin Husky. One of your co‑workers.
Gant: If you've ever gone to any of our conferences, he's lead of design. So everybody's like, oh, conference is so beautiful. The designs are so great or this blog post has cool stuff. That's him and his team making sure I don't look like trash.
Jason: Yeah. I'm still wearing some socks that Justin designed. I see Darren in the chat. Chris in the chat. Both Chris' in the chat. Chris, you're calling me boomy. Does that mean I'm like audio problems or that I just put on my radio voice today? The auto tune.
Gant: That's a scary story I'll have to mention sometime. Yes, it's good to see all the fun people in there.
Jason: Absolutely. We've got Ben dropping boops on us. We got everybody in here.
Gant: This is your virtual conference happening right here.
Jason: My favorite thing that ever happens on this show is when I completely lose control of it.
Gant: Well, we'll have to make sure that happens.
Jason: So, one of the things that I think you are particularly known for is you run fun machine learning. And you have released a product on machine learning. The course, right? Or a book?
Jason: You can drop it in the Twitch chat, too.
Gant: My favorite thing is I always wanted to write a book for O'Reilly and when they came in, I said no, I'm not writing the book for you. I can't do it right now. Then the pandemic hit. I was like, oh, it was a good thing I didn't say I was going to write a book. No one could write a book during this. Then a whole bunch of stuff happened then I kind of like turned into crazy writer mode. You know, like the one where you see them, they have a typewriter, they're hunched over.
Jason: You went manic is what you mean.
Gant: Yes, I did.
Jason: You wrote as a coping mechanism. I unfortunately made pizza as a coping mechanism, which means I've been on a diet for a while.
Gant: Pizza as a coping mechanism is probably a lot more fun. I was creepy crazy guy in the corner. Like you know, just nothing and then type, type, type. I was like, hey, O'Reilly, here's a book. Here you go.
Jason: Your wife's like sliding food under the door. Don't go in there. Like protecting the child.
Gant: Is it Tuesday, we spray the de-scenting spray underneath the door. That's what happens. So weird.
Jason: So congrats on that, by the way. A book is no small feat and an O'Reilly book, I'm looking at it now. Looks like you got a tortoise as your book cover, which is amazing.
Gant: Yes. I was like, hey, I'm in New Orleans, can I have an alligator? They're like you get a turtle. Okay.
Jason: I'm not mad at it. So, okay, so you're doing a whole bunch of stuff in the machine learning space. When you've been on the show in the past, we've done stuff in the machine learning space. Today, we're going to make TikToks.
Gant: So there's this kind of cool, here's a weird part of what kind of comes in with machine learning. Is data is all over the place in all these mad kind of spots and one of the cool things that happens is images as input meant that you know, for instance, machine learning got faster when they were able to hook it up to the GPU. Cool. Then also, when the inputs there is images and content like that. You usually have to crop stuff or find people's faces or do all these cool little things. So secretly attached to TensorFlow.js, the framework for this, which checks in on your GPU and does all this other cool stuff, is this ridiculous graphical GPU accelerated image and data collection thing. So I was watching a TikTok because I'm young.
Jason: Greetings, fellow kids.
Gant: Greetings, fellow children. Let's go skateboarding. I would kill myself if I skateboarded. So I was watching it and there's this one really cool filter that they do, which people were having fun with. Then of course, my brain goes, you could do that. Like that's just some of the image manipulation and cropping and stuff that they use. That's all inside of TensorFlow.js. So like here's an evil version of what happens here. You can use TensorFlow.js to do all kinds of cool graphical things besides just machine learning. So I can mix this with machine learning. I can mix this with graphic stuff. I said, you know what, let's do that. I was thinking about, I know, I have, like I think maybe, like it would only take us about seven steps‑ish, maybe.
Jason: What's the over under on this? How close is Gant to correct?
Gant: Subtract steps to be right. You're messing with the wrong guy. Still step six.
Jason: By‑laws out here. Section 2.2A.
Gant: Subsection tetrahedron.
Jason: So I'm excited about this. To just repeat this back, what we're going to do today is we are going to ‑‑ thank you for the host and for the boops. I love it. I love the chaos. We are going to take TensorFlow.js, which is to my understanding, as you've said, a machine learning thing which in my head is not associated with TikTok at all.
Jason: But what we're going to do then is we're going to take that machine learning library, I'm going to turn on my other webcam and we're going to do weird things using the machine learning. We're going to do it all in exactly seven steps.
Jason: I am very excited to see how this goes. Let's make this happen. First and foremost, here is Gant's book. You should go and get yourself a pre‑order of that. Also, Infinite Red Academy. This is some Justin Huskey design, right?
Gant: You see that amazing stuff like him, and honestly, him and his team are like, he helps like really orchestrate all the stuff together. Jenna does a fantastic job. He like reaches out to people, but that whole design has been under the tutelage of like, if you had seen the stuff that I threw at him before this, you would appreciate this.
Jason: Look, I drew some stick figures on my napkin. Here's a little bit of grease. Don't worry about that.
Gant: You see this napkin right here? Make this internet.
Jason: Good. Good. Okay. So.
Gant: Let me send you, I grabbed just to make sure, here's a TikTok and I could put it over here inside the chat as well.
Jason: Let's do watch one of these.
Gant: Here's a person giving themselves a Jason Lengstorf head. See, look, there they are.
Jason: Here's a person. Wait, wait, wait. I did a thing wrong.
Gant: There you go. They've got the Jason hairstyle.
Jason: Wow. That was like some excellently done shade. That is absolutely atrocious.
Gant: Perfect. Let me get you another, what's happening here. So the idea is we are basically scanning down and then put it inside of this. Some people said it's their first TikTok. You're in trouble now.
Jason: Get ready. TikTok is weird.
Gant: Here's like a person making a fake helmet. This is hilarious. It's just scanning then they move the stuff right there. As the bar is going down, it's basically like freezing pixels.
Jason: So this is effectively like when like in the old days of the internet when you would download an image and get it one line of pixels at a time. We're building images that way using this filter.
Gant: Yeah, that's it. And the stuff that people have done on here is hilarious. Let me send you one more. I'll give one more here.
Jason: What are we doing? Ben says trolling Jason.
Gant: That's the most correct answer so far.
Jason: Oh, good. I'm glad the submarine is on my face again. I'm going to have to fix that. We just built this on a stream last week where now you can control a submarine. Or if you turn it into, right now, we've only got two. We've got the submarine and the dumpster fire.
Gant: Perfect. We will use that for sure. Yeah.
Jason: But what I didn't anticipate is the only thing the chat wants to do with this is put it on my face.
Gant: Block you out. I love it. Dumpster fire eyes, that would be the most important thing.
Jason: I know. One of these days.
Gant: I sent one more TikTok so you can get the idea here, the simplicity of it. This person's like doing it like artistically, so.
Jason: Let's look at this one here.
Gant: It's like the finality, and there's a million things you can do once you can do this. It's pretty crazy. Go down, grab it and moves her mouth as it comes down and it makes this really crazy, possessed photo.
Jason: That is nightmare fuel.
Now I'm a crew mate.
Jason: What did I do?
Gant: I don't know.
Jason: I actually, before we get too far, let me do a quick shoutout to the sponsors because what I thought just happened was I thought we had crossed a line and Aimee who was captioning was like, nah, I'm out. So we have Aimee with us today doing live captioning with White Coat Captioning, which is, it means so much to me to make this show more accessible to more people. So I'm really, really happy. Thank you, Aimee, for being here today. That is available on the homepage of Learn With Jason if you want to get the live captions. We've got Netlify to make the show more accessible to more people which means a lot to me. Head to the homepage and you can follow along with those. So, that gives us some context. Also, make sure you follow Gant because Gant is a very funny person to follow. And I don't know if you didn't see this, this is sort of what we're trying to work on today is this idea of being able to make these weird things inflow. And so I think we can, I'm ready now.
Gant: Perfect. Let's do it. So let's dig in. The trick here that we're going to do is first thing that honestly is the classic here is just getting a webcam set up on a page, right?
Jason: Let's do it.
Gant: If you want to, we could, just creating any kind of classic website.
Jason: We can make this and then we'll get in it then I'm going to open this thing up.
Gant: You have the history of the terribleness we'll do.
Jason: Oh, yeah, we can't just let this fade into the ether. This is on record now. We have to live with the consequence of our actions. Well, well, well. If it isn't the consequences of my own actions.
Gant: How many pizzas is this going to cost us?
Jason: One of the things I love, my partner, Marissa, she can always tell how stressed I am if I want to order a pizza.
Gant: Like a deep connection into the universe right there.
Jason: It is. It's the thing you really want. For your partner to understand you at the ‑‑ TikTok timewarp filter with TensorFlow.js. So then I'm going to put just a, I'm going to copy paste this down, actually.
Jason: Okay. Then I'm going to, you know what, let's just make this thing run like. I'm going to add, oh, I'm going to rename this folder, actually, because we don't need to rebuild it. We're going to move source to public.
Gant: Sweet. Good.
Jason: Add public. Got that. Now I'm going to commit it.
Jason: Work in progress. We've got basic HTML we're going to get to create. Learn With Jason, and this one's going to be called TensorFlow TikTok Timewarp.
Gant: I love that you pronounce things like I do when you type. Kick, kick.
Jason: Then we're going to get push domain and finally I'm going to define it and we'll get this thing rolling. So this is going to go on my nobody look at my beta teams. Okay, it, no.
Gant: Nothing to see over here.
Jason: TensorFlow Timewarp. Is that what I called it? Yes. Got it backwards. I'm going to name it the same thing or else there's zero chance I'll find it. TensorFlow TikTok Timewarp and we don't have a build command. Our deploy directly is public. Leave the rest of those defaults in there. All right. And so now we go and take a look at this. It's already going to be live probably.
Gant: Yeah. Doesn't need to do too much.
Jason: Let me get the actual domain name here. There it is. Okay, there's our site. And now we can nullify dev and now we've got a local version of it. That if we go and make some edits, so we can just say to do. Add webcam. Delete that nonsense that I made. And refresh the page and it is here. It's not here. Why isn't it here? Where did I ‑‑ oh, dammit. I'm going to delete this one that I'm not using anymore. Save that again. Reload. I'm ready. I want to get this right and the site is ready to go. We'll just push as we go.
Gant: Yeah. So for me, we're going to skirt around any kind of front end for right now, but I do promise whatever we don't get to in this, I will come back and add all kinds of features.
Jason: I will find you.
Gant: I have a very special set of skills. It includes terrible TikTok dances.
Jason: Are we going to get one of those? Yeah, let's make Gant dance today.
Gant: I wouldn't mind.
Jason: Gant, have you been back since we ‑‑ did you know we made that?
Gant: No. That's just not something I'm willing to do. I love that. I'm immortalized.
Jason: Okay. Now I want to add my video. So to add a video, am I just straight up doing a video?
Gant: Yeah. Just do straight up video like that. And then what we're going to do is so, let's go ahead and give it like an ID. What we're going to need to do here is because I think we're going to overlay some stuff. You know they have that cool bar that comes down? So wrap this all in a div. That way, video, a canvas that's going to be the one that's like the locked pixels on top of the video and then on top of that, we'll do that little like slide down bar that kind of comes down and makes everybody see where it's locking pixels.
Gant: It will just be putting things on top of each other. So at the bottom of that, I think you can put the video. So because we'll put everything on top of the video.
Jason: And then the bar will go on top of the canvas.
Gant: So video first. You'll have to flip this because the Z order is going to be whatever.
Jason: Z order is whatever I tell it to be. ZX999. Find out what the overflow energy limit is for CSS.
Gant: You find the last nine and just keep it there. Good luck overflowing this.
Jason: Okay, so.
Gant: Webcam, perfect. Then the filter.
Jason: So confusingly named. I'm like, I don't know. Maybe I'll use a data attribute, I don't know.
Gant: So to kind of get these things on top, I don't know how you do it, but I usually like, the div, I do position relative then you can set like a height on it. So the container div there. Good. Look at you.
Jason: We'll take our overlay and we'll do a position relative.
Gant: So professional not doing in‑line styles.
Jason: You know, I've got a reputation to uphold of severely overthinking all of my CSS.
Gant: Yes, indeed.
Jason: Let's add this wrapper on here and then our wrapper, oops, get away. We'll do a main and make this one like a margin five. Yeah, that's fine. Then we'll give it an auto. We'll give it a display grid.
Jason: Display flex, actually.
Gant: Takes me back to what I know.
Jason: With this one, we can justify content center. And then we'll set like a gap of, I don't know, we'll give it two. How about that? So then in here, what we've got is position relative, which should give us a pretty good set. Maybe for this we can do like a max width of 90 with an actual width of like 500 pixels. What we should see when we refresh this page is it's kind of doing the thing. All right. That's sort of close. Let's make this h1 a little bit smaller. Font size of like 1.5.
Jason: I have a font family.
Gant: I just go on the show to torture Justin with my terrible styles and you're undoing my torture here.
Jason: Sorry, can't help myself.
Gant: Already looks decent. Damn you.
Jason: Make this a little bit wider. What's going on? Why isn't this centering? No, come back. Please, okay. You are the right width. You are absolutely not the right width. Oh, because guess why, everybody.
Gant: Why? Oh.
Jason: So that actually makes life a little bit easier. We can take it back to 500. Everybody's happy again.
Jason: Life is good. So then we can take our overlay and let's just give ourselves a border of like two solid red so we can see what's happening. That's where our thing is going to go.
Gant: Love it.
Jason: Then everything else that's inside of this is going to be, we're going to have the webcam be positioned absolute. It's going to be top left width and height of 100%. For now, let's give it a background of like blue so we can make sure that did what I wanted.
Gant: That's pretty close. We probably want to give this a, let's see, so we can make the height zero.
Jason: And we can put the padding bottom at 56.25%, which I think is 16 by 9? Close enough. That's just a thing that I have memorized because of ratio containers.
Gant: The 16 by 9 equation. It's forever in Jason's head next to pizza.
Jason: Exactly. This one is actually going to be the same thing so we can just reuse this. Then I'm going to give this one a background of orange. So it's on top.
Jason: Then for the bar, we want the bar to be positioned absolute.
Gant: Same everything.
Jason: But this one we want at the top and we want a width of 100%. Then we want a left of zero, but we want a height of like ten pixels?
Gant: I would do, make it the entire thing and we'll draw, because this is a canvas, we'll make the bar a canvas and we'll just draw the bar along the canvas. You can just make all these the same size. As long as they're on top of each other rather than moving the bar along. One of the things that will happen is every time we move the bar, we'll have to clear the bar canvas and draw a new one at a different height and clear it and draw another one.
Jason: This is not a div. This is a canvas.
Gant: Let's move that to a canvas as well. Although with a div sounds fun.
Jason: We're approaching seven steps already here, Gant.
Gant: I don't know what you're talking about. This is step one. Get the webcam on.
Jason: Okay. So this seems okay. I'm happy with this. I'm going to, just one, I'm so sorry. I'm so sorry to everybody.
Gant: Actually appreciate that you're fastidious with this stuff because I am not. Then I basically have this like entire set of tech debt, I have design debt.
Jason: Okay, so you've got that. That looks more like what I want. You know what I'm going to also do? What I want to do is I want to maybe just line-item center. Because then we've got vertical centering.
Gant: We need to do two semicolons. It makes it very center. Sounds like my dogs are trying to chime in a little bit. They have some design opinions.
Jason: You are supposed to have no margin. Am I helping? There it is. Look at that. Vertically centered, no scroll, we are so ready. Break something though. My overlay is supposed to be max width 90. Width 500. Why did you lose your sizing? Got my overlay. Got my main. Are you supposed to be just width 100%? Is that the thing? Oh, there it is. All right. So now we have some styles and that's all the time we have today, everybody. Just kidding.
Gant: Today, brought to you by orange, pink, and red boxes.
Jason: Okay, so I'm happy, I think. We've got this thing mostly set up. I'm going to take the border, well, I'll leave borders and everything on until we decide it's time. So with that being said ‑‑
Gant: Now we get into, it's time to do the web, okay, perfect. Yeah. We need to go ahead and get that webcam set up.
Jason: Let's do this.
Gant: We need to create a function that's going to get run when the website's done. So we'll go ahead and call it, yeah, webcam, sounds good to me. The first thing we're going to need to do when we're setting up the webcam is get access to ‑‑ this is classic. There's a navigator.mediadevices to get user media. We can kind of check to make sure that we have access to that and if as long as there's navigator.mediadevices.getusermedia that exists, they have a webcam. We could do a little alert saying like, you don't have a webcam. This one can be, yo, get a webcam. What's up? What did you do last year? Sat. Appreciate you having more of a message there. Perfect.
Jason: Okay. So, no webcam.
Gant: Let's go into, they do have webcam or you can be nicer and give them more messages. Perfect. So we assume now that if they've gotten here, they have a webcam. So what we need to do is, can you make this an async function for me?
Jason: I can do anything.
Gant: And then what we'll do is let's, you could go ahead and call this from a script from the body tag or something or just call it right underneath this.
Jason: Just do it this way.
Gant: Love it. So there's all the hoopla on this kind of setup where you have to get the stream.
Jason: Does all this work, hopefully?
Gant: Let me send you some terrible code. See here. Classic.
Jason: Please send me your terrible code.
Gant: There's some terrible code and it's terribly formatted.
Jason: So we're going to get the webcam stream and that is going to be awaiting the navigator.mediadevice.getusermedia.
Gant: This is where you tell the properties to the webcam. So we're not going to care about audio and we're going to grab the video that's facing the user. This is awesome for mobile. Way less.
Jason: Here we go, stampede. Yeah.
Gant: The '80s made fun of this would happen and this is what the future would be on TV shows, but here we are so I'm really glad we did it.
Jason: We did it. We ruined entertainment, everybody. Congratulations, '80s. Lived up to your worst projections.
Gant: We need to get a reference to that video as well. We're going to need to get element by D for that video.
Jason: Let's get the, and this is for the actual video element.
Gant: Yeah. So get video ref.
Jason: Document.queryselector then we'll get the webcam?
Gant: You could do that. Or get element by ID. This way is fine, too.
Jason: This one feels, because it's the same because you get classes, you get IDs, you get whatever. It's just how my brain works now.
Gant: Nice, I like it.
Jason: I don't need to know multiple things. I only need to one thing.
Gant: Perfect. Perfect.
Jason: Okay, so this then is checking.
Gant: That's just not something I'm willing to do.
Jason: Correct. That is correct.
Gant: Here we are. We're getting the webcam. Now you can kind of like protect against the old stuff right here because they have some of them. So what I would say is you look here, you want to set the video's source object to the stream that we're getting here on line 85. So you can set that there. But what happens is some, and this is just like the pain is setting these things up needs to get easier. We need to start creating some components to do these things in my opinion. But it's fine. Then if they don't have that, this is the old style. Actually, you know what? We don't need to do the old style if you don't want to.
Jason: For the sake of making sure we don't run out of time, let's stick to modern only. But this will work in basically everything but IE, right?
Gant: I think so. Yeah. We'll find out today.
Jason: Yeah. I think the thing that's interesting to note is a lot of times, we talk about browser compatibility, but these days, the only real browser incompatibility problem we run into, internet explorer is the only thing, but Edge, Chrome, Safari, they support the modern stuff. So unless you're using something very new, you don't have to do the full fall back because most browsers in use are supporting almost everything.
Jason: Okay, so we've got this. We're setting up our webcam. Does this mean if I refresh, I should get a request for my webcam?
Gant: I think so. Yeah.
Jason: It wants to webcam. Okay.
Gant: Got a little question there. Love it.
Jason: So now because everything is stacked, we don't see anything. So what I'm going to do is turn off the background. And then turn off this background and there's my face. Oh, it's frozen because, you know why? We didn't tell it to play, right?
Gant: Right. You want to set auto play.
Jason: We need to make it muted so the browser will let it auto play. I'm going to turn off these backgrounds because now we're working with video. So let's make that work. And it did not remember. Did you immediately forget what camera I wanted to use?
Gant: That's the latest and greatest there.
Jason: I think this is like a Firefox thing. I don't know why it doesn't remember this stuff, but it never, ever does. So you're using also the wrong one, but it has a ‑‑ oh, my goodness. Killing me. All right. So then if I reload, it should use the right one.
Gant: The modern world. There you are.
Jason: Now if I look at this camera, that's my day‑to‑day. Camera one, camera two.
Gant: Thank you, Wayne. Perfect. Yeah. I love the movie references. We're old.
Jason: Prefer to think a finely aged vintage.
Gant: Yes. Perfect. We've got the webcam and now, what we can do now is essentially we can gather the contents of the webcam, right, and then we can just, you know, let's get back to the line eventually, but let's just kind of go through it and then chop off each of the pixels. Now, here's one thing that can happen with this. Is that we could like continuously chop out a set of pixels and another set of pixels then like grow it, but what would be really cool is if we chopped it, if we could stick that on a canvas and then all we have to do is like paint the chopped out part as we kind of go down. So can you create for me a, let's create a canvas, it could be anywhere on the page and what we'll do is we'll hide it with display none. This is where we're going to like paint the chopped out pixels as it's like going down. Then we'll just be able to draw that to the result we have like overlayed. So this will sort of like be our hidden work. Did you call it the chopper?
Jason: Chopper painter. So we've got a hidden chopper painter.
Jason: And now I'm ready to do things with it. And so this is the part that I think, like I feel like we can get pretty close. How do I get you to stop showing me that? Please stop. Do I have to click a button? I don't know. I don't know what I just did. Everybody stop yelling at me. So this is the part where I think we can imagine thing, but now we have to get the pieces to fall into place. Because getting a webcam, I get. TensorFlow, I get. Actually, running things into this, like between the two of them, now I don't even know where to start. So if I want to do this, what's my first step here?
Gant: Okay. So here's what I want to do. Let's go ahead and add a button. Just toss a button on there.
Gant: What we'll do is when we press this button, we will tell TensorFlow to like take a picture of the webcam. Like take a moment from the webcam, right? Then if you want to, actually so we can even see things, that hidden canvas down there, we can like take it from the webcam and immediately paint it down there. This will be like a moment. Press the button. Take a frame from the webcam and then paint it directly to this like sort of like intermediate canvas for us.
Jason: Okay. So let's do that. I'm taking off the display none so we'll be able to see it. And I have created a button. There's a button to press. We do not have ‑‑
Gant: And click, oh, yeah, you're going to make it fancy. I know.
Jason: This way, we can, actually, we only have one button. Let's just do the button. So down here, we've got the button is going to be document, query, selector button. And then button add listener. Click. Then we'll get, there's not really an event so we don't need to listen for anything. For now, we can ‑‑
Jason: So then come out here, verify. All right. That's doing at least some of it. And then from here, what's, so okay.
Gant: Here, let's bring in TensorFlow.js, which sent you a little script flag in there inside of Zoom. Just a CDN, bring it in.
Jason: Let's find that. Where would I find that on this home page here? I'm going to throw in link in the chat.
Gant: That's a good question.
Jason: If I go to get started maybe.
Gant: I would type TensorFlow.js CDN and just grab the first one. It's under tutorials then set up. Look at that. There you go. There's the CDN.
Jason: Got it. So then I'm going stick that above here.
Jason: And we're going to make it blocking because we need it before this runs. Okay. I'm going reload the page and make sure that nothing explodes.
Jason: I'm actually going to stop looking at that one because I'm going to remember we did it in here. So we've got our TensorFlow is loaded, but not apparently doing anything. So then in here, we're going to be able to do stuff with it.
Gant: Yes, indeed. Okay, so when we press the button there, here's some of the magic there. Let's say we wanted to get a image moment from a, from a picture. Let's say we wanted to get it from a video. Let's say we wanted to get it from a webcam. It's the same thing. Because machine learning depends so much on like data, they've normalized all this for us. So when we want to simply get, get that information from there, we can get a TensorFlowish like representation of that information with, because you have access to TensorFlow in here, you can just say current frame or whatever you want to call it or my tensor, anything like that. It equals tf.browser. So now you have access to TF.
Jason: Just lower case?
Gant: Lower case tf.browser.frompixels. This is like magic. It gets GPU accelerated. So now we need to have a reference to the video.
Jason: So I need to make this outside of our ‑‑
Gant: Is it? Perfect.
Jason: Why don't we do this. Let's put this one here. So we'll kind of have our, we'll load our elements. Have our function definitions then logic down below. That will make sense to my brain and help me find things later when I inevitably forget what we're doing at all.
Gant: I love that you organize as you go.
Jason: I have to man. It's my only defense.
Gant: It's so good. That's my, that's like the first thing I do whenever I'm working with someone, too. I'm thinking about where we're going and you're making sure it's organized. It's perfect. So this grabs like a current frame from what's going on there and then like so ‑‑
Jason: Hold on. That's it?
Gant: That's it, dude. You just grabbed it and it pulled in. If you had given it a jpeg, it would have snapped that. It's all GPU accelerated, too. Honestly, the first time this thing runs, it will probably go a little bit slower, but it's in a tensor form right here, but it will go slower. But then once it's got hold of the memory and GPU, we can do it faster. So to put it back on there, there's a command that you can use, which is tf.browser.topixels. And then you just go ahead and you give that the first thing is the current frame and the second parameter is the canvas. Yep. Then that should, when you press the button, take like a snapshot moment of your webcam and then paint it for you inside the canvas. There you go.
Jason: Hold on. Oh, it only did the one. Fine.
Gant: Did you press the button?
Jason: Wait, did I miss the button? I may have just missed the button.
Gant: Let's see. That should be there. There you go. That's it. I don't know why we're not selling this.
Jason: Okay. We'll screenshot that one later. Poor decisions were made. Okay.
Gant: Perfect. So, okay. Now if we wanted to kind of like take your stuff and immediately put it there, that's fine. What we can do now because we have this magic of TensorFlow, we can cut like a thin ribbon of you and just like put that inside there and kind of like keep that as we go. So ‑‑
Jason: I'm going to ask for a check on phrasing. This is, please don't cut me to ribbons.
Gant: Too late. We have ways of making you talk, Jason.
Jason: I'm ready. So what we've done is so we've got our webcam loaded. It is using the webcam video. We have a button. When you click the button, we have TensorFlow, which we've loaded from the CVN. We get a still from the active webcam then we turn that into, through magic, I assume this comes in as like a buffer or something then this turns into whatever the canvas needs to display it as a frame. So the next thing that we need to do then is we need to, so we're going to do this on kind of like a loop where we'll break the image up either top to bottom or left to right in like if there's 100 pixels, we'll do one at a time. So we would do 100 loops and at each point, we would say take a screen shot, like loop number pixels over and grab that one pixel strip and then stick it in.
Gant: Yes. So now we're just like, we're slicing pieces of this thing in memory. So between like 103 and 104, you can do what you want with this like moment of like this current frame that you've caught.
Gant: So if we want to slice this up, the command is TF slice. So let's create a new variable. Because like it's very, it's immutable, always, these kind of things. So we'd have to create like a new one called like your wonderful slice here. And so it would be a TF slice. So the first parameter is what you want there. You want the current frame. Okay? And then the second parameter is an array and the array is like where to start slicing. So we could start at 000 and what that would be like is that basically at the top left. And then ‑‑
Jason: Is that like 000 ‑‑
Gant: Yes. Three dimensions. So yeah, it's indices. What happens is that first zero is up down. The second zero is left right. And that third zero is RGB. So we could do like cool coloration stuff.
Jason: I get it.
Gant: Just a red channel. Just a green channel. Things like that.
Jason: That's actually really fun. We should definitely play with that. Okay. So then now that I've got that, is it the same array or is it like a second array?
Gant: Then you have a second array. That's the starting point. And then the next one is, one for one pixel high.
Jason: One pixel high.
Gant: And I know you're weirded out, like why are we doing height then width? The answer is evil mathematicians. I can elaborate if we have time.
Jason: Good. Good.
Gant: Now we want to grab the full width. So we're going to grab like one pixel full line. So that would be, we can grab either.
Jason: Get bounding client.
Gant: Yeah. What I'm interested in is like what it's getting here might be bigger than what we're showing on the screen, but that's okay. We can mess with that.
Jason: I see what you mean. Let's try it and find out, I guess?
Gant: Yeah, we could do it ‑‑
Jason: So then I would put the slice in, right?
Gant: Yes, you'd put the slice in. And the last number instead of being zero, would be three. You want to grab all three channels. R, G, and D. You're just basically saying now, give me that. It will crash. Let's crash something.
Jason: Okay. Just do nothing. Negative size value should be exactly negative one but got undefined for the slice size at index one, which means my width is undefined because I used this wrong.
Gant: No worries. Let's do an evil cheat real quick.
Jason: You see that I got it working? I got a line. I'll make it higher so we can see if it's getting the right width.
Gant: Yeah. It's probably grabbing the actual video.
Jason: It is grabbing the actual video. It's going over to like here.
Gant: No worries. We can fix that in a couple of different ways. Easy enough to fix. But yeah, you just grabbed your hat. So like what's perfect about this is like if you just grab that, move down, grab, and move down and grab, you've kind of just made like a very fat timeline filter. Grabs entire chunks at a time, which is perfect though.
Jason: I'm happy with that. That's cool. So what we need to do then is we've got this bit, which is a, this is going to end up being a function. So we'll call this get slice and that's going to need arguments, but for now, we can ‑‑ let's do this. We will have this one be the element and then we need like the, actually, we should do it like this so we can understand what we're doing is element. We'll do height, width, and target. So then we can get a slice and that should, we'll look at our source element. Then we'll have our target element. And height.
Jason: Our width, we can make it 100 to start so if I break it, it doesn't fail. Then down here, we will call get slice. And we'll pass in the source element will be video. The target element will be chopper. The width will be one and the, or, no.
Gant: The height.
Jason: The height would be one. Okay. So if that works, then we should be able to delete all of this and nothing should functionally change.
Gant: There you go. Then a little cut of the line. Perfectly.
Jason: Change our height to 100 again, that should, okay. Now what we have to do is I guess loop this?
Gant: Yeah. Then we need to move that number down, you know, increment that number as we go. What we could do, so you're basically have taken slice, slice, slice. Here's the next thing we need to do is like take that slice and then draw that to the result context. Because what we're doing now is we've got, you've got all these different slices, but what we also want to do is like you have that overlay thing that they're doing and so we'll draw that. But if you want to, we can kind of, let's get the slices going. We should see it like the slice going down and we'll see those slices change. That should be fine. All you have to do is you see how we're starting at 000? We would have a counter between 0 to the height of the thing and we would bump that number up each time so that you would start at 100 then 200 then 300. Things like that.
Jason: Need to figure out what the actual height is. Do we have an ability to just say like videoelement.height?
Gant: I think it's from the context of the canvas, we can get the canvas's actual height before we get the client height. We could do that. Also, inside of TensorFlow, we can do it if we want to as well. But I know you're grabbing it inside of this function.
Jason: I guess we can hard coat it for now, but I was thinking in terms of like figuring this out based on the webcam.
Gant: Yes, dynamic. TensorFlow knows that size and I think the webcam size. Pretty sure there's a way to go ahead and find that out as well. It's probably like video.height or something simple. From the element.
Jason: Okay. Yeah. So let's do, I guess for now, we can do, I mean we can just kind of eyeball it, right? So we'll do ‑‑
Gant: Like 500 high.
Jason: Yeah, we'll start 500 high then do an I plus plus. Actually, this is not right because it's going to run them all instantly. So instead, I need to do a set timeout.
Gant: One of the things you can do is do a request animation frame before each call. So if we were going to do this recursively, let's do like, you have this function here and we could do like a do scan of it and when we, if you're calling it recursively until you know, it gets to the end of it or something like that, we could, in the recursion, request an animation frame in between. So that would probably be recursion would be better here.
Jason: So we'll set our video height at 500 and video width we'll set out ‑‑
Gant: So later on, we can adjust it.
Jason: We'll call it 700 or something. Won't be quite right, but it will make it work. And we'll do a video width here. Then what we need to do is you want to call this one, you want this one to call itself recursively?
Gant: Yes, just at the bottom of this. At the bottom of get slice. Yeah. Go ahead and just do get slice and then inside there, you could say request animation frame.
Jason: So the width is always going to be the same, but the height would be height plus one for now and we would need to figure out what that interval is.
Gant: Exactly. Before you do this here, do a ‑‑
Jason: Request animation frame. And the callback will be ‑‑
Gant: Inside there, it would be, you could do like the little, perfect. There you go. This will make sure that it's waiting between those.
Jason: So it will do it once per frame, so about 60 frames a second. Is it 60 or 30?
Gant: I think it depends on the browser.
Jason: Okay, so then the other thing that I need to do is add a check for when it's done so that we don't end up with ‑‑
Gant: Goes off the edge, right?
Jason: So we're going to say if height plus one is greater than video height return.
Jason: Or actually, I want this outside, don't I?
Gant: Yeah. I would say that as soon as, so we're going to need to like have a counter, right? And the counter, when you press the button, we can start at like zero. Then and so ‑‑
Jason: This will be our counter because we're passing in the video height.
Gant: Is the height from our code, is that how much you're cutting because like we don't want to ‑‑
Jason: This is going our total video height. So basically what we're going to do is take that video height and start at zero pixels and add one until we get to 501 then we'll stop taking slices. So we'll basically run this 500 times.
Gant: But what you're seeing here, so one little, that does work. Okay, good. I was looking at how height was used. One thing I'll say is we need to remember, okay, I see what's happening here. You are, instead of cutting one pixel then you're cutting two, then you're cutting three, four, five, six.
Jason: Oh, no. Yeah, you're right.
Gant: You'll want to cut one pixel at one. Cut one pixel at two. So that slice right there on line 109, you'll want to, that very first index zero is what we're going to want to have increment and then you'll want the height to always be one pixel for now. But we could like make different chop sizes.
Jason: Yeah, that makes sense. That makes sense. So the height is going to be, so I don't actually need that.
Jason: All right. So now we're going to get one pixel. Right? Then we'll scan down and when we get the last one, good. So now the challenge with this is that right now we're only sticking this into the target element.
Gant: Right. So we should just see one pixel get created and then it will override it again. One pixel. One pixel. So essentially, we'll see the things that need to get frozen.
Jason: Let's run this with a little bit more size so we can actually see things change. So I'm going to push this button and what we should see is that it will move down the size of the video.
Jason: I might have done something wrong? No! Begin one with overflow input, shape one, 640. The size 700 would overflow the input which means our video is 640. Then I think this one is also going to overflow, but that will tell us what it is.
Gant: I like this trick.
Jason: You see it happening?
Gant: Do you see it moving? It's scanning down. Do you see it? Perfect. Then you overflow it a little bit at the end.
Jason: We figured it out, everybody. It's 640.
Gant: What are those dimensions in your head?
Jason: You can see it moving down there.
Gant: So now the last part of this is we're going to paint that to your result canvas. Now I know it's a little bit bigger, so could you do me a favor? Just in the interest of time. We'll have that canvas that's overlaying on top of you, I think you might want to move it. Whatever, we'll just do it on top of you. It will be fine.
Jason: I just realized we don't even need this.
Gant: Probably not.
Jason: Yes, we do. It needs to be a start. Start is going to be zero. Then we'll just change ‑‑
Gant: As the start increments?
Gant: It's just telling you where to start.
Jason: We call it like this. And everything is good. And then I want that to be video width instead of width so that things make sense. Make it make sense. That makes me happy. This makes me happy. I am happy.
Gant: Awesome. Let's make sure it runs real quick and then I think that what we could do here is simply draw to that result canvas. That looks like it's scanning down.
Jason: It's scanning.
Gant: Perfect, perfect, perfect. Okay, so now what we're going to do on each iteration, we're going to take that little bar we've cut and paint it into your result canvas. Just that bar. So we can do that with, so the resulting like filter canvas, we'll get access to that, so you want to grab that as well.
Jason: Chat's just throwing shade. Seven steps, huh?
Gant: I don't know if you know this, but we're only on step ‑‑
Gant: This is the curse of, you know, honestly, this is why it's so important. I was talking to a developer in morning, a little side note. And he was telling me that he was having trouble with motivation. And I was really thinking about it as like, all developers can have a problem with motivation. All of the problems that you're solving are super hard, but it's like the people you meet, the conferences you go to, and the attitude that you kind of keep around you that's ridiculously important. So like as a fun tidbit, I love that you're not afraid to ever fail on this show. It's super cool because you like, yeah, everybody wants everybody to succeed on it, but it's good to see a little bit of the struggle as we're only on step four of seven. It's not like this is hard.
Jason: We're on step four, everybody. All right.
Gant: So now, all we have to do is once we've drawn, so where are we drawing that little pixel?
Jason: It's our target element.
Gant: Just await that for a second. Put an await in the beginning of that. Once that's been drawn, all we have to do is draw that to your result canvas and that's as simple as do we have a reference to your filter? Your result canvas in our list of all those fun things?
Jason: No, but I'm now wondering if I need to put these up at the top so we can get to them. I guess it won't matter. That will be fine. We decided that that one is called filter. So we'll call this one filter and that's going to be document queries selector.
Jason: Filter. So now I have the filter down here. Probably going to yell at me for using it without defining it.
Gant: The first thing we need to do, because we're going to use like a little canvas thing, we're going to need to get the 2D context of filter. So we'll need filter CTX to come out of that.
Jason: Should I just make that happen then?
Gant: Yeah. Let's go ahead.
Jason: Like I can do filter canvas then filter. It's going to be filter canvas get 2D. Wait.
Gant: Get ‑‑ what is it? Get context then you pass 2D as a parameter. Yep. That's it. So now you've got the ‑‑
Jason: Filter. Okay.
Gant: So now what we can do is we can draw that image that that little bar that we have there to the result with filter.drawimage. Okay? Then we give it our little canvas that we cut out. What was the name of the thing?
Jason: It's called target element right now. That's what we're passing in is we ‑‑
Gant: Okay. Perfect. Then we can start at left. So now it goes left and right. So left is zero and this is just a zero. Then comma, then we need to give it the height of it. So this will be our start, where we're at. So this is the height. Remember with canvases, it's width then height and with tensors, it's height then width. Just to drive everybody nuts.
Jason: That's the way I want this to go. So I'm assuming it's going to be start X, start Y, end X, end Y.
Gant: As each thing comes in, we're just going to draw that right on top of it.
Jason: And that's it, right? That's what we're doing?
Gant: I don't think you need to do, so you have the width.
Jason: Can I leave this out?
Gant: I think you can leave those out, yeah.
Jason: Yes, that's what I want.
Gant: You can go ahead and draw this. The difference in size will become apparent. That's okay. Did we break it? Good. What did we break?
Jason: What did we break. Only valid on async functions. This is a problem I can solve. Look at it go. Look at it. I'm wiggly. Watch.
Gant: Look at your wiggly head! Yeah!
Jason: Okay. This is really fun. I can already see the chaos that will ensue here. But this is really excited. So then we just need to size it up.
Gant: We can size it up in the tensor pretty easily or we could size it up if there's a way we can do it in drawing.
Jason: How about you tell me what we can do. We're a little short on time.
Gant: Perfect, perfect, perfect. And hopefully we'll have it all set for the end here. So once you've actually grabbed the snap from the camera, let's resize that snap from the camera right there. So that, so when you get that current frame, right, let's create a resized version of that then that way, we're just painting with the resized version. So resize and that equals tf.image. Dot resize by linear. Perfect. Then so the first parameter is going to be the current frame. Then we want to go ahead and we'll set this to, so this is going to be an array here. So the first parameter's going to be the height. Second parameter's going to be the width. We can grab that from your context that you have. So you could say actually, or if you have it there, that will work.
Jason: We're already passing it in here, so we can just grab it.
Gant: Then the third parameter, you'd want to say true because that's fun. Outside the array.
Jason: Oh, got you.
Gant: There you go. Perfect. That should resize it to that size. Do me a favor. At the end of that, do .div255. I can explain this. Then just put the 255 in parenthesis. It doesn't edit it. It just creates a new one.
Jason: Resize by linear is not a function.
Gant: Let me see your function? That is totally a function.
Jason: Oh, resize by linear.
Gant: That's the Python way. You want ‑‑
Jason: Oh, does it need to be, I think I heard one thing and wrote another thing. So bilinear.
Gant: Yes, yes, yes.
Jason: I thought it was like a perfume line. Like Resize by Linear.
Gant: Hold on one second before you go. I know you want to press the button.
Jason: Too late. I pushed the button.
Gant: Wiggly. So see resized? Wherever we were using current frame, use resize. If that's the only place, that should do it. Okay. Let's see.
Jason: So here's the part I'm a little confused by though. We just resized it to its own size, right? Do I need to resize it ‑‑
Gant: So do this. The context is, so do filter.canvas.clientheight there. Yeah. And so that's the client height we're working with. Then on the next one, do filter, canvas and client width. Because we're doing crazy stuff on our page. Do it.
Jason: Would overflow shape 496.
Gant: So it's smaller now. Just change that 640. No, no, that's perfect. You could use those there. Where we hard coded our heights and stuff.
Jason: Yeah, so this is going to be the width.
Jason: And this is going to be the height.
Gant: We're going to totally get there, dude. I've made your write disgusting code.
Jason: I have fears. We're getting closer, but we're still not quite, yeah, overflow. Things are ‑‑
Gant: Didn't we hard code those heights and widths somewhere? The 645 480 is the camera size.
Jason: Let's get what it thinks is true. 496. And then 496. Now let's try it again. And get the other one that fails. And 281.
Gant: Why is it not showing it though?
Jason: Okay. So then, okay. It's doing that, but I wonder if it, my current suspicious is that the video is being like squirshed. So we are pulling the video source, but then we're using the resized, like we're using the canvas and not the source video.
Gant: It should have been resized by that resize bilinear. So we might be sizing it back up by using the wrong width or height and that's what it's going to do there.
Jason: Maybe. Let's set the filter canvas to be like a set size maybe.
Gant: Yeah. Go ahead and set the ‑‑
Jason: So we'll go 500.
Gant: We can just force everything to be 640 by 480 and then it will work.
Jason: That's not a bad idea. Let's do 640 by 480.
Gant: Now everything ‑‑
Jason: We'll make that 640 and this will be 640 as well. And that will be 480. Does that explode?
Gant: I undid all your fun code here.
Jason: This is fine. This might work better. I hope this works better because otherwise we're in trouble. This will be 480. This will be 640 again. Then when we refresh, press the button. It's still way too big.
Gant: Just got bigger.
Jason: So I wonder if maybe we can just force it?
Gant: We missed a spot.
Jason: Something is wrong. We're doing something incorrect here and I don't know what it is.
Gant: It's one of the things we have kind of coded in here.
Jason: Resize from the current frame. What happens if we just use the current frame again now that we've resized everything? Do we need to resize? No, that changes nothing. What is happening? Size 240, 370. Is that math? 640, 320. Can't do math. That does nothing.
Jason: Because it's resized. That fails and that fails because 640 would overflow input shape. Something is weird. I don't know what is weird. I'm wondering, do we need to resize the ‑‑
Gant: If you're making everything the same thing, you don't need to resize. If you're taking what the video content is, there's no need to resize.
Jason: Okay so, let's, current frame.
Gant: See now we're on step 6.88888.
Jason: Right. Yeah, I don't know why.
Gant: Just move your head back and forth while it's happening.
Jason: Which way do I need to go?
Gant: Just keep moving it.
Jason: Okay. So, but I mean, what we've shown here is that this works, and weird, bad code is really what we've landed on.
Gant: We've got a mistake somewhere right there.
Jason: So we need to go figure out what happened with the math. But the guts of what's going on here is not only functioning but working exactly as intended. The problem is that we're getting the sizes wrong, not that the, so like TensorFlow is doing the scan. We can see that it's doing the work. If I was more clever, I could have come up with something cool to do instead of just melting my face. But this is like really, really powerful stuff and you can see like the part that we spent the most time was trying to figure out math. Not trying to figure out TensorFlow. That's a really kind of empowering thing. You know?
Gant: Yeah, the trick here is like the webcam, right? Getting it set up and getting that input and drawing it there. The lines of TensorFlow that we wrote, so if you move that from left to right, top to bottom, it's no problem. It's just basically freezing these things as they go over. I wish I could see, I feel like it's right there where we're probably drawing something back at the wrong size then drawing with that. Because it's right there to draw it right over your face.
Jason: So we're almost there. I think what's going to happen is I can't imagine that you and I are going to not be able to fix this. I pushed what we did, which mean that is this will be live on the internet here momentarily once that builds, which I think honestly it might be. Yeah, it's done. So now, and it won't remember it. That's why I get angry.
Gant: Remember button that does not work.
Jason: Now this is Firefox and I think it's going to do is same thing. I think it's right here. It's bad. It's real bad. But so and actually, this even looks like the sizing was a little bit different.
Gant: It's perfect though.
Jason: That one was fun.
Gant: Is that your new speaker photo there? I think it is. I need to look at your conference.
Jason: But so with that being said, I feel like we'll get that working. If you want to go check this out, it is live on the interwebs. Right now, you can go look at this. You can go look at the code and you can go look, you can go try this, right?
Gant: Bonus if you just point out where we made the mistake. I guarantee we'll find it pretty fast though.
Jason: We'll go figure that out. In the meantime, follow Gant at Gant Laborde and also one more shoutout, thank you again to Aimee, who's been hanging out with us today doing live captioning. She's from White Coat Captioning and that is made possible through our sponsors all of whom are kicking in to make the show more accessible to all people. Gant, if someone wants to learn more, where should they go next?
Gant: Don't live code. Two places. One, check out the book that's coming out. What we're doing here is we're basically abusing all the fun GPU access and stuff because the world of machine learning has got all kinds of cool things. We build some awesome stuff. I built a Harry Potter data set. We do drawings in the book and you'd identify what houses you'd be sorted into. It teaches you how to like draw stuff with dice and all kinds of fun things. That's using the machine learning and AI to do a lot of the hard work for you versus what we're trying to do. Which is math and learned our lesson. Then the #MadeWithTFJS on Twitter will have all kinds of really cool stuff. So if you look for MadeWithTFJS, that's TensorFlow's official hashtag. Silly, goofy stuff to check out.
Jason: Some really, really cool stuff there.
Gant: Show and tells coming up soon. That's the show and tell that Jason leads. All kinds of really cool stuff, and people building amazing things for showing it off.
Jason: All right, so I think that's all we've got time for today. Check the schedule. We've got some really fun stuff coming up. You can add the schedule to your Google calendar. David's coming back. He's going to teach us how to do more with state machines. I've been doing a lot with state machines. I'm really bullish on them. So come back and check them out. We've got more state machines on Kubernetes. Then the representation of sunshine. That's what Prince Wilson is. Cassidy and I are going to do professional trolling, then a whole bunch of people that I don't have on the website, but it's going to be a lot of fun. Make sure you get on the schedule. That being said, I think we're out of time. Gant, as always, thank you so, so much for hanging out with us.
Gant: I love coming on here. It's so much fun to watch. So much fun to be a part of. I can't wait to come back and bring another seven set. Maybe next time we'll just do four steps.
Gant: Bye, everybody.