A fundamentally AI-powered game

Intro

Recently, I had a discussion with some friends about what makes games different, and how to make a unique game. We talked about things like gameplay mechanics, graphics, even controllers.

If you couldn't tell from my last post, lately I've been intrigued by AI, and its potential applications in video games. Which brought me to the question: What if there was a game whose feedback and story were generated entirely by an AI?

Is it possible?

Can it be fun?

---

More definitions

AI is an emerging topic in many communities, particularly in the field of research. As a starting point, it could be good to get some ideas from research papers and see if I can adapt some of what I find to the world of game development. But before we go, I should clarify some of the terms that might come up.

Disclaimer: In an effort to not upset certain people, and since I keep on throwing the term around, I ought to clarify what I mean by AI. Often times, terms like "machine learning" and "neural networks" are used interchangeably with "AI." In fact, those are really just stepping stones towards the goal of achieving what most people think of when they hear "AI", or a true artificial intelligence.

Or maybe people think of Haley Joel Osment

There are a few definitions of AI. Paraphrasing from a few uncredited sources (classy):

AI is broad, and can be described simply as a computer program doing intelligent things. In an effort to make the computer program gain intelligence, we employ algorithms, or AI methods known as Machine Learning. A specific class of machine learning algorithms are Neural Networks, which operate like (and are somewhat inspired by) neurons in the human brain.

You might imagine that if we're doing something that sounds insanely complex like modeling neuron activity, it might take a hell of a lot of computational power. And it totally does. Fortunately, computational power only gets cheaper as time goes on, and we're at a point where even your desktop can experiment with neural networks. It goes beyond that, though. Chances are, you've even interacted with neural networks in some way through the smartphone in your pocket. Ever used Snapchat face filters or Augmented Reality world lenses? The company has published a paper on mobile based neural networks for image recognition.

Remarkable

Similarly, if you've ever used an app that makes your photo look like a particular artist's painting, then you've experienced neural networks in action. This type of neural network algorithm is called "Style Transfer." The network is trained to take an input of a photo, and run it through the algorithm to output a photo as if a famous artist had painted it. The results can be very impressive.

Much art. So wow.

To process the photo, your phone sends it off to the company's servers that then do the heavy lifting and quickly (a few seconds later) transform your image into art.

What if this style transfer technique could be implemented into a video game? Could each frame of the game be passed through the algorithm fast enough to make an insanely trippy, artistic playable game?

The short answer: Holy shit yes.

The long answer:

---

Real-time Neural Network Style Transfer in a Video Game (or, holy shit this game is trippin' balls)

For this topic, I think I'll opt for a summary of my experiences versus getting into the nitty-gritty of code as I have done in previous blog posts. I worked on this over a span of three nights, from about 9pm to 2am on average. Let's break it down day by day.

Day 1: Researching Research

I scoured the internet, going from one Github page to the next, looking for anything that mentions "fast" and "neural network style transfer." After all, if this effect is going to work in a game, we need to pull off visual performance of at least 30 frames per second. I came across this Github page that contains Python code to do "fast" style transfer using neural networks.

A couple hours after installing every requisite Python plugin, we were up and running. Using their example picture and a style of my choosing, a stylized image was generated. It took about five seconds to generate the frame, though. Good enough for now, let's get this thing working with Unity!

I whip up a simple game of "beach ball roll and hit wall simulator."

I wrote some code to save a screenshot in-game, and wrote some horribly dirty code in a batch file that repeatedly executes the Python script that accesses the recently-saved screenshot to apply the style transfer. Cut me a little slack, I just want a proof of concept that this could remotely work. And it does! It pulled off a whopping one frame every few seconds. Needs to be faster! But it's 2am, so I called it quits.

Day 2: Proper Comms

Looking into the code, there was the opportunity to run the neural network off of the graphics card using NVIDIA CUDA, which is a much faster way of processing the data than the default of running on the CPU. An hour later, after installing the wrong drivers, uninstalling them, and installing correct ones, we're GPU bound. Another test, and we got a solid 1 FPS...before crashing after an indeterminate amount of time due to Python accessing a file that's still being written by Unity.

Right, so that whole "Ok unity, save a screenshot" followed by "Ok Python, open the screenshot" was pretty awful, but it gave me a lot of motivation to make it better. So, I set up a line of TCP communication between Unity and Python. There *shouldn't* be any latency or lost packets since everything is running through the same machine.

After a "hello world" is sent from Unity and opened in Python, I got to writing code in C# that takes the current Unity world view, stores it in a 2D texture, encodes it as a .png file, and sends the data in bytes over to Python using Serial commands. A couple hours and many Python image libraries later, I am able to open the image sent by Unity! Progress!

Once a quick rewrite for continuous comms had been typed up, I hit the big play button, and, would you look at that, two frames per second! It's a victory at 2am again, so, until tomorrow.

Day 3: Speeding things up

We've integrated the GPU, sent data over TCP, but I still was not getting the speed I wanted. I began with debugging the code at runtime, and found that every time I called on Python to run the style transfer algorithm, it would start a "session" for the GPU to fire up and do the calculations to generate the art. I had a feeling that there was a better way to do that, so after a couple of hours of research and code, I was able to rearrange some lines so that the same GPU session would stay active for each frame.

Hitting the play button, we see that we just got a HUGE bump - up to 10 frames per second.

At this point, I start profiling the code. I put timers before and after individual lines of code on the Unity side, and nothing there is contributing to that 0.1 seconds of delay. Monitoring the GPU usage through a Windows utility shows that we haven't hit close to what the GPU is capable of. After profiling the Python code, I see one line in particular that is the culprit. I'm using an array function from the Numpy library to convert the bytes from Unity into a format that the CV2 library can open in Python. Googling "numpy array slow python", it turns out that it's a known issue with the function. Go figure.

It takes me another hour and a half to work around this, but I end up streaming the bytes in the buffer directly to CV2 through an alternate Numpy function. We're officially hitting 30fps. Here's the result:

And here's the source image for the inherited style:

In short, as I mentioned earlier: Holy shit it works.