FORWARD

neural renderer

Leveraging img2img generation in real-time game engines

THAT ‘VIDEO GAME’ LOOK

There are numerous games that look gorgeous while still looking digital, but many of the games lauded for their visuals fall into two categories: photoreal, or stylized.

What makes a game look stylized?

In many cases, it comes down to the countless hours entire teams of artists pour into carefully crafting each texture — at times, even normal maps are hand-painted to achieve a painterly aesthetic in an otherwise 3D game environment:

PRACTICALITY

As a solo developer (who already has to wear a dozen different hats to get anything done,) hand-painting multiple textures for each model is unfortunately out of scope for any project. Due to budgetary constraints, I am lucky to be able to commission a single sculpt at a time.

That’s when I started looking towards other methods…

KUWAHARA

Screen-space shaders can be inserted into the rendering pipeline to great effect. Through these shaders, some formula can be applied to change what the end-users view on their screens. One of these filters is Kuwahara, which creates the appearance of paintbrush strokes to form the image in realtime.

An explanation of how it all works by @Acerola

A screenspace UE4 Kuwahara implementation

IMPLEMENTATION

After some time dabbling with Kuwahara implementation in game engines such as Unreal and Unity, I found the effect to lend an interesting aesthetic, but not precisely the one I was going for; the broad application of the convolution to everything on screen felt, at times, repetitive and uninspired. See for yourself:

But then, while rendering some concept imagery out of Midjourney, I wondered:

could this effect be achieved with AI?

Then I began to spiral: would AI image generation eventually overtake traditional rendering methods, not just in terms of stylization, but in general —

could img2img models achieve real-time speeds within the decade?

Today, rapid advancements in the field have rendered this possibility an imminent reality. Implementations demonstrating real-time capabilities can already be found in super-sampling techniques such as NVIDIA’s DLSS.

To the frenzied excitement of some (and the worried dread of others) the progress in this field is evident week by week.

I believe there is a future for this technology where it is not simply used for short-sighted goals

— cost-cutting by way of automation and the subsequent mass layoffs —

but to revolutionize the methodologies by which we render all things digital.

This page seeks to highlight what I’ve gathered as important breakthroughs in realtime AI image generation, as well as detailing my firsthand experience implementing these technologies in real-time game engines like Unity.

My aim is to see the creation of such a process, by which I’ve been referring to as neural rendering