LATENT

SPACE

CONVERGENCE

Combining StreamDiffusion with

pointclouds

First impressions

AI image generation, rendered in realtime through Unity VFX Graph

The results yielded a captivating exploration through latent space, whereby prior generations were used as the input for future frames.

No input environment needed, neural renderer was able to generate one itself.

Reintroducing depth

Techniques to extract depth from image are commonplace in the ai space, but none so pertinent to the issue of realtime than MiDAS, which I was able to call via RenderPass

img2depth

By using both color output from StreamDiffusion and depth from MiDAS, my neural renderer was finally able to construct fully explorable worlds . . .

In conclusion,

Improvements to this system will determine its potential and eventual widespread adoption.

Between introducing temporal consistency through motion LoRAs, performing realtime img2txt to generate recursive prompts for the renderer to adhere to, and implementing developments in NeRF (neural radiance field) and Gaussian splatting technology, the potential for realtime neural rendering is evident.

It is my personal belief that, in the near future, traditional VFX & realtime rendering pipelines alike will offload more and more processes to neural rendering passes.