Florent's Tech Blog

Tuesday, January 26, 2021

Real-time style transfer in Unity using deep neural networks

This time at Unity its with the awesome folks from Grenoble ML team that we did a blog post and this is about the Real-time on the fly style transfer.
Check it out : https://blogs.unity3d.com/2020/11/25/real-time-style-transfer-in-unity-using-deep-neural-networks/

Monday, May 20, 2019

Unity GPU lightmapper: A technical deep dive.

The awesome folks at the Lighting team at Unity and I have been working on a blog post about the GPU lightmapper.
Check it out : https://blogs.unity3d.com/2019/05/20/gpu-lightmapper-a-technical-deep-dive/

Wednesday, May 20, 2015

Aliasing: Part 2

Geometry aliasing is well known and is a complex (and terrible) monster that can take different forms, let's see them:

The staircase form

We have seen that texture filtering and mipmapping can "avoid" aliasing by blurring the input signal. Unfortunately, with geometry, we don't have such a luxury (at least not directly at the hardware level). Here's a simple example :

Let's rasterize a triangle :

Then shade it in red :

Here it is: The staircase monster !

To solve it we usually blur along the edges as a post-process. Here are two very common algorithms about it :

FXAA (not very expensive, but blur too much)

SMAA (work better but more expensive)

The diffuse lighting form

So we fought well and defeated the staircase form without too much damage to our scene, but aliasing is still there lurking in the dark. It waits and then pops as a fully lit pixel just to hide in the dark again in the next frame !

This form of aliasing is more tricky, it needs the camera to be in motion to be clearly visible. Here is an example :

* Let's say that we want to simulate breathing of the main character with the camera. The player stay still. The view remains "almost" the same. However it is translated a little bit back and forth every frame from A to B.

* Let's now imagine that there is a bezel in the scene, this bezel is defined with geometry only.

* Finally let's apply a simple diffuse lighting assuming the light is above the bezel (local normal dot up vector in our case)

When camera create the A projection, the pixel on the bezel is purely red.
When camera create the B projection, the pixel on the bezel is almost black !

By going back and forth, the camera is thus creating temporal aliasing.
Thus, even something as simple as diffuse lighting is exposed to aliasing :(

Will vertex interpolation save the day?

One can say that I have oversimplified the problem, actually yes :P. Most of the time vertices will not be duplicated and vertex interpolation will happens between faces, thus making pixel shading smoother :

blue lines = vertices and associated normals
yellow lines = interpolated normals
green = blue + yellow lines

Yeah vertex interpolation save the day !
Humm actually not, it pretty much kills the lighting... not really a solution then :(

Let's try to add more vertices !

blue lines = vertices and associated normals
yellow lines = interpolated normals
green = blue + yellow lines

Nope, still not good. Temporal aliasing is still lurking :(

Actually having aliasing by using more tessellated geometry is one of the inherent flaw of the rendering pipeline. The sampling rate is too low compared to the frequency of the information carried by the geometry, and thus we cannot recreate the initial signal properly.

A possible help here could be to feed current shading with the one(s) from the previous frame(s), this is called temporal antialiasing. And this can be merged with geometry antialiasing (FXAA/SMAA/...) as well.

TXAA from NVidia is such a technique

A word about Specular aliasing

Specular aliasing can be awful and thus tend to be more known than the diffuse one. Fortunately it usually relies on a gloss map that can be pre-filtered in correlation with the normal rate of change (ie variance).

In other words specular can be made less glossy when normal changes a lot in the neighborhood of a shaded pixel. This is very effective but it's still an approximation.

Here is an excellent article by Stephen Hills about it :
Specular-showdown

However, in the real world we often approximate this.
The theory requires the normal variance to be known offline and the gloss map to be linked to it. However in reality normal maps are often in tangent space and the maps will be used on many different meshes with different geometry. Furthermore details maps variance should be taken into account if used.

Friday, May 8, 2015

Aliasing

Hi,

I'm starting a serie of posts about the different forms of aliasing. Geometry aliasing is a well known subject, however aliasing is much more than that.

Aliasing takes its evilness in one of the most fundamental notion of realtime rendering : rasterization. Let's see why.

Rasterization

When rendering an image we do it pixel by pixel, usually doing the shading only once for every pixel, and using the center of the pixel as the point to render to:

The trouble is that pixels are not points at all. Would they be infinitely small that would be a perfectly fine approach. However even in 1080p, pixels are quite big. But quite big against what ?

* Aliasing is certainly not worst on larger TV.

* On the other hand aliasing is worst if we reduce the resolution.

Let's view the scene as a collection of varying information that we can sample at any pixel position :

* Z position

* albedo

* normal

* etc

And let's now target one of these values (albedo for example) and see it how it is sampled :

Aliasing from texture sampling

Let's say our shading will simply set the pixel's value to the color from the texture :

Seems great !

But what happens if the red channel changes faster ?

Something strange is happening here ! A low frequency pattern is appearing on our pixels while the source was actually very high frequency ! Bad bad !!!!

In other words : When sampling a signal, if the sampling frequency is too low (called undersampling), we can't reconstruct it properly..

Here is another example (the well known moiré pattern) :

High enough sampling

Undersampled

In our case the sampling frequency is driven by our resolution, and the signal frequency is driven by the input data -> geometry, texture, texel ratio and sampling method (from point to anisotropic sampling).

That seems a complex problem, however the Nyquist-Shannon sampling theorem is here to help us:

In short it says that we can properly reconstruct a signal if the sampling frequency is at least twice the sampled signal maximum frequency (for proper intensity reconstruction target more like 4).

Full details here : Wikipedia Nyquist-Shannon sampling theorem

To simplify, let's agree that :

* triangle will always be far bigger than a pixel

* texel ratio (texel/pixel) will neither exceed one

Then our remaining worst case scenario is a "screenspace oriented" triangle with a texel ratio of one.

If we go back to our red shading case the maximum frequency in the texture is thus less than 2 pixels. Meaning any pattern less or equal to 2 pixels can cause problem.

In real life we will probably have cases with texel ratio > 1, triangle subpixel, and/or a complex shading.

However all is not lost : texture filtering will help a lot as mipmapping will soften the input signal. Finaly this form of aliasing is mostly visible on repeating patterns (for example tiled details maps).

Additional food for thought (Thanks Jean-Michel!): To be sure to reconstruct the input signal properly (without losing it's intensity) one should sample at 3 to 4 times the input signal frequency. Furthermore using nearest sampling can have the effect of undersampling even if a lot of pixels are shaded. The texture being already a discrete information itself.

Geometry undersampling

For the sake of sanity we have agreed that :

* triangle will always be far bigger than a pixel

* texel ratio will never exceed one

Unfortunatly this is not always the case.

We have seen that texture texel ratio and content (frequency) can create aliasing. However this is not limited to texture. Every input signal can create aliasing if its undersampled.

Geometry is a very good candidate, Especially now that triangle are becoming smaller and smaller with the new generation ! That's gonna be the subject of next post :)

Wednesday, April 15, 2015

Links collection

Hi.

This post is a collection of links collected overtime for reference, its mostly about rendering. Hopefully it can be useful to someone else :).

PS : More link will be added in the future.

Tools

Equation writter and doc (use $$ to start an equation)

Function plotter

Function plotter
Curve fitting
Bias and gain

3D scan mesh cleaning

GPU benchmark
CPU benchmark
Renderdoc (DX11 debugger)
GRemedy (Opengl debugger)
Indexed file search
Disk space visualizer
Text editors color schemes
Gaussian kernel calculator
Mercurial timelapse view
A programming language for Visualisation
Chart/Diagram web app
Regular expression nice tool

Realtime ray tracing

RTX on battlefield V (GDC2019)

RTX in metro exodus (GDC2019)

Debugging and profiling dxr & vulkan ray tracing (GDC2019)

Spatiotemporal Variance Guided Filtering

RTX Ray traced water caustics (GDC2019)

GPU memory perf test

Optimization

NVidia compute shader optim
Cuda bank conflict optim
Low level gpu doc
AMD GCN performance hints
Shader optimization
Shader optimization2 (GCN/DX11)
Optimal grid rendering
Depth Upsampling
Math lib for GCN: ShaderFastLib
GPU Scalarization

Buffers/Textures

ZBuffer precision
Unit vectors encoding
Normals encoding
Get rid of tangent and binormal
BCN formats explained
Normal mal blending
Tightening the Precision of Perspective Rendering
Depth-precision-visualized

Tiled/clustered lighting

Light indexed deferred rendering
Tiled shading
Clustered shading
Practical Clustered Shading
Practical Clustered 2013
Nvidia tiled demo (see download links)
Tiled based rendering
Tiled based rendering & Deferred
Improved culling for tiled and clustered rendering

Shadow

Dual paraboloid
Dual paraboloid 2
Nvidia PCSS
Shadow acne/filtering
Shadow acne 2 (normal offset)
Shadow technics
Shadow technics2
AMD ShadowFX
VSM
RealtimeShadows the book
Soft shadow from spherical harmonics
A good collection of links
Adaptive depth bias
Playing with realtime shadow Crytec
Beyond MSM
Improved MSM (soft shadows and scattering)
Caustics
A scalable real-time many-light shadowing system

Global illumination

Voxel cone tracing
Light Probe Interpolation Using Tetrahedral
GI on the cloud
FC3 GI
GI on GPU
Lightmap compression on metal
Packing lightmap
Spherical harmonic lighting
Various link on ray tracing
Correlated Multi-Jittered sampling
Overview of sampling techniques

Antialiasing

Overview of antialiasing
Siggraph course
MLAA
SMAA
MSAA
MSAA 2
Temporal antialiasing
TAA jitter pattern
Spec antialias : LEAN mapping
Spec antialias : mipmapping normal maps
Spec antialiasing methods
Text antialiasing
Checkboard rendering explained
Dashed line antialiasing

Cubemap/Images Effects

Parallax corrected cubemap
IBL and Parallax correct cubemap
Envmap lighting
Being more wrong parallax corrected
SSLR implementation from GPU Pro 5
Removing banding (p122)
SSRR interleaved
PostFX in bounds
Intel ASSAO
Auto exposure
SSLR in kill zone 2

Physically Based Rendering

Minute Physics: Linear color space
Maths of PBR
Understanding the shading shadow function
Specular brdf reference
UE4 PBR
Bioshock PBR
Unity PBR
Moving frostbyte to PBR
Specular occlusion hack
Specular occlusion hack v2 p77 listing 26
History of the RGB color model
Horizon based specular occlusion
Separable Subsurface scattering (skin)
Fast subsurface scattering approx by Pixar
Good tutorial
GGX for half/medium
PBR Encyclopedia
Allegorithmic PBR guide vol. 1
Google filament renderer

Painterly & non realistic rendering

Gloss Perception in Painterly and Cartoon Rendering
Painterly Rendering with Curved Brush Strokes of Multiple Sizes
Customizing Painterly Rendering Styles Using Stroke Processes
Paint by relaxation
Abstract Painting with Interactive Control of Perceptual Entropy
Gpu image flow
Stable dithering in return of the Obra Dinn

Fluid Simulation

Intel fluid simulation
Intel fluid simulation 2
Fluidic code repository
Fluid simulation for dummies

Multithreading

Lockfree
Data oriented design

GPU Driven pipeline

Rainbow6 Siege
Ubi GPU Pipe
Ubi GPU Pipe 2

Tesselation/Compute

A survey of tesselation technics
GPU Voxelisation
Summed area table
Parallel reduction
Parallel reduction 2
GPU Sorting
Compute shader pipe explained 1
Compute shader pipe explained 2
Compute shader pipe explained 3

Realtime ray tracing

NVidia intro to real raytracing
Effectively integrating rtx ray tracing for realtime
Texture Level-of-Detail Strategies for Real-Time Ray Tracing

Color spaces

Color interpolation in various spaces
GLSL Shader code to convert color from various space
Color for Graphics Programmers

Volume & scattering

Sky rendering
Atmospheric scattering
Isosurface volume rendering
Volumetric clouds

Misc

Dynamic 2d character lighting

Debugging optimized code in VS studio 2012 and up
Rendering of insight
Coding pirates material with Unity
Skin shading
Parallax occlusion mapping
White Noise in GLSL
High quality capture of eyes
Eye rendering
A tour of the graphics pipeline
Double VS tripple buffering android
How to bake normal maps for unity

Thursday, April 9, 2015

A little history : from Phong to BRDF

In this post i will start from the well known Phong reflection model and explain why we are now (mostly) moving to physically based rendering.

Phong reflection model

This shading model was proposed by Bui Tuong Phong, who published it in his 1973 Ph.D at the University of Utah.

Where :

Ambient = AmbiantColor * Albedo

Diffuse = clamp(N.L, 0, 1) * Albedo

Spec = clamp(V.R,0,1)^shininess

It's a very very efficient model and yet a good approximation. There is genius behind it !

Blinn-Phong

Few years latter in 1977 we get Blinn-Phong.

It's really close to the Phong model but for the specular :

Where:

Spec = clamp(N.H,0,1)^shininess

It is more accurate at steep angle & faster. Neat !

Furthermore it was implemented in hardware up to DX 9c.

This model was thus very efficient and heavily used by the time :)

Diffuse VS Specular

So Phong and Blinn-Phong are both doing a clear separation between Ambient, Diffuse and Specular. Let's explain that a little :

Ambient : is the light that bounce around so much that you loss any directional information.

Diffuse : is the light that come from the light source and get reflected in all direction.

Specular : is the light that come from the light source and get reflected "more" in the direction of the reflection vector.

From "perfect" specular to diffuse.

That separation may seems a little bold ? However it's not a bad approximation to be better convinced let's take a look at John Hable work:

How To Split Specular And Diffuse In Real Images

And

Everything is Shiny

Impressive isn't it ?

Morality:

Specular is much more present than we think
Specular might be colored
Blinn-Phong is a lie (ok it was more than expected)

Light is uber-complex

So obviously light is much more complex :) Here are some example :

Scattering

When light travel inside a solid and lit it "from the inside".

Can be seen on wax, human skin, clouds in the sky, ice etc etc !

Caustics

When light is reflected/refracted via a curved surface, creating cool "drawing" of concentrated light.

Can be seen on glass construct, ice, basically anything curved and semi-transparent.

Bounce

When light bounce around its color is altered (some wavelength get absorbed) based on the materials it bounce on. Here is the ultra-famous cornel-box were the effect can be seen.

Take a look at the right face of the right box for example.

And there are a lot more ! Especially if you consider the complexity of human perception/eyes on top of the complexity of light.

So :

Do we want scattering ? Yes sometime (usually faked) !

Do we want caustics ? No. It's way too expensive!

Do we want bounces ? Yes (but that's another story)

Do we want more cool stuff ? Depend on cost.

Improving over Blinn-Phong specular model

So we cannot even hope to simulate light correctly at runtime, but we still want something cool ! At this point we need to choose were we want to spend our computation power.

Caustics and Scattering are a far fetch, post effect are of subject so for the purpose of this blog post we will answers that we want a good reflection model first.

So what is a good reflection model ?

It's one that work for you !

If you want to go cartoon shading why not ! If you want to go realistic it's good too ! However in a lot of case you will probably want to have a good specular reflection. As we have seen it's very important to define the shape and details of your objects.

PS : Actually the raise of more complex BRDF is linked to the evolution of the hardware too : GPU are more and more and more powerfull, while memory is not following at the same speed. In other words the ratio of available math/per parameters have raised.

Fresnel

The first step to improve the specular is to take fresnel into account. What is that ? Please follow the link ! Another awesome post from John Hable !

Everything has fresnel

By now you should be convinced about the need to add some fresnel to our lighting.

First option : Add some factor/power of N.V, for example:

Shading += (1-(N.V)^FresnelHack) * Color

This have a few advantages :

Better than nothing it tweaked correctly
May be used to hack your BRDF for funky material (velvet for example)

This have a few draw back:

Specular will need to be tweaked down to compensate
It scrap your lighting as crazing angle become emissive
This is actually not fresnel at all even if sometime it may be called so (see below)

Second option : use Fresnel-Schlick approximation

Fresnel = F0 + (1- F0) * (1-V.H)^5

were F0 is the reflectance of the material when looking perpendicular at it.

Specular *= Fresnel

This have a few advantages :

F0 is a well known value from the real world and can be measured !
Will work no matter what the lighting is (no emissive stuff).

This have a few draw back:

You may end up adding a hacky N.V term anyway because of "velvet like" materials...

Second option is thus much more interresting, furthermore it's energy conservative ...

Energy conservation ?

When light bounce on a material it can be either absorbed or reflected. Right ?

Thus for a given point on a surface the amount of reflected light should be inferior or equal to the amount of received light. Right ?

To go further in that direction let's first introduce the "BRDF"

BRDF function

For “Bidirectional Reflectance Distribution Function”

It's function taking the incoming and outgoing vectors (each described by two angles) that allow to compute the lighting on a pixel.

Its the “reflected light VS incoming light function”

For an energy conservative BRDF the sum of all reflected light on the hemisphere is thus inferior or equal to the incoming light (L).

Is Blinn-Phong energy conservative ? The short answers is no

Diffuse and Specular are additive and we get no garanty that there sum is below L.
Diffuse and Specular are not even energy conservative individually !

Can we transform Blinn-Phong to be Energy conservative ? yes !

For Diffuse it's actually a matter dividing the albedo by Pi ! Easy enough !
For Specular what we gonna do is to normalize it. It other word no matter the specular power it will always return the amount of light defined by the specular color ! Usefull !
The final step is to ensure albedo and specular color dont sum to more than one !

I promised not to get into the math, but if you want them you can read this very good post by Rory Driscoll : Energy conservation in games

Some pictures:

Top row is a Blinn-Phong specular (and diffuse)
Bottom row is a normalized Blinn-Phong specular (and diffuse)

Energy conservative Specular and Diffuse
where (albedo + specular color <=1)

Ok but what about the fresnel tweak we have done before ?

Its fine as it can only reduce the amount of specular ! ouf :)

Physically Based Rendering

Let’s recap important concepts :

Diffuse

Its the light that go to the object and bounce in any direction

Specular

Its the light that go to the object and bounce "more" toward the reflection vector

Fresnel

It adjust the specular to take crazing angle into account.

Energy conservation

It ensure a material does not emit more light than it receive.

BRDF (bidirectional)

Its the actual function that is responsible for all of that.

PBR is all about finding a consistent (ie math/physically based) way of simulating light !

Micro-facet theory

The basis of physically based rendering is to see the surface as a lot of little face, called micro-facets.

These micro-facets are so many and so tiny that we will not defined them one by one using textures. Instead we will define them using two probability functions called D and G.

For sure these probability function will themselves being define using parameters fetched from textures.

Here is the general equation : F(Fresnel) * G * D / (4 * N.L * N.V )

D is the distribution term or "How the microfacet normal are distributed around a given direction” it can be seen as mirrorness.

Distribution term

G is the geometry/visibility term (can be name both) or “How much the microfacet is blocked by other microfacets” it can be seen as the roughness.

There are many BRDF build around the microfacet theory ! For example :

GGX, Ashikmin-Shirley, Beckman, Cook-Torrance, Blinn-Phong...

PBR Blinn-Phong

What ? Blinn-Phong ? Actually yes ! If we take the building blocks we have seen above our Blinn-Phong is really not far from being physically based and can be made to match the microfacet theory !

Let's say that :

D = (Pi+2)/2Pi * (N.H)^specularPower

G = N.L*N.V

F = fresnel term as seen before

Then you have a PBR Blinn-Phong ! There are different way to write it, here is one :

Diffuse = clamp(N.L, 0, 1) * Albedo / Pi

Specular = (Pi+2)/2Pi * (N.H)^specularPower * F * specularColor

With (Albedo+SpecularColor < 1)

Finally here is a little webgl shader to play with, feel free to experiment !

ShaderToy PBR Blinn-Phong demo

PS : You will need a welgb enabled browser (chrome for example).

PS2: When you can latest driver are a must.

PS3 : Be sure to take a look at all the awesome shaders on shadertoy !!! :)

Tuesday, March 31, 2015

Intro

Hi,

I recently had the pleasure to do a talk about physically based rendering to a wonderful group of video games students. (Hello DDJV 10th cohorte !) It make me realize something :

* Rendering knowledge evolve all the time.
* It usually evolves on top of already complex stuff.
* Some of it get deprecated over time.
* But more is added than deprecated over time.

Finally the firsts steps into realtime rendering can be harsh. There is a lot of knowledge to absorb upfront!

For sure there is a lot of excellent documentation on the internet, however it is mostly written for industry professional or for people with a good background in rendering.

I'm thus creating this blog and targeting it at students/newcomers in the wonderful world of realtime rendering. I will post about current techniques while trying to simplify the approach as much as possible.

See you soon for first post !

Tuesday, January 26, 2021

Real-time style transfer in Unity using deep neural networks

Monday, May 20, 2019

Unity GPU lightmapper: A technical deep dive.

Wednesday, May 20, 2015

Aliasing: Part 2

The staircase form

The diffuse lighting form

Will vertex interpolation save the day?

A word about Specular aliasing

Friday, May 8, 2015

Aliasing

Rasterization

Aliasing from texture sampling

Geometry undersampling

Wednesday, April 15, 2015

Links collection

Tools

Blogs

Paper collections

Books

Deep Learning / Machine Learning

Deep Learning optimisation

Shader libraries

Maths

APIs

Realtime ray tracing

RTX on battlefield V (GDC2019) RTX in metro exodus (GDC2019) Debugging and profiling dxr & vulkan ray tracing (GDC2019) Spatiotemporal Variance Guided Filtering RTX Ray traced water caustics (GDC2019) GPU memory perf test

Optimization

Buffers/Textures

Tiled/clustered lighting

Shadow

Global illumination

Antialiasing

Cubemap/Images Effects

Physically Based Rendering

Painterly & non realistic rendering

Fluid Simulation

Multithreading

GPU Driven pipeline

Tesselation/Compute

Realtime ray tracing

Color spaces

Volume & scattering

Misc

Thursday, April 9, 2015

A little history : from Phong to BRDF

Phong reflection model

Blinn-Phong

Diffuse VS Specular

Light is uber-complex

Scattering

Caustics

Bounce

Improving over Blinn-Phong specular model

Fresnel

First option : Add some factor/power of N.V, for example:

Second option : use Fresnel-Schlick approximation

Energy conservation ?

Physically Based Rendering

Micro-facet theory

PBR Blinn-Phong

Tuesday, March 31, 2015

Intro

RTX on battlefield V (GDC2019)

RTX in metro exodus (GDC2019)

Debugging and profiling dxr & vulkan ray tracing (GDC2019)

Spatiotemporal Variance Guided Filtering

RTX Ray traced water caustics (GDC2019)

GPU memory perf test