🔳 CosterGraphics Outline System Advanced
CosterGraphics Outline System Advanced introduces the use of Edge Detection Convolution Kernels for its outline shaders algorithms. These kernels can produce a whole family of different types of outlines by basically applying kernel weights to detected edge pixels. This is a more advanced form of edge detection than using unweighted samples directly, which is done for the Basic ouline shaders.
This page explains how these kernels are used to produce the advanced edge detected outline shaders of the system.
Edge Detection Kernels Library
- Unweighted Kernel
- Cross Kernel
- Sobel XY Kernel
- Prewitt XY Kernel
- Gaussian Blur Kernel
- Laplacian Kernel
- Laplacian of Gaussian (LoG) Kernel
- Difference of Gaussians (DoG) Kernel
- Scharr XY Kernel
- Method A: Normal Difference Magnitude Method (Most common, gives very clean outlines.)
- Method B: Angular Difference (Dot Product) Falloff Method (Gives extremely stable outlines. Very common in stylized NPR renderers.)
- Method C: Difference of Normals (DoN) Method (Gives super crisp outlines. Often the cleanest for view-space normals.)
About the Kernels coordinate system
Because Unity uses a left handed Y up coordinate system for UV coordinates as opposed to the widely used y-down coordinate system of a lot of image processing software. So most of the times when you come across these kernels in the wild they will have the negative y-values at the top row of the 3x3 arrays but for the sake of 'unity' I've chosen to use all the kernels with the positive y-values at the top. This makes it a little bit easier to visualize the relations between the weights in the kernels and the texture sampling offset directions in the Shader Graph shaders.
What kernels are
These 'Edge detection convolution kernels' can be seen as an array containing the weights that the texture samples that we take in eight directions around the center pixel and the one for the center pixel are multiplied/weighted by.
Usually edge detection is performed by sampling neighboring pixels in at least four directions, so up, down, left and right (or diagonally NE, SW, SE and NW), which is cheap but less precise than sampling eight directions. Sampling in four directions is called 4-tap sampling and the directions that we could take the samples in that we could then multiply by the weights of the kernels would be:
```
[0, 1]
[-1, 0][0, 0][1, 0]
[0,-1]
```
But to get better information on diagonal edges at a slightly higher cost we can also sample in eight directions (8-tap):
```
[-1, 1][0, 1][1, 1]
[-1, 0][0, 0][1, 0]
[-1,-1][0,-1][1,-1]
```
For sampling a gray scale or black and white image we can only take samples of one color channel of a texture, usually the R channel and the values returned if the texture uses 8bit 0-255 range values could look something like this:
```
[50][100][100]
[50][100][100]
[50][100][100]
```
Unweighted Kernel (No-Kernel)
```
[ 1][ 1][ 1]
[ 1][ 1][ 1]
[ 1][ 1][ 1]
```
Shown just as an example because this isn't really a useful kernel but if we were to use this then nothing would happen with the weights of the detected edges because they'd all be multiplied by 1.
Cross Kernel
```
[ 0][ 1][ 0]
[ 1][ 0][ 1]
[ 0][ 1][ 0]
```
Description
Sobel XY Kernels
```
[-1][ 0][ 1] [ 1][ 2][ 1]
[-2][ 0][ 2] [ 0][ 0][ 0]
[-1][ 0][ 1] [-1][-2][-1]
```
Description
Sobel X & Y Hybrid Kernel (for directional sharpness). By sampling the texture twice using the separate X and Y kernels as the weights we can compute the x and y gradients of the edges (the partial derivatives Gx and Gy). By then taking the inverse tangent (arc tangent) of the Gy and Gx gradients separately we can calculate the angle of the edge with the sign still preserved, so with the information on the opposing directons included:
float edgeOrientation = atan2(Gy, Gx);
We can also get an unsigned angle (less useful for outlines) by taking the arc tangent of Gx divided by Gy:
float edgeOrientation = atan(Gy/Gx);
(See the Outline3DAdv-Composite-SobelXYKernel-EdgeAngle-Example.shadergraph, which shows the angle of the edge being mapped to the Hue value of the HSV color wheel.)
From calculating the x and y gradients of the detected edges we can also derrive the total strength/magnitude/length of the edge by taking the square root of Gx squared plus Gy squared (this also removes the sign).:
float edgeMagnitude = sqrt((Gx*Gx) / (Gy*Gy));
The edge magnitude is very useful for outlines because the value represents how big an edge is at a position.
The Sobel operators are almost the same as the Prewitt operators below, (they both detect the direction of edges) but the Sobel kernels place more emphasis on the pixels that are close to the center of the kernel mask and less on the diagonals.
Prewitt XY Kernels
```
[-1][ 0][ 1] [ 1][ 1][ 1]
[-1][ 0][ 1] [ 0][ 0][ 0]
[-1][ 0][ 1] [-1][-1][-1]
```
Description
Prewitt operators are very similar to Sobel operators but they are used for detecting horizontal and vertical edges in images, so they respond less to diagonal edges than the Sobel operators.
Gaussian Blur Kernel
```
[1][2][1]
[2][4][4]
[1][2][1]
```
Description
Smooth/blur kernel (for thick, soft outlines) Great for distance estimation to the mask border. The Gaussian kernel can be useful to blur the results of the Sobel and Cross kernel operations with and can be used to generate outlines with by taking the difference of two Gaussians (DoG). Results of the Gaussian operator need to be normalized/divided by 16 since the weights don't cancel each other out like they do with the Cross, Sobel and Prewitt kernels.
Laplacian Kernel
```
[ 0][ 1][ 0]
[ 1][-4][ 1]
[ 0][ 1][ 0]
```
Description
Laplacian (zero-crossing edge detector) Sharp and high constrast.
Laplacian of Gaussian (LoG) Kernel
Description
Difference of Gaussians (DoG) Kernel
Description
Prewitt XY Kernel
Description
Scharr XY Kernels
```
[ -3][ 0][ 3] [ 3][ 10][ 3]
[-10][ 0][ 10] [ 0][ 0][ 0]
[ -3][ 0][ 3] [-3][-10][-3]
```
Description
High precision normal edges. Better directional accuracy and has smoother and less aliasing than Sobel. For normal and depth contours Scharr is probably the best.
Scharr Kernel Methods
Method A : Normal Difference Magnitude Method
Method B : Angular Difference (Dot Product) Falloff Method
The CGEdgeDetection-Normals-ScharrDot sub-graph: -Image-
The CGEdgeDetection-Normals-ScharrDot method in HLSL:
/// <summary>
/// 🟥 METHOD B — Angular Difference (Dot Product) Fall-off Using Scharr Gradient Direction
/// (Gradient Direction = Local Normal Estimate)
/// </summary>
///
/// <remarks>
/// This is the clean, stable NPR method.
///
/// Steps:
/// 1. Use Scharr to compute gx and gy
/// 2. Combine into a local neighbor normal direction
/// 3. Compare angle with center normal using dot product
/// 4. Convert angle to intensity
///
/// Pros:
/// - Super stable
/// - Cleanest outlines
/// - Best NPR look
/// - Great on curved surfaces
/// Produces smooth, technical contour lines.
///
/// </remarks>
float MethodB_ScharrAngularDifference(float2 uv, float2 texelSize)
{
float3 centerN = normalize(_NormalsTex.Sample(sampler_normals, uv).xyz);
float3 gx = 0;
float3 gy = 0;
// Apply Scharr kernels
[unroll]
for (int i = 0; i < 9; i++)
{
float2 nUV = uv + OFFS[i] * texelSize;
float3 n = _NormalsTex.Sample(sampler_normals, nUV).xyz;
gx += n * ScharrX[i];
gy += n * ScharrY[i];
}
// Build a direction vector from gradient
float3 gradDir = float3(length(gx), length(gy), 0);
if (all(gradDir == 0))
return 0;
gradDir = normalize(gradDir);
// Angular difference
float d = dot(centerN, gradDir);
// Convert similarity → edge
float edge = saturate(1.0 - d);
return edge;
}
When using the raw return values of the CGEdgeDetection-Normals-ScharrDot sub-graph on a view space normals texture in a Composite Shader Graph the result will look more like some sort of directional light than outlines. This is because the method only returns the angle of change but not the thickness or an isolated silhouette. To turn it into contour lines the raw values that the Dot method returns have to be properly remapped in the full-screen composite shader first. What the Dot method compares is the normal from the center sample versus the gradient/directional normal that is calculated using the horizontal and vertical Scharr kernels separately. The Dot product dot(centerNormal, gradientDirection) is essentially a basic Lambert lighting term. The raw output of edge = 1 - dot(centerNormal, gradientDirection) that the Dot method returns produces a soft ramp which is dark on flat areas and bright where the normals change toward the local gradient direction, which looks like a directional light. So what the Dot method returns is not a binary, 0 or 1, yes or no outline but it is something that can be shaped to use for outlines.
This shaping can be done linearly using a hard threshold to isolate the edges :
```
_EdgeMin = 0.1–0.25
_EdgeScale = 8–20
edge = saturate((edge - _EdgeMin) * _EdgeScale);
```
This will look like actual outlines.
Or instead of linearly it can be reshaped with a power curve for 'contour band' shaping :
```
_EdgePower = 6–12
edge = saturate(pow(edge, _EdgePower));
```
This gives more painterly, clean, NPR lines, which is the method that is used for the full-screen composite Outline3DAdv-Composite-Contour-ScharrDot.shadergraph shader.