🔳 Coster-Graphics Outline System Advanced
Edge Detection Kernels Library
- Cross Kernel
- Sobel XY Kernel
- Prewitt XY Kernel
- Gaussian Blur Kernel
- Laplacian Kernel
- Laplacian of Gaussian (LoG) Kernel
- Difference of Gaussians (DoG) Kernel
- Scharr XY Kernel
- Method A: Normal Difference Magnitude Method (Most common, gives very clean outlines.)
- Method B: Angular Difference (Dot Product) Falloff Method (Gives extremely stable outlines. Very common in stylized NPR renderers.)
- Method C: Difference of Normals (DoN) Method (Gives super crisp outlines. Often the cleanest for view-space normals.)
Kernels coordinate system
Because Unity uses a left handed Y up coordinate system for UV coordinates as opposed to the widely used y-down coordinate system of a lot of image processing software so most of the times when you come across these kernels in the wild they will have the negative y-values at the top row of the 3x3 arrays but for the sake of 'unity' I've chosen to use all the kernels with the positive y-values at the top. This makes it a little bit easier to visualize the relations between the weights in the kernels and the texture sampling offset directions in the Shader Graph shaders.
Cross Kernel
```
[ 0][ 1][ 0]
[ 1][ 0][ 1]
[ 0][ 1][ 0]
```
Description
Sobel XY Kernels
```
[-1][ 0][ 1] [ 1][ 2][ 1]
[-2][ 0][ 2] [ 0][ 0][ 0]
[-1][ 0][ 1] [-1][-2][-1]
```
Description
Sobel Hybrid Kernel (for directional sharpness). By sampling the texture twice using the separate X and Y kernels as the weights we can compute the direction of the edge and by combining them we can get the magnitude (intensity) of the edges.
The Sobel operators are almost the same as the Prewitt operators, (they both detect the direction of edges) but the Sobel kernels place more emphasis on the pixels that are close to the center of the kernel mask and less on the diagonals.
Prewitt XY Kernels
```
[-1][ 0][ 1] [ 1][ 1][ 1]
[-1][ 0][ 1] [ 0][ 0][ 0]
[-1][ 0][ 1] [-1][-1][-1]
```
Description
Prewitt operators are very similar to Sobel operators but they are used for detecting horizontal and vertical edges in images, so they respond less to diagonal edges than the Sobel operators.
Gaussian Blur Kernel
```
[1][2][1]
[2][4][4]
[1][2][1]
```
Description
Smooth/blur kernel (for thick, soft outlines) Great for distance estimation to the mask border. The Gaussian kernel can be useful to blur the results of the Sobel and Cross kernel operations with and can be used to generate outlines with by taking the difference of two Gaussians (DoG). Results of the Gaussian operator need to be normalized/divided by 16 since the weights don't cancel each other out like they do with the Cross, Sobel and Prewitt kernels.
Laplacian Kernel
```
[ 0][ 1][ 0]
[ 1][-4][ 1]
[ 0][ 1][ 0]
```
Description
Laplacian (zero-crossing edge detector) Sharp and high constrast.
Laplacian of Gaussian (LoG) Kernel
Description
Difference of Gaussians (DoG) Kernel
Description
Prewitt XY Kernel
Description
Scharr XY Kernels
```
[ -3][ 0][ 3] [ 3][ 10][ 3]
[-10][ 0][ 10] [ 0][ 0][ 0]
[ -3][ 0][ 3] [-3][-10][-3]
```
Description
High precision normal edges. Better directional accuracy and has smoother and less aliasing than Sobel. For normal and depth contours Scharr is probably the best.
Scharr Kernel Methods
Method A : Normal Difference Magnitude Method
Method B : Angular Difference (Dot Product) Falloff Method
The CGEdgeDetection-Normals-ScharrDot sub-graph: -Image-
The CGEdgeDetection-Normals-ScharrDot method in HLSL:
/// <summary>
/// 🟥 METHOD B — Angular Difference (Dot Product) Fall-off Using Scharr Gradient Direction
/// (Gradient Direction = Local Normal Estimate)
/// </summary>
///
/// <remarks>
/// This is the clean, stable NPR method.
///
/// Steps:
/// 1. Use Scharr to compute gx and gy
/// 2. Combine into a local neighbor normal direction
/// 3. Compare angle with center normal using dot product
/// 4. Convert angle to intensity
///
/// Pros:
/// - Super stable
/// - Cleanest outlines
/// - Best NPR look
/// - Great on curved surfaces
/// Produces smooth, technical contour lines.
///
/// </remarks>
float MethodB_ScharrAngularDifference(float2 uv, float2 texelSize)
{
float3 centerN = normalize(_NormalsTex.Sample(sampler_normals, uv).xyz);
float3 gx = 0;
float3 gy = 0;
// Apply Scharr kernels
[unroll]
for (int i = 0; i < 9; i++)
{
float2 nUV = uv + OFFS[i] * texelSize;
float3 n = _NormalsTex.Sample(sampler_normals, nUV).xyz;
gx += n * ScharrX[i];
gy += n * ScharrY[i];
}
// Build a direction vector from gradient
float3 gradDir = float3(length(gx), length(gy), 0);
if (all(gradDir == 0))
return 0;
gradDir = normalize(gradDir);
// Angular difference
float d = dot(centerN, gradDir);
// Convert similarity → edge
float edge = saturate(1.0 - d);
return edge;
}
When using the raw return values of the CGEdgeDetection-Normals-ScharrDot sub-graph on a view space normals texture in a Composite Shader Graph the result will look more like some sort of directional light than outlines. This is because the method only returns the angle of change but not the thickness or an isolated silhouette. To turn it into contour lines the raw values that the Dot method returns have to be properly remapped in the full-screen composite shader first. What the Dot method compares is the normal from the center sample versus the gradient/directional normal that is calculated using the horizontal and vertical Scharr kernels separately. The Dot product dot(centerNormal, gradientDirection) is essentially a basic Lambert lighting term. The raw output of edge = 1 - dot(centerNormal, gradientDirection) that the Dot method returns produces a soft ramp which is dark on flat areas and bright where the normals change toward the local gradient direction, which looks like a directional light. So what the Dot method returns is not a binary, 0 or 1, yes or no outline but it is something that can be shaped to use for outlines.
This shaping can be done linearly using a hard threshold to isolate the edges :
```
_EdgeMin = 0.1–0.25
_EdgeScale = 8–20
edge = saturate((edge - _EdgeMin) * _EdgeScale);
```
This will look like actual outlines.
Or instead of linearly it can be reshaped with a power curve for 'contour band' shaping :
```
_EdgePower = 6–12
edge = saturate(pow(edge, _EdgePower));
```
This gives more painterly, clean, NPR lines, which is the method that is used for the full-screen composite Outline3DAdv-Composite-Contour-ScharrDot.shadergraph shader.