Killing Fence Shimmer on Quest 3: A Custom Alpha Sharpening Function for Thin Geometry

There is a class of artifact in Quest 3 development that no off-the-shelf anti-aliasing setting will solve. You build a fence, you put it on a cage prefab, you run the scene, and the fence shimmers. Not just at the edges. The wire itself crawls in screen space as your head moves. At distance it disintegrates into noise. At very long distance it disappears entirely. None of the URP toggles fix it. Increasing MSAA from 2x to 4x helps marginally. Turning on Foveated Rendering makes the peripheral version worse. You sit there in the headset thinking: this is supposed to be a chain-link fence, and it looks like static.

This is the aliasing problem on thin geometry, and it is one of the genuinely hard rendering problems in mobile VR. The display panel resolution on Quest 3 is high, but a 3mm wire at 10 meters subtends about 0.6 pixels per eye. That is subpixel. Constant head tracking moves the camera every frame, so the wire's coverage shifts between samples, and you get crawling pixels. Standard MSAA cannot help because MSAA only anti-aliases geometric edges, not alpha-clipped edges. Standard mipmapping cannot help because thin features get averaged into uniform grey at higher mips and Preserve Coverage only rescales the total alpha, not the spatial layout of the surviving texels.

The fix that actually works on a tile-based mobile GPU is a single custom Shader Graph node that does three things in one function: fwidth-based alpha sharpening for Alpha-to-Coverage, distance-based cutoff reduction using mip-level estimation, and 4-tap rotated-grid supersampling of the alpha channel.

This post walks through why each of those three pieces exists and how they compose into one function that fits inside Quest 3's fragment shader budget.

Why not just use alpha clip

The default Unity approach to a fence texture is alpha clip. You author a texture where the wire is opaque and the gaps are transparent. The shader does if (alpha < threshold) discard; and the rasteriser drops the fragment. This works on PC. It is the wrong choice on Quest 3 for two reasons.

The first reason is that discard and clip() defeat the hardware's tile-based rendering optimisations. Adreno 740 uses Hidden Surface Removal in its binning stage. The HSR logic depends on knowing each fragment's depth contribution before the fragment shader runs. When a shader uses discard, the GPU does not know whether the fragment will write depth or not until after the shader executes, so it has to defer HSR. On a complex scene with overdraw this is a measurable cost. Meta's own developer documentation specifically calls this out as one of the top mobile VR performance traps.

The second reason is that alpha clip gives you a binary opacity decision per pixel. The wire either fully covers the pixel or fully misses it. There is no sub-pixel coverage. Combined with the small angular size of fence wires at distance, this is exactly the recipe for crawling pixels.

The right primitive on Quest 3 is Alpha-to-Coverage. A2C is a hardware feature: instead of clipping based on alpha, the fragment shader's alpha output is converted into a per-sample coverage mask. With 4x MSAA, each pixel has four sample positions, and an alpha of 0.5 will cover roughly two of them. This gives you sub-pixel opacity transitions while staying in the opaque queue with depth writes intact. It coexists cleanly with HSR because every fragment that survives the coverage test writes depth at full strength. Meta's benchmarks put 4x MSAA on Quest 3 at roughly 0.5 to 1.5 ms per frame, and A2C on top of MSAA is essentially free.

The problem is that A2C with a raw alpha channel and 4 samples per pixel gives you only 4 discrete coverage levels: 0, 1, 2, 3, or 4 samples covered. That is not enough to smoothly resolve a fence wire's edge. If your alpha texture has a sharp transition (which it will, because authoring tools snap alpha to 0 or 1 by default), A2C does not have anything to interpolate. You get the same shimmer as alpha clip, just with a slightly softer edge.

You need to widen the alpha transition zone. That is the first piece of the function.

fwidth sharpening

fwidth(x) in HLSL returns abs(ddx(x)) + abs(ddy(x)): the sum of the absolute screen-space partial derivatives. For an alpha value, it is the rate at which alpha is changing across one pixel in screen space. Near the edge of a wire, where alpha transitions from 1 to 0 over a small UV distance, fwidth(alpha) is large. In the middle of a wire or in the middle of a gap, where alpha is uniform, fwidth(alpha) is near zero.

If you rescale the alpha around the cutoff threshold using fwidth as the rescaling factor, you get a transition that always spans one pixel in screen space regardless of how sharp the underlying texture is:

Out = (alpha - cutoff) / max(fwidth(alpha), 0.0001) + 0.5;
Out = saturate(Out);

The mechanics: when alpha is well above cutoff and changing slowly, (alpha - cutoff) is large positive, fwidth(alpha) is small, the ratio is huge, and saturate clamps to 1. When alpha is well below cutoff and changing slowly, the ratio is huge negative and saturate clamps to 0. Only in the narrow band where (alpha - cutoff) is on the same order as fwidth(alpha) does the function produce intermediate values, and that band is always exactly one pixel wide. The + 0.5 recentres the curve so cutoff maps to 0.5 output (which is the threshold A2C uses internally for 2-of-4 sample coverage).

The max(fwidth(alpha), 0.0001) is a guard against division by zero on perfectly uniform regions, which would otherwise produce NaN in the corners of a tile.

This single transformation is responsible for most of the visual improvement on a chain-link fence at medium distance. The alpha transition zone is now wide enough that A2C's four coverage levels are actually being exercised, and the result is a smooth edge rather than a binary clip.

It is not enough on its own.

The mipmap problem

fwidth sharpening works as long as the texture has alpha to sharpen. The problem is what happens as the camera gets further away. Trilinear filtering pulls from higher mip levels at distance. A higher mip is the texture averaged into half the resolution, then averaged again, then averaged again. On a fence texture where the wires are 1 to 2 texels thick, after two or three mip levels the wires have been averaged away into a uniform grey. There is nothing to sharpen.

Unity's Preserve Coverage feature helps, sort of. It rescales each mip's alpha values so the total fraction of pixels passing the clip threshold stays constant across mips. The total amount of fence visible is preserved. But which pixels carry the surviving alpha is essentially noise: the spatial pattern of the fence wires is gone, replaced with a stippled approximation. At medium distance this looks acceptable. At long distance it looks like static.

The trick that actually works is to lower the clip threshold as the camera gets further away. The mip is greyer, so the texture has fewer pixels above the original threshold. Lowering the threshold lets more of those grey pixels through. The fence stays visible (faintly, washed out, but visible) instead of dissolving into nothing.

The clean way to know which mip is being sampled is to compute it from the UV derivatives. ddx(UV) and ddy(UV), multiplied by the texture's resolution, give you the size of one pixel in texel space. Take the log base 2 of the larger dimension, and you have the mip level the hardware would select if it were doing trilinear filtering from scratch:

float2 uvDeriv = fwidth(UV) * TexSize;
float mipLevel = max(log2(max(uvDeriv.x, uvDeriv.y)), 0.0);
float adjustedCutoff = Cutoff * saturate(1.0 - mipLevel * FadeRate);

FadeRate is a tunable parameter. A value of around 0.1 means each additional mip level reduces the effective cutoff by 10%. By mip 5 the cutoff is half its original value. By mip 10 it is essentially zero, which means almost everything passes, which means the fence becomes a translucent haze rather than vanishing entirely.

Why a FadeRate knob instead of a fixed curve? Because the right value depends on the texture. A fence texture authored with 2-texel-thick wires fades into noise faster than one with 4-texel-thick wires, and the artist needs to tune the curve for their content. Exposing it as a material parameter lets the Shader Graph user dial it in per-material without recompiling.

Why supersample the alpha

The fwidth approach plus mip-based cutoff reduction gets you most of the way. The remaining failure case is at oblique viewing angles, where a fence panel stretches into the distance and one UV axis is heavily compressed while the other is not. The texture's anisotropic filter handles this well for colour, but the alpha channel still only gets one sample per pixel, and that sample can land in the wrong place: between two wires, on the edge of a wire, in a gap. With anisotropic filtering raised to 4x or 8x (which is essentially free on Adreno 740), the colour channel is well-behaved at oblique angles but the alpha is still under-sampled.

The fix is to supersample the alpha channel. Take four samples at sub-pixel offsets in screen space, average them, and feed the result into the fwidth sharpening. The four samples are placed in a Rotated Grid pattern: not aligned to the horizontal or vertical pixel axes, because thin geometry tends to be roughly horizontal or vertical and an axis-aligned sample grid would over-sample one direction and under-sample the other. The rotated grid catches wires at any orientation.

The offsets in pixel-space fractions are (+1/8, +3/8), (+3/8, -1/8), (-1/8, -3/8), (-3/8, +1/8). To translate those into UV offsets we use the same ddx(UV) and ddy(UV) we already computed for the mip estimation. Each sample reads only the alpha channel of the texture, so this is four texture fetches per fragment but only one channel of each fetch contributes to subsequent computation.

float a0 = Tex.tex.Sample(Tex.samplerstate, UV + dx * 0.125 + dy * 0.375).a;
float a1 = Tex.tex.Sample(Tex.samplerstate, UV + dx * 0.375 - dy * 0.125).a;
float a2 = Tex.tex.Sample(Tex.samplerstate, UV - dx * 0.125 - dy * 0.375).a;
float a3 = Tex.tex.Sample(Tex.samplerstate, UV - dx * 0.375 + dy * 0.125).a;
float alpha = (a0 + a1 + a2 + a3) * 0.25;

The cost is real but bounded. Four texture fetches plus the ALU for the averaging is approximately the cost of one extra full texture sample. On a fence material that covers 5 to 10 percent of screen area in a typical scene, this is well within budget.

The full function

All three pieces compose into one Shader Graph Custom Function Node, File mode, function name AlphaSharpen:

void AlphaSharpen_float(UnityTexture2D Tex, float2 UV, float Cutoff, float2 TexSize, float FadeRate, out float Out)
{
    float2 dx = ddx(UV);
    float2 dy = ddy(UV);
 
    float a0 = Tex.tex.Sample(Tex.samplerstate, UV + dx * 0.125 + dy * 0.375).a;
    float a1 = Tex.tex.Sample(Tex.samplerstate, UV + dx * 0.375 - dy * 0.125).a;
    float a2 = Tex.tex.Sample(Tex.samplerstate, UV - dx * 0.125 - dy * 0.375).a;
    float a3 = Tex.tex.Sample(Tex.samplerstate, UV - dx * 0.375 + dy * 0.125).a;
    float alpha = (a0 + a1 + a2 + a3) * 0.25;
 
    float2 uvDeriv = fwidth(UV) * TexSize;
    float mipLevel = max(log2(max(uvDeriv.x, uvDeriv.y)), 0.0);
    float adjustedCutoff = Cutoff * saturate(1.0 - mipLevel * FadeRate);
 
    Out = (alpha - adjustedCutoff) / max(fwidth(alpha), 0.0001) + 0.5;
    Out = saturate(Out);
}

There is also a _half variant for shaders running in half precision. The implementation is identical, with all types switched to half.

The inputs to wire up in Shader Graph are:

• Tex: the same UnityTexture2D reference you use for BaseColor. The function calls it directly, so you do not feed it an alpha value
• UV: the post-tiling, post-offset UV that you also feed into the BaseColor sample. The derivatives must reflect the actual sampling coordinates
• Cutoff: the base clip threshold, typically 0.5
• TexSize: from a Texture Size node connected to the same texture, providing the float2 dimensions
• FadeRate: a material property, starting value around 0.1

The output goes into the Alpha block of the fragment stage. The material must have AlphaToMask enabled (which our project's existing RC_SimpleLit_LUT and RC_Unlit_Lightmapped_LUT shaders already support as a property, defaulting off). The render queue stays opaque.

Texture authoring rules

The function does the heavy lifting in the shader, but it cannot rescue a badly-authored texture. The companion settings on the import side that matter:

• Aniso Level: at least 4, ideally 8. Quest 3's GPU handles anisotropic filtering cheaply, and fence textures are almost always viewed at oblique angles. Leaving Aniso Level at 1 is the single biggest miss in default Unity texture imports for this kind of content
• Mipmap Filtering: Kaiser gives sharper mip levels than the default Box filter. The difference is small but real for thin features
• Preserve Coverage: on, with the alpha threshold set to match the shader's Cutoff (0.5 in our case). This is what keeps the total alpha consistent across mips. Without it, the mip cutoff adjustment in the shader has too much work to do
• Trilinear filtering: on, so blends across mip levels are smooth and the shader's mip estimation maps to actual hardware sampling behaviour

These settings are not optional. The shader assumes them. If Preserve Coverage is off, the cutoff adjustment over-corrects. If Aniso is at 1, the oblique-angle case is worse than the supersampling alone can compensate for. If mipmaps are off entirely, the mip estimation produces garbage and the cutoff snaps to its base value at all distances.

LOD strategy

The shader handles the medium-distance and oblique-angle cases. It does not handle the very-far case where the fence subtends less than one pixel. Nothing in screen space can render below one pixel coherently. The right answer is to switch to a different representation at that distance.

The LOD chain we landed on for fences and cages:

• LOD0, screen height above 15 percent: full 3D mesh with individually modelled wires, opaque shader, MSAA-only AA
• LOD1, 8 to 15 percent: simplified mesh where adjacent wires are merged into thicker strips, A2C plus the AlphaSharpen function
• LOD2, 3 to 8 percent: billboard quad with a baked fence texture, A2C plus AlphaSharpen with anisotropic filtering doing most of the angle work
• LOD3, 1 to 3 percent: simplified card with a uniform translucent appearance, transitioning out via dithered fade
• Below 1 percent: not rendered

The transitions need dithered cross-fade rather than alpha-blended cross-fade. Dithered fade keeps the material in the opaque queue and preserves depth writes. Alpha-blended cross-fade requires either rendering both LODs in the transparent queue (which fights depth) or a more complex sort. Dithered is the right primitive here.

What this costs in profiling

A RenderDoc capture of a scene with two fence panels (one near, one mid-distance) using the full AlphaSharpen function shows:

• The fence draw calls themselves remain in the opaque queue. Depth writes happen. Hidden Surface Removal still works for everything behind the fence
• Each fence fragment performs four texture fetches for the alpha supersample, plus the standard BaseColor fetch. Total approximately 5 fetches per fragment instead of 1
• Fragment ALU is dominated by the texture sample cost. The fwidth, log2, and saturate operations are negligible against the cost of memory access
• Total measured impact on a scene with around 8% screen coverage of fence material was approximately 0.15 ms additional fragment time at 90 Hz

That is the cost on Quest 3. On a more sparsely-fenced scene the cost scales linearly with coverage. On a heavily-fenced scene you would want to dial back the supersample or accept the cost as the price of the visual.

What this does not solve

The function does not handle the case where the fence material is also using Meta's depth occlusion subgraph (SG_MetaOcclusion). The occlusion subgraph outputs its own alpha contribution that gets combined with the texture alpha before reaching the AlphaSharpen function. The fwidth sharpening still works on the combined alpha, but the supersample only reads from the texture, so the occlusion alpha is single-sampled. In practice the occlusion boundary is a relatively smooth depth comparison that already AA's reasonably, but if you needed both edges to be A2C-clean you would need a second supersample of the occlusion alpha as well. We have not gone that far. The occlusion-fence interaction in our scenes looks fine without it.

The function also does not address shimmer caused by specular highlights on the fence's BaseColor. Specular shimmer is a separate problem that requires either specular anti-aliasing in the lighting model (LEAN, CLEAN, or Toksvig mapping) or roughness biasing at distance. We use a lookup-table-based simple lit shader (RC_SimpleLit_LUT) which is reasonably forgiving on this front because the specular response is precomputed and low-frequency, but it is not zero.

Where this leaves the LOD ticket

This shader is one part of a larger LOD system for fences and cages. The other parts (the prefab with LOD groups, the dithered cross-fade implementation, the RenderDoc captures across all LOD tiers) are downstream of the shader being right. With the shader stable, the rest is plumbing.

The Blender side of the original ticket (template fence and cage models with consistent LOD chains) is deferred. That is its own piece of work and outside the rendering scope.

The thing I take away from this is that thin-geometry aliasing on mobile VR is solved by being aggressive about Alpha-to-Coverage as the primitive, then layering the right shader transformations on top. fwidth sharpening alone is well-known. Mip-aware cutoff reduction is less common but easy. Supersampling just the alpha channel is the move that closes the gap on oblique angles. Put all three in one Custom Function node and the fence stops shimmering on the headset, which is the entire point.