This image shows the artifacts I am trying to eliminate:

In this thread, it is proposed that the max value of the depths be stored in a third channel in the blur buffer. I did something similar, but a little different, I just did an extra texture read of the unblurred original image to get the exact depth at that pixel. (My “blur” currently just consists of a blit downsample of the texture mipmaps.)

float dropoff = 0.5;
float hiresz = textureLod(shadowmap, texcoord.xy, 0.0f).r;
if (texcoord.z > moments.x + dropoff) return 0.0f;
if (texcoord.z > moments.x) shadow *= 1.0f - (texcoord.z - moments.x) / dropoff;

This eliminates the errors above, but it creates strange sharp edges on other shadows:

I really like how variance shadow maps look aside from this problem. Do you have any suggestions how I can fix these issues?

Exponential shadow maps eliminate this problem, but create another one…

After investigating all of this very thoroughly, I am sorry to say that PCF filter with depth shadow maps and hardware depth bias is really the best thing there is. I tried VSM, ESM, and my own filtering, and every approach has problems.

I really wish there was an optional blend range for texture lookups, over which the distance of each sample would be interpolated linearly before the texture sampling filter was applied.