Thanks to a tip from this guy I got an idea how to do some sort of fake ambient occlusion in a 2D scene. I improvised in Quake 2D from 2012 by manually placing black shadowy textures around background objects, trying to make the scene look less flat, but that was very time consuming.
Gaussian blur with a large enough convolution kernel/matrix makes the object's colors "bleed" into its surroundings, basically scaling it while blurring. So, all you need is that blurry scaled object rendered in black before you render the actual object and... BOOM !
No ambient occlusion
Manually placed textures in Quake 2D (2012)
Ambient occlusion using FBOs and a blur shader
Shader
Here is the GLSL shader code I used:
#version 120
uniform sampler2D img;//texture 1
varying vec2 texcoord;//vertex coordinates
uniform float sw, sh, z;//screen width, height, zoom
uniform float hpass, vpass;//set in application to do vertical or horizontal pass(0 or 1)
//weights
float w[33] = {0.040,0.039,0.038,0.037,0.036,0.035,0.034,0.033,0.032,0.031,0.03, 0.029,0.028,0.027,0.026,0.025,0.024,0.023,0.022,0.021,0.020,0.019,0.018,0.017,0.016,
0.015,0.014,0.013,0.012,0.011,0.01,0.009,0.008};
void main()
{
vec4 sum = vec4(0.0);
//pixel size
float px = 1.0/(sw*z);
float py = 1.0/(sh*z);
sum += texture2D(img, vec2(texcoord.x, texcoord.y)) * w[0];
for(int i=1;i<33;i++)
{
//add values right/down from current position
sum += texture2D(img, vec2(texcoord.x + i*hpass*px , texcoord.y + i*vpass*py)) * w[i];
//add values left/up from current position
sum += texture2D(img, vec2(texcoord.x - i*hpass*px , texcoord.y - i*vpass*py)) * w[i];
}
gl_FragColor = sum;
}
Rendering steps
1.) First render whatever you have in the background.
2.) Then you need an offscreen framebuffer(FBO1) that supports alpha channel and that is the same size as your screen*. Clear it to (0,0,0,0) so that it is fully transparent and render all the objects/textures(a), that will have the outline around them, in black(b). The FBO needs to be transparent so that when you render the final result over your scene, you can see the stuff you rendered before.
a) b)
c) d)
3.) Render FBO1 to another transparent frame buffer(FBO2) like the first one using the vertical(or horizontal) blur shader.
4.) Then take FBO2 and render it over your final scene using the horizontal(or vertical) blur shader(c).
(If you are using a single pass blur you could render it directly in 3 and skip this step.)
5.) Then render all the objects you rendered to FBO1 in step 2 to your final scene as they are. Those will cover up unnecessary middle parts so you get the outline effect aka ambient occlusion(d).
BOOM !
*You could speed up this technique by using a frame buffer that is smaller then your scene and using a smaller kernel. Then you need to render the final result scaled up. More info in the first linked tutorial (Working in lower resolution).