This is how it all works:
- light is a textured quad/polygon
- Box2D is used to get all the sprites inside the light polygon by either using light as a sensor, or using b2World::QueryAABB()
- for convex shapes visible edges are calculated using a dotproduct of light and edge normals
- for concave shapes(which can only be chains in Box2D) all edges are considered visible
- shadow polygons are created by extending visible edges
- using polygon clipping library called Clipper a "difference" operation is performed between the light polygon and shadows to get the final light polygon
- UV mapping is performed on the resulting vertices
- the resulting polygon is rendered as a GL_TRIANGLE_FAN
No raycasting is performed, which you usually find in tutorials.
The "depth" mask uses the green channel to mark which pixels should be illuminated, and for the intensity (ex. transparent smoke should be illuminated less than a solid surface). Blue channel is used for depth, and works basically the same. It's not perfect, but I can get some nice effects with it.
Soft shadows are generated by rendering the lights on a FBO smaller than the screen size and rendering it with a blur shader.