OpenGL ES create stencil effect using texture

Question

Environment

This is the environment I am working in:

• OpenGL ES 2.0
• iPhone Simulator & iPhone 4
• iMac 27" using NVIDIA GeForce GTX 680MX 2048 MB

Hope that helps.

The Problem

Been searching high and low from multiple sources and multiple sites including Stackoverflow but haven't got a working stencil effect going.

What I have is this:

The black 'S' thing is not a polygon but a texture drawn onto a rectangle quad that is the same width and height as the background image.

I am trying to do a stencil effect where the background and the little yellow guy should only be visible if it's within that black 'S' texture.

In my fragment shader, I have this:

varying lowp vec4 destinationColor;

varying lowp vec2 TexCoordOut;
uniform sampler2D Texture;
uniform highp float TexScale;

void main()
{
    highp vec4 color = texture2D(Texture, TexCoordOut);

    if(color.a == 0.0)
    {
        discard;
    }

    gl_FragColor = color;
}

For my setup Depth Stencil Buffer, I have set it up like this:

-(void)setupDepthStencilBuffer
{
    GLint width;
    GLint height;

    glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &width);
    glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &height);

    glGenBuffers(1, &depthStencilBuffer);
    glBindRenderbuffer(GL_RENDERBUFFER, depthStencilBuffer);
    glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8_OES, width, height);
    //glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_COMPONENT24_OES, width, height);

    NSLog(@"depthStencilBuffer = %d", depthStencilBuffer);
}

According to Apple's documentation (which I think is outdated):

http://developer.apple.com/library/ios/#documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/WorkingwithEAGLContexts/WorkingwithEAGLContexts.html#//apple_ref/doc/uid/TP40008793-CH103-SW1

Note some of the things like GL_RGBA enum doesn't exist when I tried to type it in Xcode (I think Apple must have removed it and made it obsolete).

I have also tried Apple's way of defining the so called "depth/stencil" buffer in the above link but it gave the same error below.

The code I have above is how you would create a stencil buffer.

In my setupFrameBuffer() method, I attach it like so:

-(void)setupFrameBuffer
{
    GLuint frameBuffer;

    glGenBuffers(1, &frameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, colorRenderBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, depthStencilBuffer);

    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

    if(status != GL_FRAMEBUFFER_COMPLETE)
    {
        NSLog(@"failed to make complete framebuffer object %x", status);
    }
}

The error I got when I attach it like shown above is:

failed to make complete framebuffer object 8cd6

For my render method, I have this:

-(void)render:(CADisplayLink *)displayLink
{
    glClearColor(0.5f, 0.5f, 0.5f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);



    glClearStencil(0);
    glEnable(GL_STENCIL_TEST);

    // ----------------------------------------
    // Don't write to color or depth buffer
    // ----------------------------------------
    glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
    glDepthMask(GL_FALSE);

    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

    // ----------------------------------------
    // First set the alpha test so that
    // fragments greather than threshold
    // will pass thus will set nonzero
    // bits masked by 1 in stencil
    // ----------------------------------------
    glStencilFunc(GL_ALWAYS, 1, 1);

    // ----------------------------------------------------------------
    // Drawing our stencil
    // ----------------------------------------------------------------

    glBindBuffer(GL_ARRAY_BUFFER, bgVBO);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, bgIBO);

    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, stencilTexture);
    glUniform1i(textureUniform, 0);

    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));
    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_FAN, sizeof(bgIndices)/sizeof(bgIndices[0]), GL_UNSIGNED_BYTE, 0);

    /*
    // -----------------------------------------
    // Second pass of the fragments less
    // or equal than the threshold will pass
    // thus will set zero bits masked by 1
    // in stencil
    // -----------------------------------------
    glStencilFunc(GL_ALWAYS, 0, 1);

    glBindBuffer(GL_ARRAY_BUFFER, bgVBO);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, bgIBO);

    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, stencilTexture);
    glUniform1i(textureUniform, 0);

    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));
    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_STRIP, sizeof(bgIndices)/sizeof(bgIndices[0]), GL_UNSIGNED_BYTE, 0);
    */


    // ---------------------------------------------------
    // RE-ENABLING THE COLOR AND DEPTH MASK AGAIN
    // TO DRAW REST OF THE SCENE AFTER STENCIL
    // ---------------------------------------------------
    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
    glDepthMask(GL_TRUE);




    Mat4 frustrumMatrix = [CameraMatrix createOrthographicMatrixUsingLeft:-(self.bounds.size.width / 2.0)
                                                                       Right:(self.bounds.size.width / 2.0)
                                                                      Bottom:-(self.bounds.size.height / 2.0)
                                                                         Top:(self.bounds.size.height / 2.0)
                                                                        Near:-1.0f
                                                                         Far:1.0f];

    glUniformMatrix4fv(projectionUniform, 1, 0, frustrumMatrix.matrix);

    glViewport(0, 0, self.bounds.size.width * self.contentScaleFactor, self.bounds.size.height * self.contentScaleFactor);

    // ----------------------------------------------------------------
    // Drawing our background first
    // ----------------------------------------------------------------

    glBindBuffer(GL_ARRAY_BUFFER, bgVBO);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, bgIBO);

    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, bgTexture);
    glUniform1i(textureUniform, 0);

    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));
    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_FAN, sizeof(bgIndices)/sizeof(bgIndices[0]), GL_UNSIGNED_BYTE, 0);



    glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);


    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, floorTexture);
    glUniform1i(textureUniform, 0);


    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));

    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_FAN, sizeof(Indices)/sizeof(Indices[0]), GL_UNSIGNED_BYTE, 0);

    glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
    glEnable(GL_BLEND);

    [context presentRenderbuffer:GL_RENDERBUFFER];
}

The result of this is obviously a pink screen that means my setup is wrong:

Anyone able to shed some light please ?

Answer 1

The Solution

Hell yeah!!!

I'm one happy chap now! :D

OK, I finally manage to get Stencil working with texture :)

(also learned a number of things along the way, e.g. we can check color.alpha and use discard as a way to remove transparent pixel and the glBlend(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) trick becomes obsolete)

So the first thing I noticed I did wrong was incorrectly generating the names for my stencil buffer.

In the above source code in my question, I had typed:

glGenBuffers(1, &depthStencilBuffer);

This should really be:

glGenRenderbuffers(1, &depthStencilBuffer);

d'oh!

Second of all, I commented out the important second glStencilFunc() that needed to be called:

glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

// draw black 'S' textured quad here

. . .

// -----------------------------------
// I WAS MISSING THIS IMPORTANT LINE
// -----------------------------------
glStencilFunc(GL_ALWAYS, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

/* draw the rest of the scene here (the background and yellow guy) */

The final result:

Hope that helps everyone else who is trying this cool stencil with a texture feature :D

I also modified a number of other source code but those two were the main changes that got it working.

Some helpful changes helped me debug the problem too:

• I have found that I don't need to attach depth buffer to my framebuffer to get it to work
• I also don't need to clear GL_DEPTH_BUFFER_BIT
• I changed the setupFrameBuffer() method to this with the extra if-else conditions to clearly point to me the possible errors.

The new setupFrameBuffer() method:

-(void)setupFrameBuffer
{
    GLuint frameBuffer;

    glGenBuffers(1, &frameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, colorRenderBuffer);
    //glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, depthStencilBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, depthStencilBuffer);

    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

    if(status == GL_FRAMEBUFFER_COMPLETE)
    {
        NSLog(@"framebuffer complete");
        //NSLog(@"failed to make complete framebuffer object %x", status);
    }
    else if(status == GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT)
    {
        NSLog(@"incomplete framebuffer attachments");
    }
    else if(status == GL_FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT)
    {
        NSLog(@"incomplete missing framebuffer attachments");
    }
    else if(status == GL_FRAMEBUFFER_INCOMPLETE_DIMENSIONS)
    {
        NSLog(@"incomplete framebuffer attachments dimensions");
    }
    else if(status == GL_FRAMEBUFFER_UNSUPPORTED)
    {
        NSLog(@"combination of internal formats used by attachments in thef ramebuffer results in a nonrednerable target");
    }
}