Page-Curl Shader -- Pixel transparency check is wrong?

Given

  • I do not understand much at all about how to write shaders
  • I do not understand the math associated with page-curl effects

I am trying to:

  • implement a page-curl shader for use on SwiftUI views.

I've lifted a shader from HIROKI IKEUCHI that I believe they lifted from a non-metal shader resource online, and I'm trying to digest it.

One thing I want to do is to paint the "underside" of the view with a given color and maintain the transparency of rounded corners when they are flipped over.

So, if an underside pixel is "clear" then I want to sample the pixel at that position on the original layer instead of the "curl effect" pixel.

There are two comments in the shader below where I check the alpha, and underside flags, and paint the color red as a debug test.

The shader gives this result:

The outside of those rounded corners is appropriately red and the white border pixels are detected as "not-clear". But the "inner" portion of the border is... mistakingly red?

I don't get it. Any help would be appreciated. I feel tapped out and I don't have any IRL resources I can ask.

//
//  PageCurl.metal
//  ShaderDemo3
//
//  Created by HIROKI IKEUCHI on 2023/10/17.
//

#include <metal_stdlib>
#include <SwiftUI/SwiftUI_Metal.h>
using namespace metal;

#define pi float(3.14159265359)
#define blue half4(0.0, 0.0, 1.0, 1.0)
#define red half4(1.0, 0.0, 0.0, 1.0)
#define radius float(0.4)

// そのピクセルの色を返す
[[ stitchable ]] half4 pageCurl
(
 float2 _position,
 SwiftUI::Layer layer,
 float4 bounds,
 float2 _clickedPoint,
 float2 _mouseCursor
 ) {
    half4 undersideColor = half4(0.5, 0.5, 1.0, 1.0);
    float2 originalPosition = _position;
    // y座標の補正
    float2 position = float2(_position.x, bounds.w - _position.y);
    float2 clickedPoint = float2(_clickedPoint.x, bounds.w - _clickedPoint.y);
    float2 mouseCursor = float2(_mouseCursor.x, bounds.w - _mouseCursor.y);
    
    float aspect = bounds.z / bounds.w;
    
    float2 uv = position * float2(aspect, 1.) / bounds.zw;
    float2 mouse = mouseCursor.xy  * float2(aspect, 1.) / bounds.zw;
    float2 mouseDir = normalize(abs(clickedPoint.xy) - mouseCursor.xy);
    float2 origin = clamp(mouse - mouseDir * mouse.x / mouseDir.x, 0., 1.);
    
    float mouseDist = clamp(length(mouse - origin)
                            + (aspect - (abs(clickedPoint.x) / bounds.z) * aspect) / mouseDir.x, 0., aspect / mouseDir.x);
    
    if (mouseDir.x < 0.)
    {
        mouseDist = distance(mouse, origin);
    }
    
    float proj = dot(uv - origin, mouseDir);
    float dist = proj - mouseDist;
    
    float2 linePoint = uv - dist * mouseDir;
    half4 pixel = layer.sample(position);
    
    if (dist > radius)
    {
        pixel = half4(0.0, 0.0, 0.0, 0.0); // background behind curling layer (note: 0.0 opacity)
        pixel.rgb *= pow(clamp(dist - radius, 0., 1.) * 1.5, .2);
    }
    else if (dist >= 0.0)
    {
        // THIS PORTION HANDLES THE CURL SHADED PORTION OF THE RESULT
        // map to cylinder point
        float theta = asin(dist / radius);
        float2 p2 = linePoint + mouseDir * (pi - theta) * radius;
        float2 p1 = linePoint + mouseDir * theta * radius;
        
        bool underside = (p2.x <= aspect && p2.y <= 1. && p2.x > 0. && p2.y > 0.);
        uv = underside ? p2 : p1;
        uv = float2(uv.x, 1.0 - uv.y); // invert y
        
        pixel = layer.sample(uv * float2(1. / aspect, 1.) * float2(bounds[2], bounds[3])); // ME<----
        
        if (underside && pixel.a == 0.0) { //<---- PIXEL.A IS 0.0 WHYYYYY
            pixel = red;
        }

        // Commented out while debugging alpha issues
//        if (underside && pixel.a == 0.0) {
//            pixel = layer.sample(originalPosition);
//        } else if (underside) {
//            pixel = undersideColor; // underside
//        }

        // Shadow the pixel being returned
        pixel.rgb *= pow(clamp((radius - dist) / radius, 0., 1.), .2);
    }
    else
    {
        // THIS PORTION HANDLES THE NON-CURL-SHADED PORTION OF THE SAMPLING.
        float2 p = linePoint + mouseDir * (abs(dist) + pi * radius);
        
        bool underside = (p.x <= aspect && p.y <= 1. && p.x > 0. && p.y > 0.);
        uv = underside ? p : uv;
        uv = float2(uv.x, 1.0 - uv.y); // invert y

        pixel = layer.sample(uv * float2(1. / aspect, 1.) * float2(bounds[2], bounds[3])); // ME
        
        if (underside && pixel.a == 0.0) { //<---- PIXEL.A IS 0.0 WHYYYYY
            pixel = red;
        }
        
        // Commented out while debugging alpha issues
//        if (underside && pixel.a == 0.0) {
//            // If the new underside pixel is clear, we should sample the original image's pixel.
//            pixel = layer.sample(originalPosition);
//        } else if (underside) {
//            pixel = undersideColor;
//        }
    }
    
    return pixel;
}
Answered by J0hn in 808900022

:facepalm: I needed to call .compositingGroup() before applying the .layerEffect.

I'm gonna go lie down.

Accepted Answer

:facepalm: I needed to call .compositingGroup() before applying the .layerEffect.

I'm gonna go lie down.

Page-Curl Shader -- Pixel transparency check is wrong?
 
 
Q