Mutual TLS using Private Key and Certificate

Have you looked at SecPKCS12Import? I use this for my mutual auth and it works fine. Much less code too.

Hi pnelson, I've seen some info on PKCS12 import using SecPKCS12Import and that could be a reasonable fallback but the requirements I've got at the moment are to allow users to bring a separate certificate and key file as a baseline and add support for PKCS12, PKCS11, etc... in the future.

Are you using native C to set up your connections? Do you have a sample anywhere I could check out?

Thanks for the input!

I’m going to second pnelson’s recommendation here; SecPKCS12Import is much easier.

If you’re being forced to do this the hard way, some hints…

Read the following:

• SecItem: Fundamentals

• SecItem: Pitfalls and Best Practices

The SecItem API has numerous traps for the unwary, and the above should help you avoid them.

Switch this source file from C to Objective-C (or C++ to Objective-C++). That’ll make your code much nicer, allowing you to use dictionary literals (@{ … }) rather manually creating CF dictionaries by hand.

Don’t add a key using kSecValueData. Rather, import the key and then add it using kSecValueRef. That way, if there’s a problem with the key, you’ll detect that at the import stages.

I have a bunch of info about key formats, and how to import them, in:

• On Cryptographic Key Formats

• Importing Cryptographic Keys

Likewise for your certificate. Import it using SecCertificateCreateWithData and then add it using kSecValueRef. The biggest single gotcha here is that folks pass in PEM text rather than DER data.

Using kSecValueRef rather than kSecValueData has another critical benefit: It ensures that the keychain item has the correct attributes. This is critical for identity formation, something I explain in SecItem attributes for keys.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Hi Quinn, thanks for the comments! I am indeed going down the hard path... Don't quite understand WHY it's the hard path and not set up to be easier to use but that's a separate issue. ( I really wish iOS had an implementation of the SecIdentityCreateWithCertificate macOS function )

I read those SecItem links as a starting point for this task and they were incredibly helpful! In large part, they got me to the point I'm at now.

Unfortunately, I cannot switch to Objective C or CPP. The entire library is written in native C and then bound out for use in CPP, Java, JavaScript, and Python. Swift is the next language we are adding support for which is why we're looking into using SecItem and iOS support.

I've generated a SecKeyRef and SecCertificateRef from the CFDataRefs to use kSecValueRef instead of kSecValueData as you suggested for both SecItemAdd certificate and private key. Those both are resulting in OSStatus of 0 in tests.

Definitely ran into the gotcha you mentioned related to PEM text in DER data and I've resolved that earlier with a conversion. Prior to the change to SecItemAdd above, I was already generating a SecCertificateRef to extract the serial from the certificate to use as a unique key during SecItemAdd via SecCertificateCopySerialNumberData. It's a bummer the framework doesn't do this PEM->DER conversion for you.

I'm happy to have set up the SecItemAdd functions in a way that will detect issues at import but so far, it's still resulting in zero kSecClassIdentity items being found in the keychain. I'll continue by reading through your linked docs and report back if I find a solution for anyone else attempting to do things the hard way. I'll add my updated import functions for certificate and key below.

int aws_secitem_add_certificate_to_keychain(
    CFAllocatorRef cf_alloc,
    SecCertificateRef cert_ref,
    CFDataRef serial_data,
    CFStringRef label,
    SecCertificateRef *out_certificate) {

    int result = AWS_OP_ERR;
    OSStatus status;

    CFDictionaryRef delete_query = NULL;
    CFDictionaryRef attributes = NULL;

    /* If the certificate is already in the keychain, delete it before adding. */
    const void *delete_keys[] = {
        kSecClass,
        kSecAttrSerialNumber
    };
    const void *delete_values[] = {
        kSecClassCertificate,
        serial_data
    };
    delete_query = CFDictionaryCreate(
        cf_alloc,
        delete_keys,
        delete_values,
        2,
        &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);
    status = SecItemDelete(delete_query);
    if (status == errSecSuccess) {
        AWS_LOGF_INFO(
            AWS_LS_IO_PKI,
            "static: keychain contains matching certificate that was previously imported.  "
            "Deleting existing certificate in keychain.");
    }

    // Attempt to add the certificate with all set attributes to the keychain.
    const void *add_keys[] = {
        kSecClass,
        kSecAttrLabel,
        kSecAttrSerialNumber,
        kSecValueRef,
        kSecReturnRef };
    const void *add_values[] = {
        kSecClassCertificate,
        label,
        serial_data,
        cert_ref,
        kCFBooleanTrue };
    attributes = CFDictionaryCreate(
        cf_alloc,
        add_keys,
        add_values,
        5,
        &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);
    status = SecItemAdd(attributes, (CFTypeRef *)out_certificate);

    if (status != errSecSuccess) {
        result = aws_raise_error(AWS_ERROR_SYS_CALL_FAILURE);
        goto done;
    }
    result = AWS_OP_SUCCESS;

done:
    if (delete_query) CFRelease(delete_query);
    if (attributes) CFRelease(attributes);

    return result;
}

int aws_secitem_add_private_key_to_keychain(
    CFAllocatorRef cf_alloc,
    SecKeyRef key_ref,
    CFStringRef key_type,
    CFStringRef label,
    CFStringRef application_label,
    SecKeyRef *out_private_key) {

    int result = AWS_OP_ERR;
    OSStatus status;

    CFDictionaryRef delete_query = NULL;
    CFDictionaryRef attributes = NULL;

    /* If the private key is already in the keychain, delete it before adding. */
    const void *delete_keys[] = {
        kSecClass,
        kSecAttrKeyClass,
        kSecAttrKeyType,
        kSecAttrApplicationLabel
    };

    const void *delete_values[] = {
        kSecClassKey,
        kSecAttrKeyClassPrivate,
        key_type,
        application_label
    };
    delete_query = CFDictionaryCreate(
        cf_alloc,
        delete_keys,
        delete_values,
        2,
        &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);
    status = SecItemDelete(delete_query);
    if (status == errSecSuccess) {
        AWS_LOGF_INFO(
            AWS_LS_IO_PKI,
            "static: keychain contains matching private key that was previously imported.  "
            "Deleting existing private key in keychain.");
    }

    // Attempt to add the private key with all set attributes to the keychain.
    const void *add_keys[] = {
        kSecClass,
        kSecAttrKeyClass,
        kSecAttrKeyType,
        kSecAttrApplicationLabel,
        kSecAttrLabel,
        kSecValueRef,
        kSecReturnRef };
    const void *add_values[] = {
        kSecClassKey,
        kSecAttrKeyClassPrivate,
        key_type,
        application_label,
        label,
        key_ref,
        kCFBooleanTrue };
    attributes = CFDictionaryCreate(
        cf_alloc,
        add_keys,
        add_values,
        7,
        &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);
    if (attributes == NULL) {
        result = aws_raise_error(AWS_ERROR_SYS_CALL_FAILURE);
        goto done;
    }
    status = SecItemAdd(attributes, (CFTypeRef *)out_private_key);

    if (status != errSecSuccess) {
        result = aws_raise_error(AWS_ERROR_SYS_CALL_FAILURE);
        goto done;
    }
    result = AWS_OP_SUCCESS;
done:
    if (delete_query) CFRelease(delete_query);
    if (attributes) CFRelease(attributes);

    return result;
}

I've re-read the links provided and it appears as though adding the key/cert is working as expected with the following logs:

After adding the Certificate, CFShow on the returned SecCertificateRef logs:

<cert(0x105f04df0) s: AWS IoT Certificate i: Amazon Web Services O=Amazon.com Inc. L=Seattle ST=Washington C=US>

After adding the Private Key, CFShow on the returned SecKeyRef logs:

<SecKeyRef algorithm id: 1, key type: RSAPrivateKey, version: 4, 2048 bits (block size: 256), addr: 0x600000c21570>

The "Digital Identities Aren't Real" section of pitfalls and best practices mentions that when a certificate or private key are added that pair up, a digital identity is "added". I am running the following search after SecItemAdd of the cert/key and it is resulting in OSStatus -25300 (errSecInteractionNotAllowed).

CFArrayRef stored = NULL;
    const void *keys[] = { kSecClass, kSecReturnRef, kSecMatchLimit };
    const void *values[] = { kSecClassIdentity, kCFBooleanTrue, kSecMatchLimitAll };
    CFDictionaryRef query = CFDictionaryCreate(NULL, keys, values, 3,
        &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);

    status = SecItemCopyMatching(query, (CFTypeRef *)&stored);
    if (status == errSecSuccess) {
        CFIndex count = CFArrayGetCount(stored);
        for (CFIndex i = 0; i < count; i++) {
            SecIdentityRef identity = (SecIdentityRef)CFArrayGetValueAtIndex(stored, i);
            CFShow(identity);
        }
        CFRelease(stored);
    } else {
        printf("No identities found or error: %d\n", (int)status);
    }

This status is returned whether I'm doing a SecItemCopyMatching for a specific identity using the certificate's serial or for any kSecClassIdentity items. This feels weird as I think I'm authorized based on being able to add to the keychain.

I have tried adding the attribute kSecAttrAccessible with kSecAttrAccessibleAlways for both the key and the certificate to make them as openly accessible as possible and am still running into the errSecInteractionNotAllowed status result.

Is there something else I can try or may be doing wrong?

Sorry, I missread the OSStatus. -25300 is errSecItemNotFound. So I guess after adding the cert and key into the keychain, there is no kSecClassIdentity being "added".

Everything looks in order, is there any way to generate an identity manually in iOS the way there was for macOS? Or is there anything else that can be tried at this point?

I think I may have run into part of why an identity isn't being generated. I'm running the following to check whether the public key data in the certificate matches the private key:

SecKeyRef public_key = SecCertificateCopyKey(secitem_certificate);

CFDataRef publicKeyData = SecKeyCopyExternalRepresentation(public_key, &error);

    CFDataRef privateKeyData = SecKeyCopyExternalRepresentation(secitem_private_key, &error);

Boolean match = CFEqual(publicKeyData, privateKeyData);

if (match) { printf("it matches"); }

This fails to produce a match. CFShow on the two also result in pretty different results.

CFShow(publicKeyData):

<CFData 0x600000c78ff0 [0x7ff86002ed20]>{length = 270, capacity = 1024, bytes = 0x3082010a0282010100bf3c1fb172fccf ... dd76fb0203010001}

CFShow(privateKeyData):

<CFData 0x11181b010 [0x60000390c340]>{length = 1190, capacity = 1190, bytes = 0x308204a20201000282010100bf3c1fb1 ... 5bcd75daed351328}

using openssl to check the certificate and key, they are in fact paired properly.

$ openssl x509 -in  cert.pem > cert_pub_key.pem
$ openssl rsa -in priv_key.pem > priv_key_pub_key.pem
$ diff cert_pub_key.pem priv_key_pub_key.pem
 <results in no diff>

I'm stumped as CFShow on the certificate and the privatekey appear to be producing the expected results. Will keep digging...

Additional wrinkle is when I extract the public key from both the certificate and private key

SecKeyRef public_key = SecCertificateCopyKey(*secitem_certificate);

SecKeyRef public_key_from_private = SecKeyCopyPublicKey(*secitem_private_key);

and CFShow on the resulting public keys, the output matches exactly:

public key extracted from certificate
<SecKeyRef algorithm id: 1, key type: RSAPublicKey, version: 4, 2048 bits (block size: 256), exponent: {hex: 10001, decimal: 65537}, modulus: BF3C1FB172FCCF6E1C99D1FEFB4A4CBBD1AADC994923C483AF201B9F33DF256339F951860E77629E9C43114837A0A5F902E59A90521E4EB12B10DAE865942A15B293A4D646F7DC6384AFDA050EFFD0EEC4C8AF35F1A97A0A0935FD514A6634E4857D03EAA38756146F43D4BAB6E78F0DB906809B8C7AB3D454C3FC64F820C3F6C13AAFB910523CBB29C268B38B6124AC97E9A186FACBAA804F2E5692BCAE390056FD4B3C95DA81E1C287D5953BDE77344079B5C751535ED51946C5F5C2146852BA0BC101B984E6BA74AD429F2155C929FBC0FFED004C74EFB44BD5D7852520080C65EB6F71877B5AC95A57B2B0428AA9A14CF06C4FFEAEF69CB02C8634DD76FB, addr: 0x600000297780>
public key extracted from private key
<SecKeyRef algorithm id: 1, key type: RSAPublicKey, version: 4, 2048 bits (block size: 256), exponent: {hex: 10001, decimal: 65537}, modulus: BF3C1FB172FCCF6E1C99D1FEFB4A4CBBD1AADC994923C483AF201B9F33DF256339F951860E77629E9C43114837A0A5F902E59A90521E4EB12B10DAE865942A15B293A4D646F7DC6384AFDA050EFFD0EEC4C8AF35F1A97A0A0935FD514A6634E4857D03EAA38756146F43D4BAB6E78F0DB906809B8C7AB3D454C3FC64F820C3F6C13AAFB910523CBB29C268B38B6124AC97E9A186FACBAA804F2E5692BCAE390056FD4B3C95DA81E1C287D5953BDE77344079B5C751535ED51946C5F5C2146852BA0BC101B984E6BA74AD429F2155C929FBC0FFED004C74EFB44BD5D7852520080C65EB6F71877B5AC95A57B2B0428AA9A14CF06C4FFEAEF69CB02C8634DD76FB, addr: 0x600000c35300>

At a loss as to what's going on here. Maybe I can add the public key into the keychain in a way that will cause the private key and the certificate to match up to form an identity.

Don't quite understand WHY it's the hard path and not set up to be easier to use

Don’t get me wrong, there should be better APIs for this. The Security framework APIs are both very limited and tricky to use correctly. There’s no good reason for that )-: [1]

However, given that situation it’s better to not make your life harder. If, for example, you started with a PKCS#12, you’d have a lot fewer problems. And you’d be writing a lot less code if you use Objective-C or Swift.

Unfortunately, I cannot switch to Objective C or CPP. The entire library is written in native C and then bound out for use in CPP, Java, JavaScript, and Python.

I don’t understand your logic here. Consider:

• These are Apple specific APIs, which means you have to build this code with Apple tools and the Apple SDK.

• Given that, you have ready access to Objective-C:

  1. Move this code into a separate .m file.

  2. Create a .h file that exports a C interface that’s tailored to your specific requirements.

  3. In the .m file, implement those C functions using Objective-C. You don’t even need any classes or objects; the fact that it’s an Objective-C file means that you can use Objective-C shortcuts.

  4. Adjust your build system to build the .m file with Objective-C enabled. You might not even need to do this; most build systems understand .m.

As an example of just how much this buys you, consider these two functions based on one of your earlier code snippets:

extern int clearKeysC(const void * serialData, size_t serialDataCount) {
    CFDataRef serial_data = CFDataCreate(NULL, serialData, (CFIndex) serialDataCount);
    const void *delete_keys[] = {
        kSecClass,
        kSecAttrSerialNumber
    };
    const void *delete_values[] = {
        kSecClassCertificate,
        serial_data
    };
    CFDictionaryRef delete_query = CFDictionaryCreate(
        NULL,
        delete_keys,
        delete_values,
        2,
        &kCFTypeDictionaryKeyCallBacks,
        &kCFTypeDictionaryValueCallBacks);
    BOOL success = SecItemDelete(delete_query) == errSecSuccess;
    CFRelease(serialData);
    return success;
}

extern int clearKeysObjC(const void * serialData, size_t serialDataCount) {
    NSData * serialDataObj = [NSData dataWithBytes:serialData length:serialDataCount];
    return SecItemDelete( (__bridge CFDictionaryRef) @{
        (__bridge NSString *) kSecClass: (__bridge NSString *) kSecClassCertificate,
        (__bridge NSString *) kSecAttrSerialNumber: serialDataObj,
    } ) == errSecSuccess;
}

Life it too short to be wrangling CF dictionaries by hand (-:

Swift is the next language we are adding support for

Cool.

That actually speaks to this point:

I really wish iOS had an implementation of the SecIdentityCreateWithCertificate macOS function

Normally I’d suggest that you file an enhancement request for that but, honestly, I don’t think that’s worth it in this case:

• I think you’re currently stuck on identity formation, and SecIdentityCreateWithCertificate follows the same identity formation rules.

• If we did add a new API for this, it’s most likely be in Swift, so asking for SecIdentityCreateWithCertificate unlikely to get much traction.

Maybe I can add the public key into the keychain in a way that will cause the private key and the certificate to match up to form an identity.

No, that won’t help [2].

The next debugging step is to check that the attributes responsible for identity formation are correct. So, something like:

1. Add the private key and certificate to the keychain.

2. Call SecItemCopyMatching to fetch their attributes (kSecReturnAttributes).

3. Make sure that:

   • The certificate has kSecAttrPublicKeyHash (pkhh) set to its public key hash.

   • The private key has kSecAttrApplicationLabel (klbl) set to the same value.

I talk about this more in SecItem attributes for keys, which is linked to from SecItem: Pitfalls and Best Practices but that’s easy to miss. I just updated the latter to make this more obvious.

Oh, wait! After writing the above I took another look at your code and it seems that you’re overriding kSecAttrApplicationLabel on your key. That’s almost certainly the cause of this failure to form an identity.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] It’s partly for historical reasons, but there are Apple internal factors as well, and that’s not something I can discuss publicly.

[2] And I’ve encountered cases where it makes things worse!

Hey! Thanks for the continued help!

Don’t get me wrong, there should be better APIs for this. The Security framework APIs are both very limited and tricky to use correctly. There’s no good reason for that )-: [1]

As someone who builds SDKs and APIs, I totally get this and especially support for C not being on anyone's list of things to do. I think the work you're doing with documenting a lot of the pitfalls of using SecItem are an incredible resource (and honestly, some of the only I've been able to find) out there and it'd be great if some amount of it could be integrated into the Framework's API documentation.

Life it too short to be wrangling CF dictionaries by hand (-:

Believe me, nobody feels this more than I do. This specific library has to be written in native C and only using native C. If we used Objective C, it'd be as a separate bindings repo the same way we have for CPP. I don't make the rules, I just have to live with them >,<

The next debugging step is to check that the attributes responsible for identity formation are correct. So, something like:

Since last communication, I've checked and compared individual attributes kSecAttrKeySizeInBits and kSecAttrApplicationLabel, from the extracted public keys which matched as well as the private key itself. They all matched up as expected.

Oh, wait! After writing the above I took another look at your code and it seems that you’re overriding kSecAttrApplicationLabel on your key. That’s almost certainly the cause of this failure to form an identity.

Removing the kSecAttrApplicationLabel from the private key with SecItemAdd appears to have resolved the issue (Hurray!). At least to the point that I'm able to find an identity.

As kSecAttrApplicationLabel is listed as a unique attribute for use with queries, I was setting this manually. Looking at the documentation, it appears as though this is normally set to the hash of the public key so I'm assuming it's also used as the point of comparison when generating identities. So even though the public keys extracted from both private key and certificate match, if this doesn't, it's game over.

Thanks again for all your help! Hopefully this is the last block to getting TLS working in native C for our library >,<