Support for custom Matter endpoints, clusters and attributes

I am working on an app for a home automation device.

If I were using HomeKit exclusively I could add custom services or custom characteristics on standard services and these things would all be reported to my app via HomeKit. There is sample code from Apple that demonstrates how to do this.

When a Matter device is commissioned using HomeKit you might expect custom clusters and/or custom attributes in a standard cluster would be translated to appropriate HomeKit services and characteristics, but this doesn't appear to be the case.

Is there a way to have HomeKit do this?

If not it seems I would need to use Matter directly rather than via HomeKit to access custom features. But if I commission the device using Matter in my app then I understand a new fabric is created and the device would not show in the Home app. Maybe the user needs to commission the device twice, once with my custom app and once with the Home app? That seems like a poor user experience to me. Perhaps that is the price paid for using a cross-platform standard?

Is there a better way to get the same level of customization using Matter that I am able to get using HomeKit?

Answered by DTS Engineer in 791790022

When a Matter device is commissioned using HomeKit you might expect custom clusters and/or custom attributes in a standard cluster would be translated to appropriate HomeKit services and characteristics, but this doesn't appear to be the case.

Is there a way to have HomeKit do this?

Yes, though the APIs involved are not entirely obvious.

The general process is that you can create a MTRDeviceController using specific parameters you get from HomeKit. That controller will then give you access to the relevant matter accessory through the HomeKit ecosystem, instead of needing to create your own ecosystem (which is how Matter.framework is typically used).

Note: Matter.framework has changed considerably over a very short period, so it's very possible that some of the specific details described may shift or change over time. Please explore the API and look for alternate paths before assuming the general approach here "doesn't work".

Getting into details, you'll create the MTRDeviceController controller using sharedController(withID:xpcConnect:). That method requires a controllerID and an xpcConnectBlock, both of which come from methods in HMHome.

Once you have your MTRDeviceController, you can then create an MTRBaseDevice using init(nodeID:controller:). The nodeID for that method come from the matterNodeID property of HMAccessory.

Two things to note about this process:

  1. The MTRDeviceController you create here is NOT the same object you'd end up with if you created your own object for your own ecosystem. By design, large parts of it simply do not work- for example, you cannot commission an accessory "through" that object. Note that MTRBaseDevice will also not allow you to open a commissioning window. The role of this API path is SPECIFICALLY to provide access to HomeKit accessories through HomeKit's ecosystem, NOT facilitate broader ecosystem configuration.

  2. The access you have through the MTRBaseDevice device is also somewhat limited. The issue here is that while the MTRBaseDevice itself should work fine, the MTRBaseDevice itself is not fully "compatible" with the all of the other classes in the matter framework, which contrains how you can use the object. Making that concrete, direct interaction with the accessory using the read/write/subscribe method of MTRDevice should work fine. However, if you create one of the higher level Matter device objects (for example, something like MTRClusterDoorLock) you'll find that the new object does't really work.

Note that while the usage restrictions on MTRBaseDevice usage (#2) are primarily a side effect of it's current implementation state (so they could change and improve over time), the commissioning limitations (#1) are an intentional design choice to prevent API abuse (and are unlikely to change).

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Accepted Answer

When a Matter device is commissioned using HomeKit you might expect custom clusters and/or custom attributes in a standard cluster would be translated to appropriate HomeKit services and characteristics, but this doesn't appear to be the case.

Is there a way to have HomeKit do this?

Yes, though the APIs involved are not entirely obvious.

The general process is that you can create a MTRDeviceController using specific parameters you get from HomeKit. That controller will then give you access to the relevant matter accessory through the HomeKit ecosystem, instead of needing to create your own ecosystem (which is how Matter.framework is typically used).

Note: Matter.framework has changed considerably over a very short period, so it's very possible that some of the specific details described may shift or change over time. Please explore the API and look for alternate paths before assuming the general approach here "doesn't work".

Getting into details, you'll create the MTRDeviceController controller using sharedController(withID:xpcConnect:). That method requires a controllerID and an xpcConnectBlock, both of which come from methods in HMHome.

Once you have your MTRDeviceController, you can then create an MTRBaseDevice using init(nodeID:controller:). The nodeID for that method come from the matterNodeID property of HMAccessory.

Two things to note about this process:

  1. The MTRDeviceController you create here is NOT the same object you'd end up with if you created your own object for your own ecosystem. By design, large parts of it simply do not work- for example, you cannot commission an accessory "through" that object. Note that MTRBaseDevice will also not allow you to open a commissioning window. The role of this API path is SPECIFICALLY to provide access to HomeKit accessories through HomeKit's ecosystem, NOT facilitate broader ecosystem configuration.

  2. The access you have through the MTRBaseDevice device is also somewhat limited. The issue here is that while the MTRBaseDevice itself should work fine, the MTRBaseDevice itself is not fully "compatible" with the all of the other classes in the matter framework, which contrains how you can use the object. Making that concrete, direct interaction with the accessory using the read/write/subscribe method of MTRDevice should work fine. However, if you create one of the higher level Matter device objects (for example, something like MTRClusterDoorLock) you'll find that the new object does't really work.

Note that while the usage restrictions on MTRBaseDevice usage (#2) are primarily a side effect of it's current implementation state (so they could change and improve over time), the commissioning limitations (#1) are an intentional design choice to prevent API abuse (and are unlikely to change).

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

This is exactly what I wanted but wasn't hopeful I would get it. Thanks so much, I'll start implementing this!

The response from Kevin has been quite helpful - I have been able to read data from a Matter attribute I obtained using the process he described. However, I have not been successful with subscribe methods. The specific subscribe method Kevin linked returns an HMError #48 to my errorHandler closure when I call it. The localized description of that error is "The operation couldn’t be completed.", which certainly tells me something went wrong but provides no actionable remedies. The name of code 48 in HomeKit.HMError is "operationNotSupported" which is similar and yet somehow more discouraging. In the spirit of Kevin's suggestion that I "explore the API and look for alternate paths before assuming the general approach here 'doesn't work'." I tried the other 3 subscribe methods that haven't been deprecated. Two of them give me the result above via my reportHandler. The remaining one, subscribeToEvents, never calls my reportHandler closure (where I would get an error or a report) and also never calls my subscriptionEstablished closure.

These methods appear in the documentation as though the docs were auto generated and have no further description of how to use them. Can anyone provide any insight into how these methods should be used as well as some insight into what I might be doing wrong? Frankly the subscribe method Kevin linked looks the most useful in my situation, but I can't make it work.

I am using Xcode Version 16.0 beta 2 (16A5171r) and have tried these methods on an iPad running iPadOS 18.0 Developer Beta 3. I also tried the subscribe:with method an iPhone running iOS 17.5.1 and got the same result.

Thanks for any additional insight you can offer.

i'm running into the same problem.

none of the subscribe methods work. the primary one mentioned above gives error 48, the other ones never confirm subscription or give any other results.

during my trials yesterday i might have seen a single subscription confirmation from one of them but unfortunately i can't reproduce the circumstances. also there i never got a change reported.

does anyone know what is supposed to be working for an MTRBaseDevice initialised from homekit?

i did some more test...

  • i can use readAttributes and read everything from the device. good.
  • except: i never get data for attribute 0xFFFA which should be the EventList. does this indicate that the MTRBaseDevice initialised from HomeKit is not capable of generating events?
  • all other global attributes 0xFFF8-0xFFFD except 0xFFFA return the expected data.
  • also: when i try to use invokeCommand to initiate an identify or off command. this also does not work and results in Error Domain=MTRErrorDomain Code=4 "An argument is invalid." UserInfo={NSLocalizedDescription=An argument is invalid.}. does this indicate that the MTRBaseDevice initialised from HomeKit is readonly?
  • on the other hand using an MTRBaseClusterOnOff initialised from the same MTRBaseDevice allows me to use toggle without any problems. so it seems not to be readonly...

the invokeCommand error from above might have been a problrm with the commanfFields. now that i changed this a little i get absolutely no reaction. no error, no data. like subscribe. so again the question: is anything besides reading the attributes supposed to work when using the homekit provided data?

i did another fix to the commanfFields to get on/off/toggle to work. basically every command without data fields.

but i can not figure out how to encode a parameter...

besides the encoding only the subscription seems to be still open for me.

I'm going to share a couple of things, I'll let you decide if they are helpful at all.

#1 This conversation I had with Kevin has more back and forth than I see in this thread that includes your comments. I suggest you search again and see if you can find the full conversation.

#2 A key part of the missing conversation referred to in #1 suggested that I submit a bug for the subscribe APIs not working. I did that and received the following information, which give no timetable 'cuz Apple doesn't talk about anything unannounced. In the bug report I specifically called out the process Kevin provided in this thread for accessing Matter devices via HomeKit and that process is what they were referring to when they say "this HomeKit API". Here is what they said:

"MTRBaseDevice.subscribe API is not supported by this HomeKit API to create Matter Controllers for Apple Home via Node ID.

In the future, we'll be adding a new API that makes use of MTRDevice interface, which is better suited for subscription via this interface."

found the correct encoding for parameters. so invokeCommand now works for me also.

so the only one still missing is subscription to events. and the missing data for attribute 0xFFFA

First off, a disclaimer. Everything below is based on my understanding of the state of our current implementation. That implementation is both complex and under very active development, which means there may be mistakes in the details of what I'm describing. I think the information will still be helpful, but you should not assume this is correct in every detail, particularly in the future.

These methods appear in the documentation as though the docs were auto generated and have no further description of how to use them.

The Matter framework is under very active and unusually public development. The fact that it has method that were released and the deprecated with two software update (16.1-> 16.4) is very different than the way we generally develop frameworks. It's documentation state is a natural outgrowth of that reality. My main recommendation here is that you file bugs asking for Matter to be better documented. Note that the goal here isn't to "tell" the matter team about the lack of documentation (they obviously already know that), it's to document your interest so we can properly prioritize the necessary work.

does anyone know what is supposed to be working for an MTRBaseDevice initialised from homekit?

So, first as some background context, what's actually going on here is that the MTRBaseDevice you've initialized through HomeKit is NOT in fact a "real" MTRBaseDevice. What the object actually is partial implementation of MTRBaseDevice that uses XPC to send it's method calls to homed, which is where HomeKit's matter ecosystem implementation actually runs. That implementation also explains most of the limitations around how it operates.

That leads to here:

linked returns an HMError #48 to my errorHandler closure when I call it. The localized description of that error is "The operation couldn’t be completed.", which certainly tells me something went wrong but provides no actionable remedies.

Please file a bug on this and post the bug back here. What's going on here is that while the Matter implementation of this object should "work" (at least in theory), HomeKit is intentionally failing the command over concerns about the load the current subscription implementation inside the Matter framework could end up overloading the accessory target.

As a side note here, any time you're receiving HomeKit errors from this API, that error is coming from the intermediate XPC layer, not the underlying matter implementation.

Expanding on that point...

I tried the other 3 subscribe methods that haven't been deprecated. Two of them give me the result above via my reportHandler. The remaining one, subscribeToEvents, never calls my reportHandler closure (where I would get an error or a report) and also never calls my subscriptionEstablished closure.

Having spent more time looking at our code, here is my best summary of the state of things:

  1. subscribeToAttributesWithEndpointID is implemented by making two different XPC calls, one call directly fetches the report while the other tries to setup a subscription using the same call path as subscribeWithQueue. That second call actually fails (just like subscribeWithQueue), but you don't "see" that error because the existing API doesn't provide an error path.

  2. subscribeToAttributePaths is difficult to trace as it's using the default MTBaseDevice implementation and which calls into the controller controller object (which is also an XPC wrapper).

  3. subscribeToEventsWithEndpointID is actually implemented by calling subscribeToAttributePaths with a NULL attributePaths and constructed array for eventPaths.

The main thing to be aware of here is that I think all of these are really just retrieving report data, not actually establishing a valid subscription.

The remaining one, subscribeToEvents, never calls my reportHandler closure (where I would get an error or a report) and also never calls my subscriptionEstablished closure.

My recommendation would be to avoid this API and just use 1 or (possibly) 2.

except: i never get data for attribute 0xFFFA which should be the EventList. does this indicate that the MTRBaseDevice initialised from HomeKit is not capable of generating events?

I'm not sure. I assume you're calling readAttributesWithEndpointID? What error did you get back?

Also, one final note here:

so the only one still missing is subscription to events.

Keep in mind that you also have an HMAccessory you can interact with, including being notified of state changes to the characteristics visible to HomeKit. That obviously won't be "everything" but I'd expect to cover the data most relevant to the user.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Support for custom Matter endpoints, clusters and attributes
 
 
Q