Personal Voice authorization requires app restart

I'm trying to include Apple's Personal Voice feature in an app I'm working on, but I want to use a button or toggle to request access, rather than firing the request on first launch. The problem is that, if AVSpeechSynthesizer is used during the same session, before Personal Voice is authorized, the app has to be restarted to use the feature.

Here is a basic example that demonstrates the issue on my iPhone (running 18.1 beta, but the issue was present at least in 18.0, maybe before):

import AVFoundation
import SwiftUI

struct TestView: View {
    let synthesizer = AVSpeechSynthesizer()
    @State private var personalVoices: [AVSpeechSynthesisVoice] = []
    
    var body: some View {
        VStack(spacing: 100) {
            Text("Personal Voices Available: \(personalVoices.count)")
            
            Button {
                speakUtterance(string: "Hello, world!")
            } label: {
                Image(systemName: "hand.wave.fill")
                    .font(.system(size: 100))
            }
            
            Button("Fetch Personal Voices") {
                Task { await fetchPersonalVoices() }
            }
        }
    }
    
    func fetchPersonalVoices() async {
        AVSpeechSynthesizer.requestPersonalVoiceAuthorization() { status in
            if status == .authorized {
                personalVoices = AVSpeechSynthesisVoice.speechVoices().filter { $0.voiceTraits.contains(.isPersonalVoice) }
            }
        }
    }
    
    func speakUtterance(string: String) {
        let utterance = AVSpeechUtterance(string: string)
        
        if let voice = personalVoices.first {
            utterance.voice = voice
        } else {
            utterance.voice = AVSpeechSynthesisVoice(language: Locale.preferredLanguages[0])
        }
        
        synthesizer.speak(utterance)
    }
}

If you tap the hand symbol first (before authorizing Personal Voice), you'll probably notice that the Personal Voices Available number never increases. If you authorize Personal Voice before tapping the hand symbol, it should speak using your Personal Voice as expected.

The example code is mostly taken directly from this WWDC23 video (Personal Voice info begins around the 10-minute mark).

Does anyone have any idea what could be causing this?

Note: Personal Voice can't be tested in Simulator. The code will need to be run on a physical device that has Personal Voice set up, to test.