Stereo video HLS

Question

Created Dec ’23

Replies 5

Boosts 1

Views 1.8k

Participants 4

I am trying to set up HLS with MV HEVC. I have an MV HEVC MP4 converted with AVAssetWriter that plays as a "spatial video" in Photos in the simulator. I've used ffmpeg to fragment the video for HLS (sample m3u8 file below).

The HLS of the mp4 plays on a VideoMaterial with an AVPlayer in the simulator, but it is hard to determine if the streamed video is stereo. Is there any guidance on confirming that the streamed mp4 video is properly being read as stereo?

Additionally, I see that REQ-VIDEO-LAYOUT is required for multivariant HLS. However if there is ONLY stereo video in the playlist is it needed? Are there any other configurations need to make the device read as stereo?

Sample m3u8 playlist

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:13
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:12.512500,
sample_video0.ts
#EXTINF:8.341667,
sample_video1.ts
#EXTINF:12.512500,
sample_video2.ts
#EXTINF:8.341667,
sample_video3.ts
#EXTINF:8.341667,
sample_video4.ts
#EXTINF:12.433222,
sample_video5.ts
#EXT-X-ENDLIST

Boost

Answer 1

altonelli OP

Jan ’24

Update on my end: I decided to using mediafilesegmenter Version 1.20.9 (618.21b-230614) to keep all software tooling Apple, however I ran into the following error:

Jan  1 2024 16:44:06.294: Fragmented MP4 is the only supported container format for the segmentation of HEVC content
Jan  1 2024 16:44:06.294: Unsupported media type 'hvc1' in track 0
Jan  1 2024 16:44:06.294: Unable to find any valid tracks to segment.
Segmenting failed (-12780).

Again, I'm trying to segment MV-HEVC (mp4) files which I converted using use the mvhevc1440x1440 AVOutputSettingsPreset with some overrides. My codec key is still hvc1 from the preset which makes sense for this error. What is confusing is that the hls recommendations (1.10) recommend hvc1. So I'm not sure why I'm getting an unsupported media type error.

0

Answer 2

altonelli OP

Jan ’24

I arrived at a solution. I came to realized that fMP4 (fragmented MP4) in the error was in reference to the output format rather than the source format. mediafilesegmenter defaults to ts segmentation rather than fMP4. I added -iso-fragmented option to the command so it looked similar to the demo script below.

mediafilesegmenter -iso-fragmented -t 4 -b https://www.example.com/spatial_video_stream/ -f ~/path/to/dir/with/file spatial_video.mov

Reference post that helped: https://developer.apple.com/forums/thread/71433

0

Answer 3

jontg OP

Feb ’24

Did you figure this out? I'm getting the same error:

$ mediafilesegmenter -iso-fragmented -t 2 ./sample_spatial.mov       
Feb 17 2024 10:04:00.249: ISO fragmented mode, forcing segments to start with I-Frame
Feb 17 2024 10:04:00.249: Processing file /Users/jon/my_dir/sample_spatial.mov
Feb 17 2024 10:04:00.259: Format description for track index 3 mismatch with another track with same media type
Feb 17 2024 10:04:00.259: detected mismatch in track - don't use this file /Users/jon/my_dir/sample_spatial.mov
Feb 17 2024 10:04:00.259: Unable to find any valid tracks to segment.
Segmenting failed (-17185).

I'm using the latest mediafilesegmenter (Version 1.22 (659.13b-240124)

0

Answer 4

brrruski OP

Feb ’24

Apple engineer mentioned in this video Deliver video content for spatial experiences where it says The HLS multivariant playlist format should be:

EXT-X-VERSION is updated to version 12
New REQ-VIDEO-LAYOUT attribute
- For EXT-X-STREAM-INF tag for video content
- Video Channel Specifier can have values CH-STEREO or CH-MONO or a mix

Firstly I tried using the simple command to parse a mv-hevc spatial video recored by a vision pro, but it ended up with an error "Unsupported media type 'hvc1' in track 0"

Then I tried @altonelli 's answer, i.e. added a param "-iso-fragmented", it worked.

mediafilesegmenter -iso-fragmented -t 4 -f output/sp_video-1-ip ./videos/sp_video-1-vp.MOV

Feb 22 2024 19:38:59.996: ISO fragmented mode, forcing segments to start with I-Frame
Feb 22 2024 19:38:59.997: Processing file /Users/sizhe/workspace/projects/avp-spatial-video/videos/sp_video-1-vp.MOV
Feb 22 2024 19:39:00.020: Finalized 2300 bytes in 0: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence0.mp4
Feb 22 2024 19:39:00.027: Finalized 14560188 bytes in 1: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence1.m4s
Feb 22 2024 19:39:00.027: Wrote full file size 14560188 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence1.m4s
Feb 22 2024 19:39:00.027: segment bitrate 27.96 Mbits/sec is new max
Feb 22 2024 19:39:00.053: Finalized 14689506 bytes in 2: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence2.m4s
Feb 22 2024 19:39:00.053: Wrote full file size 14689506 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence2.m4s
Feb 22 2024 19:39:00.078: Finalized 15339992 bytes in 3: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence3.m4s
Feb 22 2024 19:39:00.078: Wrote full file size 15339992 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence3.m4s
Feb 22 2024 19:39:00.078: segment bitrate 28.76 Mbits/sec is new max
Feb 22 2024 19:39:00.103: Finalized 15668274 bytes in 4: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence4.m4s
Feb 22 2024 19:39:00.103: Wrote full file size 15668274 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence4.m4s
Feb 22 2024 19:39:00.103: segment bitrate 29.38 Mbits/sec is new max
Feb 22 2024 19:39:00.128: Finalized 14482468 bytes in 5: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence5.m4s
Feb 22 2024 19:39:00.128: Wrote full file size 14482468 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence5.m4s
Feb 22 2024 19:39:00.138: Finalized 4188748 bytes in 6: /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence6.m4s
Feb 22 2024 19:39:00.138: Wrote full file size 4188748 to /Users/sizhe/workspace/projects/avp-spatial-video/output/sp_video-1-ip/fileSequence6.m4s
Feb 22 2024 19:39:00.139: average bit rate is 28.19 Mbits/sec - max file bit rate is 29.38 Mbits/sec

But when I looked into the m3u8 file, there is no REQ-VIDEO-LAYOUT attribute, neither the EXT-X-VERSION is 12 but 7. Any idea why?

#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-VERSION:7
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MAP:URI="fileSequence0.mp4"
#EXTINF:4.16652,	
#EXT-X-BITRATE:27957
fileSequence1.m4s
#EXTINF:4.26653,	
#EXT-X-BITRATE:27544
fileSequence2.m4s
#EXTINF:4.26652,	
#EXT-X-BITRATE:28763
fileSequence3.m4s
#EXTINF:4.26652,	
#EXT-X-BITRATE:29379
fileSequence4.m4s
#EXTINF:4.26652,	
#EXT-X-BITRATE:27156
fileSequence5.m4s
#EXTINF:1.16663,	
#EXT-X-BITRATE:28724
fileSequence6.m4s
#EXT-X-ENDLIST

1

Answer 5

jguo OP

May ’24

There is one extra step. You need to use the variantplaylistcreator tool in the package to build the multivariant playlist. Pass in the m3u8 and plist files that's generated by the mediafilesegmenter. Use man variantplaylistcreator to see how to use it.

Here is the output I got:

#EXTM3U
#EXT-X-VERSION:12
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=40207799,BANDWIDTH=46965192,VIDEO-RANGE=SDR,CODECS="mp4a.40.2,hvc1.1.60000000.L153.B0",RESOLUTION=2200x2200,FRAME-RATE=30.000,CLOSED-CAPTIONS=NONE,REQ-VIDEO-LAYOUT="CH-STEREO"
prog_index.m3u8
#EXT-X-I-FRAME-STREAM-INF:AVERAGE-BANDWIDTH=1596040,BANDWIDTH=1907942,VIDEO-RANGE=SDR,CODECS="hvc1.1.60000000.L153.B0",RESOLUTION=2200x2200,REQ-VIDEO-LAYOUT="CH-STEREO",URI="iframe_index.m3u8"

0