Chapters
0:00 - Introduction
1:07 - Types of stereoscopic experiences
4:13 - Tour of the new APIs
13:14 - Deep dive into spatial media formats
21:16 - Wrap-up
Stereoscopic video experiences
Formats:
3D video, like 3D movies available in Apple TV and Disney+. Renders flat.
Spatial video, captured on iPhone 15 pro and Vision Pro. Renders through a window.
Apple Immersive video, for high-end professional content. 180 degree, 8k resolution, spatial audio.
Full immersion
3D video docks into an environment, screen moves backward and enlarges. (see also [Enhance video playback and immersion in your custom app environment]). Spatial video expands into its immersive presentation.
Content wraps around user.
Other stereo experiences:
Spatial photos
Custom video experiences
Interactive 3D content
See also: Bring your iOS or iPadOS game to visionOS
New APIs
No new frameworks, changes ar integrated into existing frameworks
Capturing spatial video
Code walkthrough: AVCaptureSession to capture spatial video.
isSpatialVideoCaptureSupported fails on hardware other than iPhone 15 Pro. Get full-formed spatial video file on disk.
Enhancements: improved video stabilization, and a great preview.
“Wide” and “Ultrawide” cameras have different capture characteristics. Can cause noise level variation, or focus mismatch. Not detectable until playback on Vision Pro. So have added new variable spatialCaptureDiscomfortReasons on AVCaptureDevice for feedback while shooting. See iPhone Camera App for first-party UI example of this feedback.
Detecting spatial media
Detection Methods
PhotosPicker. Can filter library to show only
.spatialMediaassets.PhotoKit. Pass
PHAssetMediaSubtype.spatialMediatofetchAssetscall. Can do just spatial photos or spatial videos.AVAssetPlaybackAssistant.
.playbackConfigurationOptionscan be configured for spatial assets.
Presentation Options
PreviewApplication API (spawn Quicklook scene). See “What’s new in Quick Look for spatial computing”.
Element Fullscreen API (Javascript). Open spatial photos in Safari. See “Optimize for the Spatial Web”.
AVPlayerViewController. Supports 2D and 3D video content, also HTTP Live Streaming for spatial video. Only displays as 3D if full screen. Shows 3D video, not spatial video. Use PreviewApplication if you need spatial video display.
Create custom spatial media
File format discussion. Two sample projects provided, for spatial video and spatial photos.
Projection. Spatial photos and videos always use the rectilinear projection.
Baseline, Field of View: camera properties.
Disparity adjustment
Example: stereoscopic image of a hummingbird.
Note: the following metadata discussion is put into practice in the Writing spatial photos sample app.
For optimal image characteristics the left and right images should be stereo rectified, have optical axis alignment, and have no vertical disparity. Rectilinear: straight lines in world are straight lines on image. Horizontal baseline: 64 mm similar to human eye, 32 mm good for closeups, larger than 64 mm good for stereo landscape photography.
FOV greater than 90 degrees is inefficient for rectilinear capture, 60 degrees used for this analysis.
Images must be coplanar (or stereorectified in post-production).
Cropping: center axis of cameras should be at centers of images to keep good alignment. Vertical disparity, if present, makes images uncomfortable to view (same feature should have same Y coordinate in each eye).
Vision Pro renderer uses the metadata to construct a camera model, renders different image for each eye.
Horizontal disparity adjustment: a hint to the renderer on where to place the zero parallax plane. Sometimes called convergence adjustment. Move left/right frames horizontally to adjust disparity. This controls perception of how far away the 3D scene appears to be.
Wrap-up. Covered:
multiple types of stereo media
capture, detect, and display spatial media
create your own spatial media
