netdesk/native/docs/cross-platform-decode.md

Cross-platform hardware H.264 decode

NetDesk's core decodes the host's H.264 (Annex-B) stream into packed-RGB8 [Frame]s for the UI to paint. Every platform shares one trait and one output type, so the viewer never knows which backend is running:

pub struct Frame { pub w: usize, pub h: usize, pub rgb: Vec<u8> } // w*h*3 bytes
pub trait VideoDecoder { fn decode(&mut self, au: &[u8]) -> Result<Option<Frame>>; }
pub fn best_decoder() -> Box<dyn VideoDecoder + Send>;

best_decoder() selects the fastest decoder for the current OS at runtime and always falls back to the portable openh264 software decoder if the native hardware path cannot initialize (no GPU, missing runtime, headless box, etc.).

Target	Backend	Module	API	Crate(s)
Windows	Media Foundation (DXVA2)	core/src/decode_mf.rs	IMFTransform (MSH264 MFT)	windows
macOS / iOS	VideoToolbox	core/src/decode_vt.rs	VTDecompressionSession	core-foundation(-sys) + frameworks
Android	MediaCodec	core/src/decode_mediacodec.rs	AMediaCodec (NDK)	ndk-sys / libmediandk
Linux	VAAPI	core/src/decode_vaapi.rs	FFmpeg/libavcodec + VAAPI	ffmpeg-next
All (fallback)	openh264	core/src/decode.rs	software	openh264

Each native module is #[cfg]-gated to its own OS, so building for one target never compiles (or even links) another target's module or its dependencies. The per-platform crates are declared under [target.'cfg(...)'.dependencies] in core/Cargo.toml; on a non-matching host they are only added to Cargo.lock (resolved), never built, and their build scripts never run.

All backends emit the true display dimensions (e.g. 1920×1080), cropping out any codec padding / stride / alignment, so frames are never stretched.

---

Windows — Media Foundation (reference, builds here)

Already shipping. Build with the portable MinGW toolchain:

export PATH="/c/mingw-netdesk/mingw64/bin:$HOME/.cargo/bin:$PATH"
cargo build -p netdesk-core            # or: --target x86_64-pc-windows-gnu

Dimension fix (2026-10)

The MS H.264 decoder MFT reports the coded size in MF_MT_FRAME_SIZE, which is padded up to the codec's macroblock alignment (1080p → a 1088+ coded height; other resolutions pad larger — this is what produced the wrong 1920x18xx in the stats). The real picture rectangle is carried separately in a video-aperture blob. decode_mf.rs now:

1. Reads MF_MT_FRAME_SIZE into coded_w/coded_h — used only for the NV12 plane layout (stride × plane height, and the UV plane offset).
2. Reads the display rectangle from, in priority order, MF_MT_MINIMUM_DISPLAY_APERTURE → MF_MT_GEOMETRIC_APERTURE → MF_MT_PAN_SCAN_APERTURE (the last only when MF_MT_PAN_SCAN_ENABLED), as an MFVideoArea blob giving offset (x, y) and visible width×height. The aperture is clamped inside the coded frame so a malformed blob can't over-read.
3. nv12_to_rgb(..) copies only the visible width×height region at (crop_x, crop_y) out of the padded NV12 (system-memory and DXGI/D3D11 staging paths both), so Frame.w/Frame.h are the true 1920×1080.

When no aperture is present the whole coded frame is treated as the picture (unchanged behaviour).

---

macOS / iOS — VideoToolbox

decode_vt.rs, gated #[cfg(any(target_os = "macos", target_os = "ios"))].

Feeds AUs into a VTDecompressionSession on the platform video engine and requests 32-bit BGRA output (VideoToolbox does YUV→RGB on the GPU), which is repacked to RGB8. The session is built lazily from the first keyframe's SPS/PPS (CMVideoFormatDescriptionCreateFromH264ParameterSets); Annex-B is repacked to AVCC for the CMSampleBuffer. CVPixelBufferGetWidth/Height already return the cropped display size, so no manual crop is needed.

Links the system CoreMedia, CoreVideo and VideoToolbox frameworks directly; core-foundation is used for the pixel-buffer attribute dictionary.

Build (requires macOS + Xcode)

xcode-select --install                 # or a full Xcode install
rustup target add aarch64-apple-darwin x86_64-apple-darwin   # macOS
rustup target add aarch64-apple-ios aarch64-apple-ios-sim    # iOS

cargo build -p netdesk-core --target aarch64-apple-darwin    # Apple-silicon mac
cargo build -p netdesk-core --target aarch64-apple-ios       # device
cargo build -p netdesk-core --target aarch64-apple-ios-sim   # simulator

The framework #[link]s resolve against the SDK the active toolchain selects (xcrun --sdk macosx|iphoneos|iphonesimulator --show-sdk-path). No extra flags.

Test

cargo test -p netdesk-core --target aarch64-apple-darwin

Smoke-test end to end by running the viewer against a live host; the log line decode: VideoToolbox H.264 (hardware) confirms the path was selected.

---

Android — MediaCodec

decode_mediacodec.rs, gated #[cfg(target_os = "android")].

Drives a byte-buffer-mode AMediaCodec AVC decoder (the SoC hardware decoder via OMX/Codec2). The codec is configured lazily from the first keyframe's SPS/PPS as csd-0/csd-1; each decode queues the Annex-B AU as one input buffer and dequeues one output buffer. Output is YUV 4:2:0 — NV12 semi-planar or I420 planar depending on the device — converted to RGB8 honouring the format's stride, slice-height, color-format, and crop-left/top/right/bottom keys, so the emitted frame is the true display size.

Talks to the stable NDK Media C ABI in libmediandk directly (the same ABI the ndk-sys crate binds), so it is version-independent.

Build (requires the Android NDK)

Install the NDK (via Android Studio's SDK Manager or sdkmanager "ndk;27.0.x"), then use [cargo-ndk]:

rustup target add aarch64-linux-android armv7-linux-androideabi \
                  x86_64-linux-android i686-linux-android
cargo install cargo-ndk
export ANDROID_NDK_HOME=$HOME/Android/Sdk/ndk/27.0.12077973

# API 24+; links libmediandk / liblog automatically via the NDK sysroot.
cargo ndk -t arm64-v8a -t armeabi-v7a -p 24 build -p netdesk-core --release

Or without cargo-ndk, point Cargo at the NDK clang linker for each target, e.g.:

export CC_aarch64_linux_android=$ANDROID_NDK_HOME/.../aarch64-linux-android24-clang
export CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$CC_aarch64_linux_android
cargo build -p netdesk-core --target aarch64-linux-android --release

Test

Run on a device/emulator (the mediandk APIs are not present on the host):

cargo ndk -t arm64-v8a build -p netdesk-core --tests
# push the test binary with `adb push` and run it on-device, or exercise it
# through the app's instrumented tests.

The log line decode: MediaCodec H.264 (hardware) confirms selection.

---

Linux — VAAPI (via FFmpeg)

decode_vaapi.rs, gated #[cfg(target_os = "linux")].

Opens the libavcodec H.264 decoder with a VAAPI hw_device_ctx (default DRI render node). libavcodec does all bitstream parsing; decode runs on the GPU, the surface is pulled to system memory with av_hwframe_transfer_data (NV12), cropped to the display rectangle with av_frame_apply_cropping, and converted to RGB8 with swscale. If no VAAPI device is usable, new() errors and the core falls back to openh264, so GPU-less / headless boxes still work.

Pulls libav* in through the ffmpeg-next crate and needs the system FFmpeg (≥ 4) development libraries and a VAAPI driver at build/run time.

Build (requires system FFmpeg + VAAPI dev packages)

Debian/Ubuntu:

sudo apt install -y clang pkg-config \
  libavcodec-dev libavutil-dev libswscale-dev libavformat-dev \
  libva-dev vainfo intel-media-va-driver   # or mesa-va-drivers for AMD

cargo build -p netdesk-core                # host target

Fedora:

sudo dnf install clang pkgconf-pkg-config \
  ffmpeg-devel libva-devel libva-utils mesa-va-drivers
cargo build -p netdesk-core

ffmpeg-sys-next locates the libraries with pkg-config; set PKG_CONFIG_PATH if FFmpeg is in a non-standard prefix. bindgen needs clang/libclang (the clang package above).

Test

vainfo                                     # confirm a VAAPI driver + H264 profile
cargo test -p netdesk-core

At runtime the log line decode: VAAPI H.264 (FFmpeg hardware) confirms the hardware path; VAAPI init failed (...); using openh264 software decode means it fell back (check vainfo, /dev/dri/renderD128 permissions, and the LIBVA_DRIVER_NAME driver).

---

Adding another backend

1. Add core/src/decode_<name>.rs implementing VideoDecoder + unsafe impl Send, with a new() -> Result<Self> that errors when the platform can't support it.
2. Declare the module in decode.rs behind the right #[cfg(...)].
3. Add a #[cfg(...)] branch in best_decoder() before the openh264 fallback.
4. Add the deps under a matching [target.'cfg(...)'.dependencies] in core/Cargo.toml.
5. Emit the true display size (crop any codec padding) and BT.601 limited- range RGB to match the other backends.