# Cross-platform hardware H.264 decode NetDesk's core decodes the host's H.264 (Annex-B) stream into packed-RGB8 [`Frame`]s for the UI to paint. Every platform shares one trait and one output type, so the viewer never knows which backend is running: ```rust pub struct Frame { pub w: usize, pub h: usize, pub rgb: Vec } // w*h*3 bytes pub trait VideoDecoder { fn decode(&mut self, au: &[u8]) -> Result>; } pub fn best_decoder() -> Box; ``` `best_decoder()` selects the fastest decoder for the current OS at runtime and **always** falls back to the portable openh264 software decoder if the native hardware path cannot initialize (no GPU, missing runtime, headless box, etc.). | Target | Backend | Module | API | Crate(s) | |---------------|----------------|---------------------------|-----------------------------|---------------------------------| | Windows | Media Foundation (DXVA2) | `core/src/decode_mf.rs` | `IMFTransform` (MSH264 MFT) | `windows` | | macOS / iOS | VideoToolbox | `core/src/decode_vt.rs` | `VTDecompressionSession` | `core-foundation(-sys)` + frameworks | | Android | MediaCodec | `core/src/decode_mediacodec.rs` | `AMediaCodec` (NDK) | `ndk-sys` / `libmediandk` | | Linux | VAAPI | `core/src/decode_vaapi.rs` | FFmpeg/libavcodec + VAAPI | `ffmpeg-next` | | All (fallback)| openh264 | `core/src/decode.rs` | software | `openh264` | Each native module is `#[cfg]`-gated to its own OS, so building for one target never compiles (or even links) another target's module or its dependencies. The per-platform crates are declared under `[target.'cfg(...)'.dependencies]` in `core/Cargo.toml`; on a non-matching host they are only added to `Cargo.lock` (resolved), never built, and their build scripts never run. All backends emit the **true display dimensions** (e.g. 1920×1080), cropping out any codec padding / stride / alignment, so frames are never stretched. --- ## Windows — Media Foundation (reference, builds here) Already shipping. Build with the portable MinGW toolchain: ```bash export PATH="/c/mingw-netdesk/mingw64/bin:$HOME/.cargo/bin:$PATH" cargo build -p netdesk-core # or: --target x86_64-pc-windows-gnu ``` ### Dimension fix (2026-10) The MS H.264 decoder MFT reports the **coded** size in `MF_MT_FRAME_SIZE`, which is padded up to the codec's macroblock alignment (1080p → a 1088+ coded height; other resolutions pad larger — this is what produced the wrong `1920x18xx` in the stats). The real picture rectangle is carried separately in a video-aperture blob. `decode_mf.rs` now: 1. Reads `MF_MT_FRAME_SIZE` into `coded_w`/`coded_h` — used **only** for the NV12 plane layout (stride × plane height, and the UV plane offset). 2. Reads the display rectangle from, in priority order, `MF_MT_MINIMUM_DISPLAY_APERTURE` → `MF_MT_GEOMETRIC_APERTURE` → `MF_MT_PAN_SCAN_APERTURE` (the last only when `MF_MT_PAN_SCAN_ENABLED`), as an `MFVideoArea` blob giving offset `(x, y)` and visible `width`×`height`. The aperture is clamped inside the coded frame so a malformed blob can't over-read. 3. `nv12_to_rgb(..)` copies only the visible `width`×`height` region at `(crop_x, crop_y)` out of the padded NV12 (system-memory and DXGI/D3D11 staging paths both), so `Frame.w`/`Frame.h` are the true `1920×1080`. When no aperture is present the whole coded frame is treated as the picture (unchanged behaviour). --- ## macOS / iOS — VideoToolbox `decode_vt.rs`, gated `#[cfg(any(target_os = "macos", target_os = "ios"))]`. Feeds AUs into a `VTDecompressionSession` on the platform video engine and requests 32-bit BGRA output (VideoToolbox does YUV→RGB on the GPU), which is repacked to RGB8. The session is built lazily from the first keyframe's SPS/PPS (`CMVideoFormatDescriptionCreateFromH264ParameterSets`); Annex-B is repacked to AVCC for the `CMSampleBuffer`. `CVPixelBufferGetWidth/Height` already return the cropped display size, so no manual crop is needed. Links the system **CoreMedia**, **CoreVideo** and **VideoToolbox** frameworks directly; `core-foundation` is used for the pixel-buffer attribute dictionary. ### Build (requires macOS + Xcode) ```bash xcode-select --install # or a full Xcode install rustup target add aarch64-apple-darwin x86_64-apple-darwin # macOS rustup target add aarch64-apple-ios aarch64-apple-ios-sim # iOS cargo build -p netdesk-core --target aarch64-apple-darwin # Apple-silicon mac cargo build -p netdesk-core --target aarch64-apple-ios # device cargo build -p netdesk-core --target aarch64-apple-ios-sim # simulator ``` The framework `#[link]`s resolve against the SDK the active toolchain selects (`xcrun --sdk macosx|iphoneos|iphonesimulator --show-sdk-path`). No extra flags. ### Test ```bash cargo test -p netdesk-core --target aarch64-apple-darwin ``` Smoke-test end to end by running the viewer against a live host; the log line `decode: VideoToolbox H.264 (hardware)` confirms the path was selected. --- ## Android — MediaCodec `decode_mediacodec.rs`, gated `#[cfg(target_os = "android")]`. Drives a byte-buffer-mode `AMediaCodec` AVC decoder (the SoC hardware decoder via OMX/Codec2). The codec is configured lazily from the first keyframe's SPS/PPS as `csd-0`/`csd-1`; each `decode` queues the Annex-B AU as one input buffer and dequeues one output buffer. Output is YUV 4:2:0 — NV12 semi-planar or I420 planar depending on the device — converted to RGB8 honouring the format's `stride`, `slice-height`, `color-format`, and `crop-left/top/right/bottom` keys, so the emitted frame is the true display size. Talks to the stable NDK Media C ABI in **`libmediandk`** directly (the same ABI the `ndk-sys` crate binds), so it is version-independent. ### Build (requires the Android NDK) Install the NDK (via Android Studio's SDK Manager or `sdkmanager "ndk;27.0.x"`), then use [`cargo-ndk`]: ```bash rustup target add aarch64-linux-android armv7-linux-androideabi \ x86_64-linux-android i686-linux-android cargo install cargo-ndk export ANDROID_NDK_HOME=$HOME/Android/Sdk/ndk/27.0.12077973 # API 24+; links libmediandk / liblog automatically via the NDK sysroot. cargo ndk -t arm64-v8a -t armeabi-v7a -p 24 build -p netdesk-core --release ``` Or without cargo-ndk, point Cargo at the NDK clang linker for each target, e.g.: ```bash export CC_aarch64_linux_android=$ANDROID_NDK_HOME/.../aarch64-linux-android24-clang export CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$CC_aarch64_linux_android cargo build -p netdesk-core --target aarch64-linux-android --release ``` ### Test Run on a device/emulator (the `mediandk` APIs are not present on the host): ```bash cargo ndk -t arm64-v8a build -p netdesk-core --tests # push the test binary with `adb push` and run it on-device, or exercise it # through the app's instrumented tests. ``` The log line `decode: MediaCodec H.264 (hardware)` confirms selection. --- ## Linux — VAAPI (via FFmpeg) `decode_vaapi.rs`, gated `#[cfg(target_os = "linux")]`. Opens the libavcodec H.264 decoder with a VAAPI `hw_device_ctx` (default DRI render node). libavcodec does all bitstream parsing; decode runs on the GPU, the surface is pulled to system memory with `av_hwframe_transfer_data` (NV12), cropped to the display rectangle with `av_frame_apply_cropping`, and converted to RGB8 with swscale. If no VAAPI device is usable, `new()` errors and the core falls back to openh264, so GPU-less / headless boxes still work. Pulls libav* in through the **`ffmpeg-next`** crate and needs the system FFmpeg (≥ 4) development libraries and a VAAPI driver at build/run time. ### Build (requires system FFmpeg + VAAPI dev packages) Debian/Ubuntu: ```bash sudo apt install -y clang pkg-config \ libavcodec-dev libavutil-dev libswscale-dev libavformat-dev \ libva-dev vainfo intel-media-va-driver # or mesa-va-drivers for AMD cargo build -p netdesk-core # host target ``` Fedora: ```bash sudo dnf install clang pkgconf-pkg-config \ ffmpeg-devel libva-devel libva-utils mesa-va-drivers cargo build -p netdesk-core ``` `ffmpeg-sys-next` locates the libraries with `pkg-config`; set `PKG_CONFIG_PATH` if FFmpeg is in a non-standard prefix. `bindgen` needs `clang`/`libclang` (the `clang` package above). ### Test ```bash vainfo # confirm a VAAPI driver + H264 profile cargo test -p netdesk-core ``` At runtime the log line `decode: VAAPI H.264 (FFmpeg hardware)` confirms the hardware path; `VAAPI init failed (...); using openh264 software decode` means it fell back (check `vainfo`, `/dev/dri/renderD128` permissions, and the `LIBVA_DRIVER_NAME` driver). --- ## Adding another backend 1. Add `core/src/decode_.rs` implementing `VideoDecoder` + `unsafe impl Send`, with a `new() -> Result` that errors when the platform can't support it. 2. Declare the module in `decode.rs` behind the right `#[cfg(...)]`. 3. Add a `#[cfg(...)]` branch in `best_decoder()` before the openh264 fallback. 4. Add the deps under a matching `[target.'cfg(...)'.dependencies]` in `core/Cargo.toml`. 5. Emit the **true display size** (crop any codec padding) and BT.601 limited- range RGB to match the other backends.