224 lines
9.5 KiB
Markdown
224 lines
9.5 KiB
Markdown
# Cross-platform hardware H.264 decode
|
||
|
||
NetDesk's core decodes the host's H.264 (Annex-B) stream into packed-RGB8
|
||
[`Frame`]s for the UI to paint. Every platform shares one trait and one output
|
||
type, so the viewer never knows which backend is running:
|
||
|
||
```rust
|
||
pub struct Frame { pub w: usize, pub h: usize, pub rgb: Vec<u8> } // w*h*3 bytes
|
||
pub trait VideoDecoder { fn decode(&mut self, au: &[u8]) -> Result<Option<Frame>>; }
|
||
pub fn best_decoder() -> Box<dyn VideoDecoder + Send>;
|
||
```
|
||
|
||
`best_decoder()` selects the fastest decoder for the current OS at runtime and
|
||
**always** falls back to the portable openh264 software decoder if the native
|
||
hardware path cannot initialize (no GPU, missing runtime, headless box, etc.).
|
||
|
||
| Target | Backend | Module | API | Crate(s) |
|
||
|---------------|----------------|---------------------------|-----------------------------|---------------------------------|
|
||
| Windows | Media Foundation (DXVA2) | `core/src/decode_mf.rs` | `IMFTransform` (MSH264 MFT) | `windows` |
|
||
| macOS / iOS | VideoToolbox | `core/src/decode_vt.rs` | `VTDecompressionSession` | `core-foundation(-sys)` + frameworks |
|
||
| Android | MediaCodec | `core/src/decode_mediacodec.rs` | `AMediaCodec` (NDK) | `ndk-sys` / `libmediandk` |
|
||
| Linux | VAAPI | `core/src/decode_vaapi.rs` | FFmpeg/libavcodec + VAAPI | `ffmpeg-next` |
|
||
| All (fallback)| openh264 | `core/src/decode.rs` | software | `openh264` |
|
||
|
||
Each native module is `#[cfg]`-gated to its own OS, so building for one target
|
||
never compiles (or even links) another target's module or its dependencies. The
|
||
per-platform crates are declared under `[target.'cfg(...)'.dependencies]` in
|
||
`core/Cargo.toml`; on a non-matching host they are only added to `Cargo.lock`
|
||
(resolved), never built, and their build scripts never run.
|
||
|
||
All backends emit the **true display dimensions** (e.g. 1920×1080), cropping out
|
||
any codec padding / stride / alignment, so frames are never stretched.
|
||
|
||
---
|
||
|
||
## Windows — Media Foundation (reference, builds here)
|
||
|
||
Already shipping. Build with the portable MinGW toolchain:
|
||
|
||
```bash
|
||
export PATH="/c/mingw-netdesk/mingw64/bin:$HOME/.cargo/bin:$PATH"
|
||
cargo build -p netdesk-core # or: --target x86_64-pc-windows-gnu
|
||
```
|
||
|
||
### Dimension fix (2026-10)
|
||
|
||
The MS H.264 decoder MFT reports the **coded** size in `MF_MT_FRAME_SIZE`, which
|
||
is padded up to the codec's macroblock alignment (1080p → a 1088+ coded height;
|
||
other resolutions pad larger — this is what produced the wrong `1920x18xx` in the
|
||
stats). The real picture rectangle is carried separately in a video-aperture
|
||
blob. `decode_mf.rs` now:
|
||
|
||
1. Reads `MF_MT_FRAME_SIZE` into `coded_w`/`coded_h` — used **only** for the NV12
|
||
plane layout (stride × plane height, and the UV plane offset).
|
||
2. Reads the display rectangle from, in priority order,
|
||
`MF_MT_MINIMUM_DISPLAY_APERTURE` → `MF_MT_GEOMETRIC_APERTURE` →
|
||
`MF_MT_PAN_SCAN_APERTURE` (the last only when `MF_MT_PAN_SCAN_ENABLED`), as an
|
||
`MFVideoArea` blob giving offset `(x, y)` and visible `width`×`height`. The
|
||
aperture is clamped inside the coded frame so a malformed blob can't over-read.
|
||
3. `nv12_to_rgb(..)` copies only the visible `width`×`height` region at
|
||
`(crop_x, crop_y)` out of the padded NV12 (system-memory and DXGI/D3D11
|
||
staging paths both), so `Frame.w`/`Frame.h` are the true `1920×1080`.
|
||
|
||
When no aperture is present the whole coded frame is treated as the picture
|
||
(unchanged behaviour).
|
||
|
||
---
|
||
|
||
## macOS / iOS — VideoToolbox
|
||
|
||
`decode_vt.rs`, gated `#[cfg(any(target_os = "macos", target_os = "ios"))]`.
|
||
|
||
Feeds AUs into a `VTDecompressionSession` on the platform video engine and
|
||
requests 32-bit BGRA output (VideoToolbox does YUV→RGB on the GPU), which is
|
||
repacked to RGB8. The session is built lazily from the first keyframe's SPS/PPS
|
||
(`CMVideoFormatDescriptionCreateFromH264ParameterSets`); Annex-B is repacked to
|
||
AVCC for the `CMSampleBuffer`. `CVPixelBufferGetWidth/Height` already return the
|
||
cropped display size, so no manual crop is needed.
|
||
|
||
Links the system **CoreMedia**, **CoreVideo** and **VideoToolbox** frameworks
|
||
directly; `core-foundation` is used for the pixel-buffer attribute dictionary.
|
||
|
||
### Build (requires macOS + Xcode)
|
||
|
||
```bash
|
||
xcode-select --install # or a full Xcode install
|
||
rustup target add aarch64-apple-darwin x86_64-apple-darwin # macOS
|
||
rustup target add aarch64-apple-ios aarch64-apple-ios-sim # iOS
|
||
|
||
cargo build -p netdesk-core --target aarch64-apple-darwin # Apple-silicon mac
|
||
cargo build -p netdesk-core --target aarch64-apple-ios # device
|
||
cargo build -p netdesk-core --target aarch64-apple-ios-sim # simulator
|
||
```
|
||
|
||
The framework `#[link]`s resolve against the SDK the active toolchain selects
|
||
(`xcrun --sdk macosx|iphoneos|iphonesimulator --show-sdk-path`). No extra flags.
|
||
|
||
### Test
|
||
|
||
```bash
|
||
cargo test -p netdesk-core --target aarch64-apple-darwin
|
||
```
|
||
|
||
Smoke-test end to end by running the viewer against a live host; the log line
|
||
`decode: VideoToolbox H.264 (hardware)` confirms the path was selected.
|
||
|
||
---
|
||
|
||
## Android — MediaCodec
|
||
|
||
`decode_mediacodec.rs`, gated `#[cfg(target_os = "android")]`.
|
||
|
||
Drives a byte-buffer-mode `AMediaCodec` AVC decoder (the SoC hardware decoder via
|
||
OMX/Codec2). The codec is configured lazily from the first keyframe's SPS/PPS as
|
||
`csd-0`/`csd-1`; each `decode` queues the Annex-B AU as one input buffer and
|
||
dequeues one output buffer. Output is YUV 4:2:0 — NV12 semi-planar or I420
|
||
planar depending on the device — converted to RGB8 honouring the format's
|
||
`stride`, `slice-height`, `color-format`, and `crop-left/top/right/bottom` keys,
|
||
so the emitted frame is the true display size.
|
||
|
||
Talks to the stable NDK Media C ABI in **`libmediandk`** directly (the same ABI
|
||
the `ndk-sys` crate binds), so it is version-independent.
|
||
|
||
### Build (requires the Android NDK)
|
||
|
||
Install the NDK (via Android Studio's SDK Manager or `sdkmanager "ndk;27.0.x"`),
|
||
then use [`cargo-ndk`]:
|
||
|
||
```bash
|
||
rustup target add aarch64-linux-android armv7-linux-androideabi \
|
||
x86_64-linux-android i686-linux-android
|
||
cargo install cargo-ndk
|
||
export ANDROID_NDK_HOME=$HOME/Android/Sdk/ndk/27.0.12077973
|
||
|
||
# API 24+; links libmediandk / liblog automatically via the NDK sysroot.
|
||
cargo ndk -t arm64-v8a -t armeabi-v7a -p 24 build -p netdesk-core --release
|
||
```
|
||
|
||
Or without cargo-ndk, point Cargo at the NDK clang linker for each target, e.g.:
|
||
|
||
```bash
|
||
export CC_aarch64_linux_android=$ANDROID_NDK_HOME/.../aarch64-linux-android24-clang
|
||
export CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$CC_aarch64_linux_android
|
||
cargo build -p netdesk-core --target aarch64-linux-android --release
|
||
```
|
||
|
||
### Test
|
||
|
||
Run on a device/emulator (the `mediandk` APIs are not present on the host):
|
||
|
||
```bash
|
||
cargo ndk -t arm64-v8a build -p netdesk-core --tests
|
||
# push the test binary with `adb push` and run it on-device, or exercise it
|
||
# through the app's instrumented tests.
|
||
```
|
||
|
||
The log line `decode: MediaCodec H.264 (hardware)` confirms selection.
|
||
|
||
---
|
||
|
||
## Linux — VAAPI (via FFmpeg)
|
||
|
||
`decode_vaapi.rs`, gated `#[cfg(target_os = "linux")]`.
|
||
|
||
Opens the libavcodec H.264 decoder with a VAAPI `hw_device_ctx` (default DRI
|
||
render node). libavcodec does all bitstream parsing; decode runs on the GPU, the
|
||
surface is pulled to system memory with `av_hwframe_transfer_data` (NV12),
|
||
cropped to the display rectangle with `av_frame_apply_cropping`, and converted to
|
||
RGB8 with swscale. If no VAAPI device is usable, `new()` errors and the core
|
||
falls back to openh264, so GPU-less / headless boxes still work.
|
||
|
||
Pulls libav* in through the **`ffmpeg-next`** crate and needs the system FFmpeg
|
||
(≥ 4) development libraries and a VAAPI driver at build/run time.
|
||
|
||
### Build (requires system FFmpeg + VAAPI dev packages)
|
||
|
||
Debian/Ubuntu:
|
||
|
||
```bash
|
||
sudo apt install -y clang pkg-config \
|
||
libavcodec-dev libavutil-dev libswscale-dev libavformat-dev \
|
||
libva-dev vainfo intel-media-va-driver # or mesa-va-drivers for AMD
|
||
|
||
cargo build -p netdesk-core # host target
|
||
```
|
||
|
||
Fedora:
|
||
|
||
```bash
|
||
sudo dnf install clang pkgconf-pkg-config \
|
||
ffmpeg-devel libva-devel libva-utils mesa-va-drivers
|
||
cargo build -p netdesk-core
|
||
```
|
||
|
||
`ffmpeg-sys-next` locates the libraries with `pkg-config`; set
|
||
`PKG_CONFIG_PATH` if FFmpeg is in a non-standard prefix. `bindgen` needs
|
||
`clang`/`libclang` (the `clang` package above).
|
||
|
||
### Test
|
||
|
||
```bash
|
||
vainfo # confirm a VAAPI driver + H264 profile
|
||
cargo test -p netdesk-core
|
||
```
|
||
|
||
At runtime the log line `decode: VAAPI H.264 (FFmpeg hardware)` confirms the
|
||
hardware path; `VAAPI init failed (...); using openh264 software decode` means it
|
||
fell back (check `vainfo`, `/dev/dri/renderD128` permissions, and the
|
||
`LIBVA_DRIVER_NAME` driver).
|
||
|
||
---
|
||
|
||
## Adding another backend
|
||
|
||
1. Add `core/src/decode_<name>.rs` implementing `VideoDecoder` + `unsafe impl
|
||
Send`, with a `new() -> Result<Self>` that errors when the platform can't
|
||
support it.
|
||
2. Declare the module in `decode.rs` behind the right `#[cfg(...)]`.
|
||
3. Add a `#[cfg(...)]` branch in `best_decoder()` before the openh264 fallback.
|
||
4. Add the deps under a matching `[target.'cfg(...)'.dependencies]` in
|
||
`core/Cargo.toml`.
|
||
5. Emit the **true display size** (crop any codec padding) and BT.601 limited-
|
||
range RGB to match the other backends.
|