NetDesk 1.5.0 — source release
This commit is contained in:
NetDesk committed
commit
21dea33089
227 files changed
+80714
No files matched your search
@@ -0,0 +1,223 @@
|
||||
# Cross-platform hardware H.264 decode
|
||||
|
||||
NetDesk's core decodes the host's H.264 (Annex-B) stream into packed-RGB8
|
||||
[`Frame`]s for the UI to paint. Every platform shares one trait and one output
|
||||
type, so the viewer never knows which backend is running:
|
||||
|
||||
```rust
|
||||
pub struct Frame { pub w: usize, pub h: usize, pub rgb: Vec<u8> } // w*h*3 bytes
|
||||
pub trait VideoDecoder { fn decode(&mut self, au: &[u8]) -> Result<Option<Frame>>; }
|
||||
pub fn best_decoder() -> Box<dyn VideoDecoder + Send>;
|
||||
```
|
||||
|
||||
`best_decoder()` selects the fastest decoder for the current OS at runtime and
|
||||
**always** falls back to the portable openh264 software decoder if the native
|
||||
hardware path cannot initialize (no GPU, missing runtime, headless box, etc.).
|
||||
|
||||
| Target | Backend | Module | API | Crate(s) |
|
||||
|---------------|----------------|---------------------------|-----------------------------|---------------------------------|
|
||||
| Windows | Media Foundation (DXVA2) | `core/src/decode_mf.rs` | `IMFTransform` (MSH264 MFT) | `windows` |
|
||||
| macOS / iOS | VideoToolbox | `core/src/decode_vt.rs` | `VTDecompressionSession` | `core-foundation(-sys)` + frameworks |
|
||||
| Android | MediaCodec | `core/src/decode_mediacodec.rs` | `AMediaCodec` (NDK) | `ndk-sys` / `libmediandk` |
|
||||
| Linux | VAAPI | `core/src/decode_vaapi.rs` | FFmpeg/libavcodec + VAAPI | `ffmpeg-next` |
|
||||
| All (fallback)| openh264 | `core/src/decode.rs` | software | `openh264` |
|
||||
|
||||
Each native module is `#[cfg]`-gated to its own OS, so building for one target
|
||||
never compiles (or even links) another target's module or its dependencies. The
|
||||
per-platform crates are declared under `[target.'cfg(...)'.dependencies]` in
|
||||
`core/Cargo.toml`; on a non-matching host they are only added to `Cargo.lock`
|
||||
(resolved), never built, and their build scripts never run.
|
||||
|
||||
All backends emit the **true display dimensions** (e.g. 1920×1080), cropping out
|
||||
any codec padding / stride / alignment, so frames are never stretched.
|
||||
|
||||
---
|
||||
|
||||
## Windows — Media Foundation (reference, builds here)
|
||||
|
||||
Already shipping. Build with the portable MinGW toolchain:
|
||||
|
||||
```bash
|
||||
export PATH="/c/mingw-netdesk/mingw64/bin:$HOME/.cargo/bin:$PATH"
|
||||
cargo build -p netdesk-core # or: --target x86_64-pc-windows-gnu
|
||||
```
|
||||
|
||||
### Dimension fix (2026-10)
|
||||
|
||||
The MS H.264 decoder MFT reports the **coded** size in `MF_MT_FRAME_SIZE`, which
|
||||
is padded up to the codec's macroblock alignment (1080p → a 1088+ coded height;
|
||||
other resolutions pad larger — this is what produced the wrong `1920x18xx` in the
|
||||
stats). The real picture rectangle is carried separately in a video-aperture
|
||||
blob. `decode_mf.rs` now:
|
||||
|
||||
1. Reads `MF_MT_FRAME_SIZE` into `coded_w`/`coded_h` — used **only** for the NV12
|
||||
plane layout (stride × plane height, and the UV plane offset).
|
||||
2. Reads the display rectangle from, in priority order,
|
||||
`MF_MT_MINIMUM_DISPLAY_APERTURE` → `MF_MT_GEOMETRIC_APERTURE` →
|
||||
`MF_MT_PAN_SCAN_APERTURE` (the last only when `MF_MT_PAN_SCAN_ENABLED`), as an
|
||||
`MFVideoArea` blob giving offset `(x, y)` and visible `width`×`height`. The
|
||||
aperture is clamped inside the coded frame so a malformed blob can't over-read.
|
||||
3. `nv12_to_rgb(..)` copies only the visible `width`×`height` region at
|
||||
`(crop_x, crop_y)` out of the padded NV12 (system-memory and DXGI/D3D11
|
||||
staging paths both), so `Frame.w`/`Frame.h` are the true `1920×1080`.
|
||||
|
||||
When no aperture is present the whole coded frame is treated as the picture
|
||||
(unchanged behaviour).
|
||||
|
||||
---
|
||||
|
||||
## macOS / iOS — VideoToolbox
|
||||
|
||||
`decode_vt.rs`, gated `#[cfg(any(target_os = "macos", target_os = "ios"))]`.
|
||||
|
||||
Feeds AUs into a `VTDecompressionSession` on the platform video engine and
|
||||
requests 32-bit BGRA output (VideoToolbox does YUV→RGB on the GPU), which is
|
||||
repacked to RGB8. The session is built lazily from the first keyframe's SPS/PPS
|
||||
(`CMVideoFormatDescriptionCreateFromH264ParameterSets`); Annex-B is repacked to
|
||||
AVCC for the `CMSampleBuffer`. `CVPixelBufferGetWidth/Height` already return the
|
||||
cropped display size, so no manual crop is needed.
|
||||
|
||||
Links the system **CoreMedia**, **CoreVideo** and **VideoToolbox** frameworks
|
||||
directly; `core-foundation` is used for the pixel-buffer attribute dictionary.
|
||||
|
||||
### Build (requires macOS + Xcode)
|
||||
|
||||
```bash
|
||||
xcode-select --install # or a full Xcode install
|
||||
rustup target add aarch64-apple-darwin x86_64-apple-darwin # macOS
|
||||
rustup target add aarch64-apple-ios aarch64-apple-ios-sim # iOS
|
||||
|
||||
cargo build -p netdesk-core --target aarch64-apple-darwin # Apple-silicon mac
|
||||
cargo build -p netdesk-core --target aarch64-apple-ios # device
|
||||
cargo build -p netdesk-core --target aarch64-apple-ios-sim # simulator
|
||||
```
|
||||
|
||||
The framework `#[link]`s resolve against the SDK the active toolchain selects
|
||||
(`xcrun --sdk macosx|iphoneos|iphonesimulator --show-sdk-path`). No extra flags.
|
||||
|
||||
### Test
|
||||
|
||||
```bash
|
||||
cargo test -p netdesk-core --target aarch64-apple-darwin
|
||||
```
|
||||
|
||||
Smoke-test end to end by running the viewer against a live host; the log line
|
||||
`decode: VideoToolbox H.264 (hardware)` confirms the path was selected.
|
||||
|
||||
---
|
||||
|
||||
## Android — MediaCodec
|
||||
|
||||
`decode_mediacodec.rs`, gated `#[cfg(target_os = "android")]`.
|
||||
|
||||
Drives a byte-buffer-mode `AMediaCodec` AVC decoder (the SoC hardware decoder via
|
||||
OMX/Codec2). The codec is configured lazily from the first keyframe's SPS/PPS as
|
||||
`csd-0`/`csd-1`; each `decode` queues the Annex-B AU as one input buffer and
|
||||
dequeues one output buffer. Output is YUV 4:2:0 — NV12 semi-planar or I420
|
||||
planar depending on the device — converted to RGB8 honouring the format's
|
||||
`stride`, `slice-height`, `color-format`, and `crop-left/top/right/bottom` keys,
|
||||
so the emitted frame is the true display size.
|
||||
|
||||
Talks to the stable NDK Media C ABI in **`libmediandk`** directly (the same ABI
|
||||
the `ndk-sys` crate binds), so it is version-independent.
|
||||
|
||||
### Build (requires the Android NDK)
|
||||
|
||||
Install the NDK (via Android Studio's SDK Manager or `sdkmanager "ndk;27.0.x"`),
|
||||
then use [`cargo-ndk`]:
|
||||
|
||||
```bash
|
||||
rustup target add aarch64-linux-android armv7-linux-androideabi \
|
||||
x86_64-linux-android i686-linux-android
|
||||
cargo install cargo-ndk
|
||||
export ANDROID_NDK_HOME=$HOME/Android/Sdk/ndk/27.0.12077973
|
||||
|
||||
# API 24+; links libmediandk / liblog automatically via the NDK sysroot.
|
||||
cargo ndk -t arm64-v8a -t armeabi-v7a -p 24 build -p netdesk-core --release
|
||||
```
|
||||
|
||||
Or without cargo-ndk, point Cargo at the NDK clang linker for each target, e.g.:
|
||||
|
||||
```bash
|
||||
export CC_aarch64_linux_android=$ANDROID_NDK_HOME/.../aarch64-linux-android24-clang
|
||||
export CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=$CC_aarch64_linux_android
|
||||
cargo build -p netdesk-core --target aarch64-linux-android --release
|
||||
```
|
||||
|
||||
### Test
|
||||
|
||||
Run on a device/emulator (the `mediandk` APIs are not present on the host):
|
||||
|
||||
```bash
|
||||
cargo ndk -t arm64-v8a build -p netdesk-core --tests
|
||||
# push the test binary with `adb push` and run it on-device, or exercise it
|
||||
# through the app's instrumented tests.
|
||||
```
|
||||
|
||||
The log line `decode: MediaCodec H.264 (hardware)` confirms selection.
|
||||
|
||||
---
|
||||
|
||||
## Linux — VAAPI (via FFmpeg)
|
||||
|
||||
`decode_vaapi.rs`, gated `#[cfg(target_os = "linux")]`.
|
||||
|
||||
Opens the libavcodec H.264 decoder with a VAAPI `hw_device_ctx` (default DRI
|
||||
render node). libavcodec does all bitstream parsing; decode runs on the GPU, the
|
||||
surface is pulled to system memory with `av_hwframe_transfer_data` (NV12),
|
||||
cropped to the display rectangle with `av_frame_apply_cropping`, and converted to
|
||||
RGB8 with swscale. If no VAAPI device is usable, `new()` errors and the core
|
||||
falls back to openh264, so GPU-less / headless boxes still work.
|
||||
|
||||
Pulls libav* in through the **`ffmpeg-next`** crate and needs the system FFmpeg
|
||||
(≥ 4) development libraries and a VAAPI driver at build/run time.
|
||||
|
||||
### Build (requires system FFmpeg + VAAPI dev packages)
|
||||
|
||||
Debian/Ubuntu:
|
||||
|
||||
```bash
|
||||
sudo apt install -y clang pkg-config \
|
||||
libavcodec-dev libavutil-dev libswscale-dev libavformat-dev \
|
||||
libva-dev vainfo intel-media-va-driver # or mesa-va-drivers for AMD
|
||||
|
||||
cargo build -p netdesk-core # host target
|
||||
```
|
||||
|
||||
Fedora:
|
||||
|
||||
```bash
|
||||
sudo dnf install clang pkgconf-pkg-config \
|
||||
ffmpeg-devel libva-devel libva-utils mesa-va-drivers
|
||||
cargo build -p netdesk-core
|
||||
```
|
||||
|
||||
`ffmpeg-sys-next` locates the libraries with `pkg-config`; set
|
||||
`PKG_CONFIG_PATH` if FFmpeg is in a non-standard prefix. `bindgen` needs
|
||||
`clang`/`libclang` (the `clang` package above).
|
||||
|
||||
### Test
|
||||
|
||||
```bash
|
||||
vainfo # confirm a VAAPI driver + H264 profile
|
||||
cargo test -p netdesk-core
|
||||
```
|
||||
|
||||
At runtime the log line `decode: VAAPI H.264 (FFmpeg hardware)` confirms the
|
||||
hardware path; `VAAPI init failed (...); using openh264 software decode` means it
|
||||
fell back (check `vainfo`, `/dev/dri/renderD128` permissions, and the
|
||||
`LIBVA_DRIVER_NAME` driver).
|
||||
|
||||
---
|
||||
|
||||
## Adding another backend
|
||||
|
||||
1. Add `core/src/decode_<name>.rs` implementing `VideoDecoder` + `unsafe impl
|
||||
Send`, with a `new() -> Result<Self>` that errors when the platform can't
|
||||
support it.
|
||||
2. Declare the module in `decode.rs` behind the right `#[cfg(...)]`.
|
||||
3. Add a `#[cfg(...)]` branch in `best_decoder()` before the openh264 fallback.
|
||||
4. Add the deps under a matching `[target.'cfg(...)'.dependencies]` in
|
||||
`core/Cargo.toml`.
|
||||
5. Emit the **true display size** (crop any codec padding) and BT.601 limited-
|
||||
range RGB to match the other backends.
|
||||
Reference in new issue
Block a user