Whitepaper · SDK API 14

Perception and media on a sealed memory budget

1The problem

Robots, drones, inspection rigs and players need the same things from a video stream: pictures on time, sound in sync, and an understanding of what moves. The usual answer is a stack of libraries, each with its own allocator, threads and copies. Its timing is the sum of everyone's worst case.

AVION owns the whole path from file or sensor to screen, speaker and host callback, under four rules:

• No third-party code. C++20, Win32, COM, Direct3D 12, DXGI, DirectComposition, Media Foundation transforms and WASAPI only.
• No heap allocation in steady state. Arenas are committed in Preparing and sealed before Primed.
• No locks on real-time paths. Stages exchange 64-byte descriptors through single-producer, single-consumer rings.
• GPU first. Decoded pictures stay in video memory; conversion, scaling, tone mapping and most perception read them there.

2Architecture

Budgets, known up front. Core 192 MiB against 177.0 MiB measured on every fixture: 4K HEVC, 7.1 audio, a 3.5-hour file. Modules 48.4 MiB and up to 5 threads.

Preparing commits every arena and creates the GPU and audio endpoints. Primed seals memory and fills the first rings. Running is the lock-free steady state. A seek changes the playback epoch; device loss drains back to Preparing without releasing the pools.

Each optional module (voice, HUD, subtitles, chapters, telemetry and perception) is one fixed region of the same arena with a known thread count. A module that would need more is disabled for that file instead of growing. The host learns every size before creating anything and provides the engine's storage, so the SDK never calls malloc.

When audio plays, the audio device is the master clock: PredictiveMasterClock runs a second-order alpha–beta estimator (in effect a phase-locked loop) on WASAPI's device position. Presentation waits on hardware events and a high-resolution timer instead of spinning.

3Perception

• Virtual IMU. The background's translation and rotation between frames from a global SAD search and per-tile Lucas–Kanade, objects left out, robust fit. Within 0.068 px rms of the true shake.
• Stabiliser. A 2nd-order Butterworth low-pass of the accumulated rotation (0.35 s delay), solved in closed form per frame interval. On screen: 14.73 → 0.92 px rms.
• Flow, time to contact, ground. Tile flow by a predicted diamond search; affine fits over 5×5-tile windows; an approach must be an isotropic expansion; TTC = 2 / div; the ground plane's expansion is modelled out.
• Objects. Up to 4 tracked objects with stable ids, crossing-safe.
• Remote vibrometry. Y′, Cb, Cr and sub-pixel displacement transformed at the frames' real timestamps: 72.00 BPM through camera shake.
• Event camera. A stabilised DVS emulation; a still scene under rolling shake fires 141 events against 1,201,527.
• Bitstream perception. H.264 and HEVC slice headers to QP, activity and cut candidates, without decoding.
• Tensors. NCHW float32 or float16, also while paused; HDR exported as displayed.

Hosts take the pictures as shared Direct3D textures and fences (verified with Direct3D 11 and Direct3D 12 hosts), in 8-bit, 10-bit PQ BT.2020 or FP16 scRGB: an HDR10 file reached a host at up to 5,849 cd/m². In host mode, where the host's thread drives the engine, H.264 and HEVC are decoded on the GPU into those textures, with 0.7 % of one core for a 1080p-class H.264 file and 6.8 % for 4K HDR HEVC, and no picture leaving the GPU. Python (a wheel) and .NET (a NuGet package) bind the same C99 SDK.

4Verification

Intermittency is investigated. A 4K60 HDR cold-start drop (about 4 % of starts) was traced to the quality governor's expensive tier on a thin queue; after a rule change, 0 of 200 starts dropped.

Every claim is the output of an automated test that ships with the source. Test media is generated with known ground truth (shaken clips with a known shake, pulse clips with a known rate); thirteen tests run on real footage. Allocation probes assert zero allocations while playing, stabilising, tracking and exporting.

5Head to head

One counter for everyone. tools/bench/malloc_count.c, preloaded into each process, counts every malloc, calloc, realloc and aligned allocation. ffmpeg's steady-state rate is the difference between a 1,000- and a 3,000-frame run.

All runs used one machine: an Intel Core i7-10700 with UHD 630 graphics under Windows 11, and WSL2 (6.6.114.1) on the same CPU for the Linux rows. The inputs were LG's 4K60 HDR reel (HEVC Main10) and a 720p59.94 H.264 film, first 3,000 frames each, plus 600 frames of the film at 640×480 grey, read into memory by every program.

Reading the table fairly

• The scan rows compare reading slice headers with decoding every pixel. That gap is the reason pre-decode inspection exists; it is not a decoder race. With all threads ffmpeg spends 9.45 ms of wall time per 4K frame; with D3D11 hardware decoding on Windows, 30.75 ms.
• AVION's motion step computes flow on 8×8-cell tiles plus time to contact, saliency and objects; DIS and Farnebäck produce dense per-pixel flow. The comparison is of the cost of a motion estimate fit for each job.
• The OpenCV program reuses every output, as a careful user would; the allocations it shows are OpenCV's own internal buffers.
• Cut accuracy, scored against scdet within 4 frames: on LG's reel 3 of its 3 cuts found, 1 extra (a still shot, checked by eye); on the 720p film 7 of 8, 1 extra. On 2026-11-02 the inspector confirmed 252 cuts on LG's reel. Its hierarchical GOP (an anchor every 4 frames, predicted from 4 back, costing 3-5× the others) was read as bursts against one mean for every B frame, and its keyframes, on a fixed 248-frame clock through fades, as cuts. Now each prediction class keeps its own mean, a frame narrower than its anchor counts only if the anchor rose too, and a keyframe on the encoder's clock scores at most 0.4. Two files are a sanity check, not a recall figure.
• Verification: Release 236/237 passed; failed: media_fixture_mkv_av1_4k60_test; Debug 236/237 passed; failed: media_host_tracks_tests; GCC 102/102 passed.
• Playing CPU, per thread: the presentation thread spins the last 0.45 ms before each picture's deadline on the high-resolution waitable timer (1.2 ms on 2026-11-02). About a quarter of Presents then come 0.1 ms after the deadline on average, with no picture dropped; a host that wants every Present inside it sets 1.2 ms (pace_spin_us). Paused, both engine threads now sleep while nothing moves (0.16 %, against 1.17 % before).

6Limits

• Stabilisation corrects one translation and one rotation per frame: no rolling shutter, no parallax.
• The bitstream inspector sees costs, not pixels: a cut exactly on a scheduled keyframe can be missed, and a long burst of new detail that stops at once can read as a cut (section 5).
• MP4 with H.264 or HEVC and AAC or PCM. AV1 is refused; the platform AAC decoder takes at most 6 channels.
• Windows is the fully exercised platform; host mode passes on Linux (its hardware decoding is Windows-only); Android and Apple toolchains have not been run.
• In the 10-bit and FP16 texture formats the windowed engine draws the picture only: no captions, controls or snapshots.
• An object hidden behind another for more than 0.5 s loses its track.

7Evaluating AVION 0.1

The evaluation bundle holds the SDK header and reference, the Release demo, static libraries, the Python wheel and the NuGet package, a sample clip, the CTest log of this build's Release run (236/237 passed), and the head-to-head's harness, raw results and summary. It is licensed under the AVION Evaluation & Commercial SDK License 1.1: free for evaluation, benchmarking, research and prototyping, and for use in non-commercial, academic and personal projects, which may also redistribute the unmodified binaries (section 1A); commercial production and commercial redistribution need a commercial licence. Its SHA-256, for integrity (not a signature):

AVION-0.1-Evaluation.zip · 15,267,912 bytes
78d8143b8766e2a66274bee5189646a34a11cb2d4d1acbbf1e1a3ec8e472b3da