01 — Vanguard

Edge computer vision that has to justify every decision.

A real-time face-recognition attendance system that runs entirely on the hardware already on site — a basic computer or a small edge device — with no cloud and no internet. Four neural networks in a Kafka-chained pipeline, a FAISS vector index, and an operator console whose whole job is to show you why the model kept or threw away every single face it saw.

Models on device

Detection, face detection, head pose, recognition

Embedding dim

512

Face embeddings, FAISS cosine index

Python pipeline

51.7k

Lines — inference, tracking, clustering

Full system

79.5k

Lines across Python + NestJS + React

Role

AI / ML engineer — pipeline, model runtimes, clustering, plus the API and console around them

Year

2026

Stack

PythonOpenCVYOLOv8ByteTrackWHENetONNX RuntimeKafkaFAISSNestJSPostgreSQLReact + Vite

Services

ML engineering · Edge deployment · Backend · Operator tooling

The Vanguard Activity screen, showing the four-step funnel and per-face verdicts — **Activity — the funnel, then every verdict.** Four numbered counters read as a funnel: 12,905 photos taken → 3,632 kept (28.1%) → 3,025 known (83.3%) → 607 strangers (16.7%). Below them, every single face the camera took, newest first, each with three dots for the three gates and the one reading that decided it, drawn as a bar against its limit.

*The face crops in these screenshots are pixelated and the roster is renamed. The system captures real biometric data; the measurements, timings and counts shown are untouched.*

The hard part, and the thing worth showing

Cameras watch a workspace. An edge device — no internet required — detects every face, measures whether that face is good enough to trust, and only then matches it against an enrolled identity. The output is an attendance record nobody had to clock in for.

Face recognition demos are easy. Face recognition that runs 24/7 on modest on-site hardware, at night, on people walking past at an angle — that fails constantly. So the product is built around a confession:

Of 12,905 faces the camera found, only 3,632 were good enough to keep.

That 28% survival rate is not hidden; it is the main screen. Every rejected face shows the exact measurement that killed it and the exact limit it was measured against, and an operator who thinks the system is missing people can widen that limit and watch the numbers move. That feedback loop — model behaviour made legible and tunable by a non-engineer — is the actual engineering achievement.

The pipeline

Three OS processes, chained by Kafka topics, all writing to one Postgres. A face reaches an identity only by passing every gate, in order.

Stage 01

Detect

OpenCV pulls and decodes the RTSP stream; YOLOv8n finds faces in each frame. ByteTrack assigns a persistent track ID so one person walking through is one subject, not forty.

Stage 02

Quality gate

Landmark count, OpenCV Laplacian blur, face area, and head pose (yaw / pitch / roll via WHENet) each checked against a configured limit.

Most attrition happens here.

Stage 03

Recognise

Survivors are aligned, embedded to a 512-d face vector, and searched against a FAISS cosine index. Match or stranger.

Beside the chain

Consolidator

Incrementally clusters ungrouped faces and suggests an enrolled identity for a human to confirm — incremental on purpose, because a full re-cluster would renumber every group and invalidate labels operators had already read.

A single grouping module (FAISS cosine + temporal tiebreaker + union-find) is the one definition of “are these the same person?”, called by both the review queue and the consolidator, so suggestions can never contradict the stored grouping.

The other half of the loop

The Vanguard Thresholds screen with day and night quality limits and plain-language guidance — **Thresholds.** Every limit the Activity screen measures against is editable here — with plain-language guidance under each one telling the operator what raising or lowering it will do to accuracy and throughput. Separate day and night values. Saving writes the same YAML file the Python stages read.

This is what makes the explainability real rather than decorative. The console does not just report the model’s decisions — it is the control surface for them, and it speaks in consequences (“Increase: accepts faces that are more side-on; can reduce recognition accuracy”) rather than in hyperparameters.

The screens

The Vanguard Today screen showing per-person attendance — **Today.** The business outcome. Seen / not seen / stranger sightings across the top, then per-person first seen, last seen, present time, sessions and photo count. A gap over 30 minutes counts as away — one number the operator sets.

The Vanguard Records screen listing attendance across a date range — **Records.** The exportable attendance history across a date range — the artefact that actually leaves the system and lands in payroll. Present time is derived from sightings, never entered by hand.

The Vanguard Logs screen streaming per-frame pipeline timings — **Logs.** Every process in the chain streams here — recognition, quality checker, consolidator, supervisor, cleanup, the ML API. Per-frame stage timings, detection confidence against the threshold, and hardware FPS, live and greppable from the console.

The Vanguard Streams screen for pointing the pipeline at a camera — **Streams.** Point the pipeline at an RTSP URL, a device index or a video file, or search an NVR for its channels — then start and stop the pipeline from the same screen, with a health log underneath.

Engineering notes

Two model runtimes, one API

The same code runs on an edge device’s on-board accelerator and on a plain laptop via ONNX Runtime. It auto-selects by trying to import the accelerator SDK — but an explicit backend is never overridden, so asking for the accelerator without its SDK fails loudly. Silently falling back to CPU would look like working acceleration running orders of magnitude slower.
Numerics reproduced by hand. ONNX preprocessing reproduces the accelerator’s own image sequence step for step — including the recognition model’s x / 127.5 − 1.0 (scale-then-offset, not (x − 127.5) / 128) — so a threshold tuned on a laptop means the same thing on the deployed machine.

Reality, and its limits

A dwell gate, because reality. Somebody sitting at a desk used to be detected on every frame, filling the review queue with hundreds of near-identical crops. The detector now samples a bounded number of readable faces per visit, then goes quiet until they leave.
Honest about what it isn’t. The repository’s own README states plainly that there is no authentication, that /media serves biometric crops openly, and that it is only acceptable on a trusted, firewalled network. Stating the limits is part of the work.