← all posts site ↗

rgb to depth for a live webcam

the idea is a custom plugin that fuses depth, person mask, and landmarks so a virtual face or character can sit correctly in a webcam scene. first the model landscape, verified against repos and papers. depth anything v2 small is the starting point: twenty five million parameters, apache license, mature onnx and tensorrt paths, and community-measured inference in the tens of milliseconds on a mid gpu. depth anything 3 is the current one, adds confidence plus camera intrinsics and pose, with the small and base weights still apache. moge is the metric geometry option that recovers scale without knowing the camera intrinsics, giving point maps, normals, and even field of view. depth pro is sharp and metric but offline territory, and marigold is diffusion-based, which is never a real-time loop.

the licensing trap is everywhere. lots of the state-of-the-art weights are non-commercial: depth anything base and up, the bigger depth anything 3 variants, unidepth. for anything distributable, apache and mit picks are what matter, which is exactly why v2 small and moGe keep winning.

then the honest limits, straight from the readmes: monocular depth flickers between frames, knows nothing about hidden surfaces, and bleeds at person to background edges. it orders layers but it does not know what a person is. so the plugin pattern is segmentation for the silhouette, landmarks for attachment, depth for occlusion order, and a temporal smoother so the whole thing doesn't shimmer. there's already a browser project, depthfield, that does depth anything v2 small plus mediapipe landmarks with depth sampled at each joint — basically the first prototype i would have built anyway. the models exist; the actual product is the fusion layer.