NEXUS
A voice-first operating system assistant built in Rust and Tauri, executing code navigation, system automation, and codebase architecture reviews through local speech models.
Bringing Ambient Intelligence to the OS
The team started NEXUS to explore a developer assistant that lives directly inside your workstation rather than inside a browser tab. We wanted an omnipresent, voice-first companion capable of inspecting codebases, executing system tasks, and responding to natural voice commands without forcing context switches. While my teammates developed the initial Windows prototype and Cloudflare Workers edge backend, I took ownership of bringing NEXUS to Linux and engineering its desktop perception systems. Modern Linux environments presented unique low-level hurdles across display protocols, audio daemons, and system security boundaries that required dedicated systems programming in Rust and Tauri.
Taming Wayland and Floating Overlays
Creating an omnipresent interface on Linux is difficult because modern compositors like Wayland intentionally isolate application windows for security. Window managers prevent arbitrary overlays from floating click-through or capturing global keyboard events. I re-architected the stage overlay in Rust and Tauri v2, configuring it as a non-activating desktop surface with custom input event passthrough. This allowed the glowing audio-reactive aura and captions to float above active IDEs without stealing keyboard focus or breaking the developer's typing flow. I also resolved WebKitGTK DOM suspension bugs that previously caused floating webviews to stall whenever they were hidden or minimized.
Perception Without Screen Scraping
Most desktop AI agents rely on periodic screen captures processed by visual language models. This approach is computationally heavy, consumes excessive battery, and struggles with small font sizes. I designed a two-tier semantic perception engine to read application state directly from OS contracts. For native GTK and Qt software, I built an AT-SPI client in Rust that traverses the accessibility tree to identify the focused window, text inputs, and actionable buttons. For Electron and Chromium browsers, I built a client that connects directly to Chrome DevTools Protocol debugging sockets. NEXUS perceives open editor tabs and terminal buffers semantically within milliseconds.
Rebuilding the Audio Pipeline
Running local neural wake-word detection alongside speech recognition requires an uninterrupted audio stream. On Linux, contending audio daemons like PipeWire, PulseAudio, and raw ALSA devices frequently lock inputs or hang when probed sequentially. I rewrote the audio ingestion pipeline using rodio and cpal. Mic discovery probes PipeWire paths first behind hard deadlines, preventing hung ALSA devices from freezing the wake loop. By streaming PCM chunks into memory and optimizing the openWakeWord ONNX runtime, I reduced wake-word inference from 106 ms down to 8.4 ms per chunk, dropping idle CPU consumption from 119% down to 19% on developer laptops.
Native Automation via D-Bus
To execute voice commands reliably, NEXUS needed clean system hooks rather than fragile shell commands. I integrated Linux desktop controls using D-Bus interfaces through Rust's zbus crate. I wired native MPRIS media handling so NEXUS automatically pauses or ducks Spotify playback when speech is detected, restoring volume once the assistant finishes speaking. I also implemented application discovery by parsing standard XDG desktop entry files across system paths, and connected native desktop notifications. Every automation path reports explicit execution status instead of failing silently, ensuring voice feedback remains truthful.
What Linux Systems Programming Taught Me
Porting NEXUS to Linux taught me that cross-platform software cannot treat the operating system as an afterthought. Every platform has distinct conventions for audio routing, accessibility trees, process isolation, and display servers. Working on low-level Rust clients and packaging Debian packages forced me to debug inter-process race conditions, profile CPU spikes with tracing tools, and handle hardware quirks gracefully. It fundamentally changed how I approach systems architecture: understanding the platform's native primitives is always superior to piling layers of abstraction on top of an ill-fitting foundation.