Omni-Assist - Autonomous Wheelchair Attachment — yHack 1st Place, Hardware + Viam Tracks

Overview

Omni-Assist is a voice-controlled autonomous navigation attachment that bolts onto any standard wheelchair, with an SO-101 robot arm for reaching and grabbing. The user speaks ("take me to the door"), the request is resolved to a navigation target — in under 500 ms for cached waypoints, with an LLM fallback for anything new — and an A* planner with DWA + potential-field local control drives the chair there on four mecanum wheels while a 4-layer spatial memory learns the space across trips. Engineered in under 24 hours at yHack (700+ participants) and awarded 1st place in both the Hardware and Viam tracks.

How It Works

The system runs as three concurrent loops on a laptop tethered to an Arduino Mega over USB serial.

The voice loop captures speech through ElevenLabs STT (with Whisper as a fallback) and turns it into a navigation target. Requested objects ("the table", "the window") are matched with YOLOv8 against an ORB landmark map on a 5 cm grid, so a cached waypoint resolves in under 500 ms; anything the map doesn't know falls back to an Anthropic LLM that returns a structured goto command, and responses are spoken back through ElevenLabs TTS. The planning loop replans at 10–20 Hz: it runs A* over a blended costmap pulled from a 4-layer spatial memory (static occupancy from LiDAR raycasting, the ORB-feature landmark database, dynamic obstacle priors learned across trips, and a learned traversal cost map), then feeds the global path through a DWA + potential-field local planner and a jerk-limited velocity smoother. The motor loop on the Mega runs a 100 Hz encoder PID loop: it takes vx/vy/ω velocity commands as line-delimited JSON over serial at 115200 baud, applies mecanum inverse kinematics (FL = vx − vy − ω, etc.), and outputs PWM through four IBT-2 H-bridge drivers. Camera, LiDAR, IMU, planning, and motor control are all integrated on real hardware, with sub-100 ms scan-to-motor latency end to end.

The attachment clamps onto the wheelchair frame and connects to the wheels through a friction-drive mechanism, so the chair can still be used manually at any time.

Motor Control

The Arduino Mega 2560 runs the inner control loop at 100 Hz, closing a PID loop on each wheel's quadrature encoder. It receives line-delimited JSON velocity commands (vx, vy, ω) over USB serial at 115200 baud, applies mecanum inverse kinematics to compute four wheel speeds, and outputs PWM through four IBT-2 H-bridge drivers (one per motor). End to end, a LiDAR scan reaches the motors in under 100 ms. A 250 ms serial watchdog stops the motors if the laptop goes silent, and an e-stop command over the same link forces instant zero velocity. Try it below — use WASD to drive and E/F to rotate. The code updates live with the actual IK math running on the Mega.

// Mecanum Inverse Kinematics — Arduino Mega
 
float vx = 0.00, vy = 0.00, w = 0.00;
 
int FL = (vx + vy - w) * 210;
int FR = (vx - vy + w) * 210;
int RL = (vx - vy - w) * 210;
int RR = (vx + vy + w) * 210;
 
// → FL=0  FR=0  RL=0  RR=0

Results

The prototype successfully navigated an indoor course at yHack, taking voice commands and reaching named waypoints while avoiding static and dynamic obstacles in real time. Out of 700+ participants, judges awarded it 1st place in both the Hardware and Viam tracks for the combination of practical impact, technical execution, and the fact that it augments existing wheelchairs rather than replacing them. The biggest open gap was pose estimation — the demo ran on dead-reckoning with LiDAR scan-matching as a guard, and proper EKF fusion of LiDAR, encoders, and visual odometry is the next milestone.