droidclaw/docs/plans/2026-02-20-voice-overlay-design.md

# Voice Overlay — Design Document

**Date:** 2026-02-20
**Status:** Approved
**Approach:** Stream audio over existing WebSocket (Approach A)

## Overview

Add a voice-activated overlay to DroidClaw's Android app. User taps the floating pill → full-screen glowing gradient border appears → speech is streamed to the server for real-time transcription → live text appears on screen → tap Send to execute as a goal.

## User Flow

```
[IDLE] → tap pill → [LISTENING] → tap send → [EXECUTING] → done → [IDLE]
                          ↓
                     tap cancel
                          ↓
                       [IDLE]
```

### States

**IDLE** — Existing floating pill: `● Ready`, draggable, tappable.

**LISTENING** — Pill disappears. Full-screen overlay:
- Animated gradient border around all 4 screen edges (purple → blue → cyan → green cycle, ~3s)
- Large transcribed text in center, updating live word-by-word
- Bottom: `Send` (primary) + `Cancel` (secondary) buttons
- Audio recording starts immediately on transition

**EXECUTING** — Overlay collapses back to pill. Pill shows agent progress as today.

**IDLE (post-completion)** — Pill shows `● Done` for 3s, then `● Ready`.

## Audio Streaming Protocol

### Android → Server

| Message | Description |
|---------|-------------|
| `{type: "voice_start"}` | Recording begun |
| `{type: "voice_chunk", data: "<base64>"}` | ~100ms PCM chunks, 16kHz mono 16-bit |
| `{type: "voice_stop", action: "send"}` | User tapped Send — finalize & execute goal |
| `{type: "voice_stop", action: "cancel"}` | User tapped Cancel — discard |

### Server → Android

| Message | Description |
|---------|-------------|
| `{type: "transcript_partial", text: "..."}` | Live streaming partial transcript |
| `{type: "transcript_final", text: "..."}` | Final complete transcript |

### Flow

1. Android sends `voice_start` → server opens streaming connection to Groq Whisper
2. Android streams `voice_chunk` every ~100ms → server pipes PCM to Groq
3. Groq sends partial transcriptions → server relays as `transcript_partial`
4. User taps Send → Android sends `voice_stop` with `action: "send"`
5. Server flushes final audio → gets `transcript_final` → sends to Android → fires goal into agent loop
6. Cancel: `voice_stop` with `action: "cancel"` → server discards Groq session, no goal

### Audio Format

- Sample rate: 16kHz
- Channels: mono
- Bit depth: 16-bit PCM (linear16)
- Bandwidth: ~32KB/sec
- Encoding for WebSocket: base64 text frames

## Full-Screen Gradient Overlay

Two separate overlay layers managed by `AgentOverlay`:

### Layer 1 — Gradient Border (non-interactive)

- `TYPE_APPLICATION_OVERLAY` with `FLAG_NOT_TOUCHABLE | FLAG_NOT_FOCUSABLE`
- `MATCH_PARENT` — covers entire screen
- Compose renders animated gradient strips (~6dp) along all 4 edges
- Colors: purple → blue → cyan → green → purple, infinite rotation ~3s cycle
- Implementation: `drawBehind` modifier with 4 `LinearGradient` brushes, animated offset via `rememberInfiniteTransition`
- Center is fully transparent — pass-through to apps behind

### Layer 2 — Text + Buttons (interactive)

- `TYPE_APPLICATION_OVERLAY` with `FLAG_NOT_FOCUSABLE` (tappable, no keyboard steal)
- Positioned at bottom ~40% of screen
- Semi-transparent dark background `Color(0xCC000000)`
- Contents:
  - Transcribed text: 24-28sp, white, center-aligned, auto-scrolls
  - Subtle pulse/waveform animation while listening
  - Bottom row: `Send` button (accent) + `Cancel` button (muted)

### Why Two Layers

Android overlays cannot be partially touchable. The gradient border must be `FLAG_NOT_TOUCHABLE` (pass-through) while the text/button area must be tappable. Separate `WindowManager` views with different flags solve this.

## Server-Side STT Handler

New file: `src/voice.ts`

### Responsibilities

- On `voice_start`: open Groq Whisper streaming connection
- On `voice_chunk`: pipe decoded PCM to Groq stream
- On `voice_stop` (send): flush stream, get final transcript, trigger `runAgent()` with transcript as goal
- On `voice_stop` (cancel): close Groq stream, discard

### Fallback

If Groq streaming is unavailable, buffer all chunks server-side. On `voice_stop`, send complete audio as single Whisper API call. No live words — final text appears all at once. Always works.

### Goal Execution

After `transcript_final`, call existing `runAgent()` from `kernel.ts` — identical to web dashboard goals. No changes to agent loop.

## Files Changed

| File | Change | Scope |
|------|--------|-------|
| `android/.../AndroidManifest.xml` | Add `RECORD_AUDIO` permission | Minor |
| `android/.../overlay/AgentOverlay.kt` | State machine: idle/listening/executing, manage 2 overlay layers | Major |
| `android/.../overlay/OverlayContent.kt` | New composables: `GradientBorder`, `VoiceOverlayContent`, `LiveTranscriptText` | Major |
| `android/.../overlay/VoiceRecorder.kt` | **New file.** `AudioRecord` capture + chunked base64 streaming | New |
| `android/.../connection/ConnectionService.kt` | Handle voice messages, route transcript events to overlay | Medium |
| `android/.../model/Protocol.kt` | New message data classes for voice protocol | Minor |
| `src/voice.ts` | **New file.** Groq Whisper streaming STT handler | New |
| `src/kernel.ts` | Route voice WebSocket messages to `voice.ts` | Minor |

### Untouched

`actions.ts`, `skills.ts`, `workflow.ts`, `sanitizer.ts`, `llm-providers.ts`, `config.ts`, `constants.ts`

## Permissions

- `RECORD_AUDIO` — new runtime permission, requested on first voice activation
- `SYSTEM_ALERT_WINDOW` — already granted (existing overlay)
- `INTERNET` — already granted

## Difficulty Assessment

**Overall: Medium.** Estimated 3-4 days.

- Android `AudioRecord` → WebSocket streaming: well-documented, straightforward
- Full-screen gradient overlay animation: standard Compose `Canvas` + `rememberInfiniteTransition`
- Groq Whisper streaming API: documented, Bun handles WebSocket/HTTP streaming natively
- Two-layer overlay management: minor complexity in `AgentOverlay` state machine
- No risky unknowns — all components have clear precedents