c395f9d83e
- Add device/session/step DB persistence in server agent loop - Add goal preprocessor for compound goals (e.g., "open YouTube and search X") - Add step-level logging to agent loop - Fix dashboard WebSocket auth (direct DB token lookup instead of auth.api) - Fix web layout to use locals.session.token instead of cookie - Add dashboard-ws.svelte.ts WebSocket store with auto-reconnect - Rewrite devices page with direct DB queries and real-time updates - Add device detail page with live step display and session history - Add Android companion app resources, themes, and screen capture consent Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
86 KiB
86 KiB
DroidClaw Android App Implementation Plan
For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
Goal: Build the DroidClaw Android companion app — a Jetpack Compose app that connects to the Hono server via WebSocket, captures accessibility trees and screenshots, executes gestures, and lets users submit goals from the phone.
Architecture: Three-layer architecture (Accessibility → Connection → UI). The AccessibilityService captures screen trees and executes gestures. A foreground ConnectionService manages the Ktor WebSocket. Compose UI with bottom nav (Home/Settings/Logs) observes service state via companion-object StateFlows.
Tech Stack: Kotlin, Jetpack Compose, Ktor Client WebSocket, kotlinx.serialization, DataStore Preferences, MediaProjection API, AccessibilityService API.
Existing Project State
The Android project is a fresh Compose scaffold:
android/app/build.gradle.kts— Compose app with AGP 9.0.1, Kotlin 2.0.21, compileSdk 36, minSdk 24android/gradle/libs.versions.toml— version catalog with basic Compose + lifecycle depsandroid/app/src/main/java/com/thisux/droidclaw/MainActivity.kt— Hello World Compose activityandroid/app/src/main/java/com/thisux/droidclaw/ui/theme/— Default Material 3 theme (Color, Type, Theme)- Shared TypeScript types in
packages/shared/src/types.tsandpackages/shared/src/protocol.tsdefine the data models and WebSocket protocol the Android app must mirror.
Task 1: Add Dependencies & Build Config
Files:
- Modify:
android/gradle/libs.versions.toml - Modify:
android/build.gradle.kts(root) - Modify:
android/app/build.gradle.kts
Step 1: Add version catalog entries
Add to android/gradle/libs.versions.toml:
[versions]
agp = "9.0.1"
coreKtx = "1.10.1"
junit = "4.13.2"
junitVersion = "1.1.5"
espressoCore = "3.5.1"
lifecycleRuntimeKtx = "2.6.1"
activityCompose = "1.8.0"
kotlin = "2.0.21"
composeBom = "2024.09.00"
ktor = "3.1.1"
kotlinxSerialization = "1.7.3"
kotlinxCoroutines = "1.9.0"
datastore = "1.1.1"
lifecycleService = "2.8.7"
navigationCompose = "2.8.5"
composeIconsExtended = "1.7.6"
[libraries]
# ... keep existing entries ...
ktor-client-cio = { group = "io.ktor", name = "ktor-client-cio", version.ref = "ktor" }
ktor-client-websockets = { group = "io.ktor", name = "ktor-client-websockets", version.ref = "ktor" }
ktor-client-content-negotiation = { group = "io.ktor", name = "ktor-client-content-negotiation", version.ref = "ktor" }
ktor-serialization-kotlinx-json = { group = "io.ktor", name = "ktor-serialization-kotlinx-json", version.ref = "ktor" }
kotlinx-serialization-json = { group = "org.jetbrains.kotlinx", name = "kotlinx-serialization-json", version.ref = "kotlinxSerialization" }
kotlinx-coroutines-android = { group = "org.jetbrains.kotlinx", name = "kotlinx-coroutines-android", version.ref = "kotlinxCoroutines" }
datastore-preferences = { group = "androidx.datastore", name = "datastore-preferences", version.ref = "datastore" }
lifecycle-service = { group = "androidx.lifecycle", name = "lifecycle-service", version.ref = "lifecycleService" }
navigation-compose = { group = "androidx.navigation", name = "navigation-compose", version.ref = "navigationCompose" }
compose-icons-extended = { group = "androidx.compose.material", name = "material-icons-extended", version.ref = "composeIconsExtended" }
[plugins]
android-application = { id = "com.android.application", version.ref = "agp" }
kotlin-compose = { id = "org.jetbrains.kotlin.plugin.compose", version.ref = "kotlin" }
kotlin-serialization = { id = "org.jetbrains.kotlin.plugin.serialization", version.ref = "kotlin" }
Step 2: Add serialization plugin to root build.gradle.kts
In android/build.gradle.kts, add:
plugins {
alias(libs.plugins.android.application) apply false
alias(libs.plugins.kotlin.compose) apply false
alias(libs.plugins.kotlin.serialization) apply false
}
Step 3: Add plugin and dependencies to app build.gradle.kts
In android/app/build.gradle.kts:
plugins {
alias(libs.plugins.android.application)
alias(libs.plugins.kotlin.compose)
alias(libs.plugins.kotlin.serialization)
}
// ... android block stays the same, but fix compileSdk ...
android {
namespace = "com.thisux.droidclaw"
compileSdk = 36
defaultConfig {
applicationId = "com.thisux.droidclaw"
minSdk = 24
targetSdk = 36
versionCode = 1
versionName = "1.0"
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
}
// ... rest stays the same ...
kotlinOptions {
jvmTarget = "11"
}
}
dependencies {
// Existing
implementation(libs.androidx.core.ktx)
implementation(libs.androidx.lifecycle.runtime.ktx)
implementation(libs.androidx.activity.compose)
implementation(platform(libs.androidx.compose.bom))
implementation(libs.androidx.compose.ui)
implementation(libs.androidx.compose.ui.graphics)
implementation(libs.androidx.compose.ui.tooling.preview)
implementation(libs.androidx.compose.material3)
// New - Ktor WebSocket
implementation(libs.ktor.client.cio)
implementation(libs.ktor.client.websockets)
implementation(libs.ktor.client.content.negotiation)
implementation(libs.ktor.serialization.kotlinx.json)
// New - Serialization
implementation(libs.kotlinx.serialization.json)
// New - Coroutines
implementation(libs.kotlinx.coroutines.android)
// New - DataStore
implementation(libs.datastore.preferences)
// New - Lifecycle service
implementation(libs.lifecycle.service)
// New - Navigation
implementation(libs.navigation.compose)
implementation(libs.compose.icons.extended)
// Test deps stay the same
testImplementation(libs.junit)
androidTestImplementation(libs.androidx.junit)
androidTestImplementation(libs.androidx.espresso.core)
androidTestImplementation(platform(libs.androidx.compose.bom))
androidTestImplementation(libs.androidx.compose.ui.test.junit4)
debugImplementation(libs.androidx.compose.ui.tooling)
debugImplementation(libs.androidx.compose.ui.test.manifest)
}
Step 4: Sync and verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 5: Commit
git add android/gradle/libs.versions.toml android/build.gradle.kts android/app/build.gradle.kts
git commit -m "feat(android): add Ktor, serialization, DataStore, navigation dependencies"
Task 2: Data Models
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/model/UIElement.kt - Create:
android/app/src/main/java/com/thisux/droidclaw/model/Protocol.kt - Create:
android/app/src/main/java/com/thisux/droidclaw/model/AppState.kt
These mirror packages/shared/src/types.ts and packages/shared/src/protocol.ts.
Step 1: Create UIElement.kt
package com.thisux.droidclaw.model
import kotlinx.serialization.Serializable
@Serializable
data class UIElement(
val id: String = "",
val text: String = "",
val type: String = "",
val bounds: String = "",
val center: List<Int> = listOf(0, 0),
val size: List<Int> = listOf(0, 0),
val clickable: Boolean = false,
val editable: Boolean = false,
val enabled: Boolean = false,
val checked: Boolean = false,
val focused: Boolean = false,
val selected: Boolean = false,
val scrollable: Boolean = false,
val longClickable: Boolean = false,
val password: Boolean = false,
val hint: String = "",
val action: String = "read",
val parent: String = "",
val depth: Int = 0
)
Step 2: Create Protocol.kt
package com.thisux.droidclaw.model
import kotlinx.serialization.Serializable
import kotlinx.serialization.json.JsonObject
// Device → Server messages
@Serializable
data class AuthMessage(
val type: String = "auth",
val apiKey: String,
val deviceInfo: DeviceInfoMsg? = null
)
@Serializable
data class DeviceInfoMsg(
val model: String,
val androidVersion: String,
val screenWidth: Int,
val screenHeight: Int
)
@Serializable
data class ScreenResponse(
val type: String = "screen",
val requestId: String,
val elements: List<UIElement>,
val screenshot: String? = null,
val packageName: String? = null
)
@Serializable
data class ResultResponse(
val type: String = "result",
val requestId: String,
val success: Boolean,
val error: String? = null,
val data: String? = null
)
@Serializable
data class GoalMessage(
val type: String = "goal",
val text: String
)
@Serializable
data class PongMessage(
val type: String = "pong"
)
// Server → Device messages (parsed via discriminator)
@Serializable
data class ServerMessage(
val type: String,
val requestId: String? = null,
val deviceId: String? = null,
val message: String? = null,
val sessionId: String? = null,
val goal: String? = null,
val success: Boolean? = null,
val stepsUsed: Int? = null,
val step: Int? = null,
val action: JsonObject? = null,
val reasoning: String? = null,
val screenHash: String? = null,
// Action-specific fields
val x: Int? = null,
val y: Int? = null,
val x1: Int? = null,
val y1: Int? = null,
val x2: Int? = null,
val y2: Int? = null,
val duration: Int? = null,
val text: String? = null,
val packageName: String? = null,
val url: String? = null,
val code: Int? = null
)
Step 3: Create AppState.kt
package com.thisux.droidclaw.model
enum class ConnectionState {
Disconnected,
Connecting,
Connected,
Error
}
enum class GoalStatus {
Idle,
Running,
Completed,
Failed
}
data class AgentStep(
val step: Int,
val action: String,
val reasoning: String,
val timestamp: Long = System.currentTimeMillis()
)
data class GoalSession(
val sessionId: String,
val goal: String,
val steps: List<AgentStep>,
val status: GoalStatus,
val timestamp: Long = System.currentTimeMillis()
)
Step 4: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 5: Commit
git add android/app/src/main/java/com/thisux/droidclaw/model/
git commit -m "feat(android): add data models (UIElement, Protocol, AppState)"
Task 3: DataStore Settings
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/data/SettingsStore.kt - Create:
android/app/src/main/java/com/thisux/droidclaw/DroidClawApp.kt - Modify:
android/app/src/main/AndroidManifest.xml(add Application class)
Step 1: Create SettingsStore.kt
package com.thisux.droidclaw.data
import android.content.Context
import androidx.datastore.core.DataStore
import androidx.datastore.preferences.core.Preferences
import androidx.datastore.preferences.core.booleanPreferencesKey
import androidx.datastore.preferences.core.edit
import androidx.datastore.preferences.core.stringPreferencesKey
import androidx.datastore.preferences.preferencesDataStore
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.map
val Context.dataStore: DataStore<Preferences> by preferencesDataStore(name = "settings")
object SettingsKeys {
val API_KEY = stringPreferencesKey("api_key")
val SERVER_URL = stringPreferencesKey("server_url")
val DEVICE_NAME = stringPreferencesKey("device_name")
val AUTO_CONNECT = booleanPreferencesKey("auto_connect")
}
class SettingsStore(private val context: Context) {
val apiKey: Flow<String> = context.dataStore.data.map { prefs ->
prefs[SettingsKeys.API_KEY] ?: ""
}
val serverUrl: Flow<String> = context.dataStore.data.map { prefs ->
prefs[SettingsKeys.SERVER_URL] ?: "wss://localhost:8080"
}
val deviceName: Flow<String> = context.dataStore.data.map { prefs ->
prefs[SettingsKeys.DEVICE_NAME] ?: android.os.Build.MODEL
}
val autoConnect: Flow<Boolean> = context.dataStore.data.map { prefs ->
prefs[SettingsKeys.AUTO_CONNECT] ?: false
}
suspend fun setApiKey(value: String) {
context.dataStore.edit { it[SettingsKeys.API_KEY] = value }
}
suspend fun setServerUrl(value: String) {
context.dataStore.edit { it[SettingsKeys.SERVER_URL] = value }
}
suspend fun setDeviceName(value: String) {
context.dataStore.edit { it[SettingsKeys.DEVICE_NAME] = value }
}
suspend fun setAutoConnect(value: Boolean) {
context.dataStore.edit { it[SettingsKeys.AUTO_CONNECT] = value }
}
}
Step 2: Create DroidClawApp.kt
package com.thisux.droidclaw
import android.app.Application
import com.thisux.droidclaw.data.SettingsStore
class DroidClawApp : Application() {
lateinit var settingsStore: SettingsStore
private set
override fun onCreate() {
super.onCreate()
settingsStore = SettingsStore(this)
}
}
Step 3: Register Application class in AndroidManifest.xml
Add android:name=".DroidClawApp" to the <application> tag:
<application
android:name=".DroidClawApp"
android:allowBackup="true"
...>
Step 4: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 5: Commit
git add android/app/src/main/java/com/thisux/droidclaw/data/ android/app/src/main/java/com/thisux/droidclaw/DroidClawApp.kt android/app/src/main/AndroidManifest.xml
git commit -m "feat(android): add DataStore settings and Application class"
Task 4: Accessibility Service + ScreenTreeBuilder
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/accessibility/DroidClawAccessibilityService.kt - Create:
android/app/src/main/java/com/thisux/droidclaw/accessibility/ScreenTreeBuilder.kt - Create:
android/app/src/main/res/xml/accessibility_config.xml - Modify:
android/app/src/main/AndroidManifest.xml(add service declaration)
Step 1: Create accessibility_config.xml
Create android/app/src/main/res/xml/accessibility_config.xml:
<?xml version="1.0" encoding="utf-8"?>
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:accessibilityEventTypes="typeWindowStateChanged|typeWindowContentChanged|typeViewFocused"
android:accessibilityFeedbackType="feedbackGeneric"
android:accessibilityFlags="flagDefault|flagIncludeNotImportantViews|flagReportViewIds|flagRequestEnhancedWebAccessibility"
android:canPerformGestures="true"
android:canRetrieveWindowContent="true"
android:notificationTimeout="100"
android:settingsActivity="com.thisux.droidclaw.MainActivity" />
Step 2: Create ScreenTreeBuilder.kt
package com.thisux.droidclaw.accessibility
import android.graphics.Rect
import android.view.accessibility.AccessibilityNodeInfo
import com.thisux.droidclaw.model.UIElement
import java.security.MessageDigest
object ScreenTreeBuilder {
fun capture(rootNode: AccessibilityNodeInfo?): List<UIElement> {
if (rootNode == null) return emptyList()
val elements = mutableListOf<UIElement>()
walkTree(rootNode, elements, depth = 0, parentDesc = "")
return elements
}
private fun walkTree(
node: AccessibilityNodeInfo,
elements: MutableList<UIElement>,
depth: Int,
parentDesc: String
) {
try {
val rect = Rect()
node.getBoundsInScreen(rect)
val text = node.text?.toString() ?: ""
val contentDesc = node.contentDescription?.toString() ?: ""
val viewId = node.viewIdResourceName ?: ""
val className = node.className?.toString() ?: ""
val displayText = text.ifEmpty { contentDesc }
val isInteractive = node.isClickable || node.isLongClickable ||
node.isEditable || node.isScrollable || node.isFocusable
if (isInteractive || displayText.isNotEmpty()) {
val centerX = (rect.left + rect.right) / 2
val centerY = (rect.top + rect.bottom) / 2
val width = rect.width()
val height = rect.height()
val action = when {
node.isEditable -> "type"
node.isScrollable -> "scroll"
node.isLongClickable -> "longpress"
node.isClickable -> "tap"
else -> "read"
}
elements.add(
UIElement(
id = viewId,
text = displayText,
type = className.substringAfterLast("."),
bounds = "[${rect.left},${rect.top}][${rect.right},${rect.bottom}]",
center = listOf(centerX, centerY),
size = listOf(width, height),
clickable = node.isClickable,
editable = node.isEditable,
enabled = node.isEnabled,
checked = node.isChecked,
focused = node.isFocused,
selected = node.isSelected,
scrollable = node.isScrollable,
longClickable = node.isLongClickable,
password = node.isPassword,
hint = node.hintText?.toString() ?: "",
action = action,
parent = parentDesc,
depth = depth
)
)
}
for (i in 0 until node.childCount) {
val child = node.getChild(i) ?: continue
try {
walkTree(child, elements, depth + 1, className)
} finally {
child.recycle()
}
}
} catch (_: Exception) {
// Node may have been recycled during traversal
}
}
fun computeScreenHash(elements: List<UIElement>): String {
val digest = MessageDigest.getInstance("MD5")
for (el in elements) {
digest.update("${el.id}|${el.text}|${el.center}".toByteArray())
}
return digest.digest().joinToString("") { "%02x".format(it) }.take(12)
}
}
Step 3: Create DroidClawAccessibilityService.kt
package com.thisux.droidclaw.accessibility
import android.accessibilityservice.AccessibilityService
import android.util.Log
import android.view.accessibility.AccessibilityEvent
import android.view.accessibility.AccessibilityNodeInfo
import com.thisux.droidclaw.model.UIElement
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.runBlocking
class DroidClawAccessibilityService : AccessibilityService() {
companion object {
private const val TAG = "DroidClawA11y"
val isRunning = MutableStateFlow(false)
val lastScreenTree = MutableStateFlow<List<UIElement>>(emptyList())
var instance: DroidClawAccessibilityService? = null
}
override fun onServiceConnected() {
super.onServiceConnected()
Log.i(TAG, "Accessibility service connected")
instance = this
isRunning.value = true
}
override fun onAccessibilityEvent(event: AccessibilityEvent?) {
// We capture on-demand via getScreenTree(), not on every event
}
override fun onInterrupt() {
Log.w(TAG, "Accessibility service interrupted")
}
override fun onDestroy() {
super.onDestroy()
Log.i(TAG, "Accessibility service destroyed")
instance = null
isRunning.value = false
}
/**
* Capture current screen tree with retry for null rootInActiveWindow.
* Returns empty list if root is still null after retries (server uses vision fallback).
*/
fun getScreenTree(): List<UIElement> {
val delays = longArrayOf(50, 100, 200)
for (delayMs in delays) {
val root = rootInActiveWindow
if (root != null) {
try {
val elements = ScreenTreeBuilder.capture(root)
lastScreenTree.value = elements
return elements
} finally {
root.recycle()
}
}
runBlocking { delay(delayMs) }
}
Log.w(TAG, "rootInActiveWindow null after retries")
return emptyList()
}
/**
* Find node closest to given coordinates.
*/
fun findNodeAt(x: Int, y: Int): AccessibilityNodeInfo? {
val root = rootInActiveWindow ?: return null
return findNodeAtRecursive(root, x, y)
}
private fun findNodeAtRecursive(
node: AccessibilityNodeInfo,
x: Int,
y: Int
): AccessibilityNodeInfo? {
val rect = android.graphics.Rect()
node.getBoundsInScreen(rect)
if (!rect.contains(x, y)) {
node.recycle()
return null
}
// Check children (deeper = more specific)
for (i in 0 until node.childCount) {
val child = node.getChild(i) ?: continue
val found = findNodeAtRecursive(child, x, y)
if (found != null) {
node.recycle()
return found
}
}
// This node contains the point and no child does
return if (node.isClickable || node.isLongClickable || node.isEditable) {
node
} else {
node.recycle()
null
}
}
}
Step 4: Add service declaration + permissions to AndroidManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools">
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_CONNECTED_DEVICE" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" />
<uses-permission android:name="android.permission.WAKE_LOCK" />
<application
android:name=".DroidClawApp"
android:allowBackup="true"
android:dataExtractionRules="@xml/data_extraction_rules"
android:fullBackupContent="@xml/backup_rules"
android:icon="@mipmap/ic_launcher"
android:label="@string/app_name"
android:roundIcon="@mipmap/ic_launcher_round"
android:supportsRtl="true"
android:theme="@style/Theme.DroidClaw">
<activity
android:name=".MainActivity"
android:exported="true"
android:label="@string/app_name"
android:theme="@style/Theme.DroidClaw">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity>
<service
android:name=".accessibility.DroidClawAccessibilityService"
android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"
android:exported="false">
<intent-filter>
<action android:name="android.accessibilityservice.AccessibilityService" />
</intent-filter>
<meta-data
android:name="android.accessibilityservice"
android:resource="@xml/accessibility_config" />
</service>
<service
android:name=".connection.ConnectionService"
android:foregroundServiceType="connectedDevice"
android:exported="false" />
</application>
</manifest>
Step 5: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 6: Commit
git add android/app/src/main/java/com/thisux/droidclaw/accessibility/ android/app/src/main/res/xml/accessibility_config.xml android/app/src/main/AndroidManifest.xml
git commit -m "feat(android): add AccessibilityService and ScreenTreeBuilder"
Task 5: GestureExecutor
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/accessibility/GestureExecutor.kt
Step 1: Create GestureExecutor.kt
This implements the node-first strategy: try performAction() on accessibility nodes first, fall back to dispatchGesture() with coordinates.
package com.thisux.droidclaw.accessibility
import android.accessibilityservice.AccessibilityService
import android.accessibilityservice.GestureDescription
import android.content.Intent
import android.graphics.Path
import android.net.Uri
import android.os.Bundle
import android.util.Log
import android.view.accessibility.AccessibilityNodeInfo
import com.thisux.droidclaw.model.ServerMessage
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlin.coroutines.resume
data class ActionResult(val success: Boolean, val error: String? = null, val data: String? = null)
class GestureExecutor(private val service: DroidClawAccessibilityService) {
companion object {
private const val TAG = "GestureExecutor"
}
suspend fun execute(msg: ServerMessage): ActionResult {
return try {
when (msg.type) {
"tap" -> executeTap(msg.x ?: 0, msg.y ?: 0)
"type" -> executeType(msg.text ?: "")
"enter" -> executeEnter()
"back" -> executeGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK)
"home" -> executeGlobalAction(AccessibilityService.GLOBAL_ACTION_HOME)
"notifications" -> executeGlobalAction(AccessibilityService.GLOBAL_ACTION_NOTIFICATIONS)
"longpress" -> executeLongPress(msg.x ?: 0, msg.y ?: 0)
"swipe" -> executeSwipe(
msg.x1 ?: 0, msg.y1 ?: 0,
msg.x2 ?: 0, msg.y2 ?: 0,
msg.duration ?: 300
)
"launch" -> executeLaunch(msg.packageName ?: "")
"clear" -> executeClear()
"clipboard_set" -> executeClipboardSet(msg.text ?: "")
"clipboard_get" -> executeClipboardGet()
"paste" -> executePaste()
"open_url" -> executeOpenUrl(msg.url ?: "")
"switch_app" -> executeLaunch(msg.packageName ?: "")
"keyevent" -> executeKeyEvent(msg.code ?: 0)
"open_settings" -> executeOpenSettings()
"wait" -> executeWait(msg.duration ?: 1000)
else -> ActionResult(false, "Unknown action: ${msg.type}")
}
} catch (e: Exception) {
Log.e(TAG, "Action ${msg.type} failed", e)
ActionResult(false, e.message)
}
}
private suspend fun executeTap(x: Int, y: Int): ActionResult {
// Try node-first
val node = service.findNodeAt(x, y)
if (node != null) {
try {
if (node.performAction(AccessibilityNodeInfo.ACTION_CLICK)) {
return ActionResult(true)
}
} finally {
node.recycle()
}
}
// Fallback to gesture
return dispatchTapGesture(x, y)
}
private suspend fun executeType(text: String): ActionResult {
val focused = findFocusedNode()
if (focused != null) {
try {
val args = Bundle().apply {
putCharSequence(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, text)
}
if (focused.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args)) {
return ActionResult(true)
}
} finally {
focused.recycle()
}
}
return ActionResult(false, "No focused editable node found")
}
private fun executeEnter(): ActionResult {
val focused = findFocusedNode()
if (focused != null) {
try {
// Try IME action first
if (focused.performAction(AccessibilityNodeInfo.ACTION_IME_ENTER)) {
return ActionResult(true)
}
} finally {
focused.recycle()
}
}
// Fallback: global key event
return ActionResult(
service.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK).not(), // placeholder
"Enter key fallback not available via accessibility"
)
}
private fun executeGlobalAction(action: Int): ActionResult {
val success = service.performGlobalAction(action)
return ActionResult(success, if (!success) "Global action failed" else null)
}
private suspend fun executeLongPress(x: Int, y: Int): ActionResult {
// Try node-first
val node = service.findNodeAt(x, y)
if (node != null) {
try {
if (node.performAction(AccessibilityNodeInfo.ACTION_LONG_CLICK)) {
return ActionResult(true)
}
} finally {
node.recycle()
}
}
// Fallback: gesture hold at point
return dispatchSwipeGesture(x, y, x, y, 1000)
}
private suspend fun executeSwipe(x1: Int, y1: Int, x2: Int, y2: Int, duration: Int): ActionResult {
return dispatchSwipeGesture(x1, y1, x2, y2, duration)
}
private fun executeLaunch(packageName: String): ActionResult {
val intent = service.packageManager.getLaunchIntentForPackage(packageName)
?: return ActionResult(false, "Package not found: $packageName")
intent.addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
service.startActivity(intent)
return ActionResult(true)
}
private fun executeClear(): ActionResult {
val focused = findFocusedNode()
if (focused != null) {
try {
val args = Bundle().apply {
putCharSequence(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE, "")
}
if (focused.performAction(AccessibilityNodeInfo.ACTION_SET_TEXT, args)) {
return ActionResult(true)
}
} finally {
focused.recycle()
}
}
return ActionResult(false, "No focused editable node to clear")
}
private fun executeClipboardSet(text: String): ActionResult {
val clipboard = service.getSystemService(android.content.Context.CLIPBOARD_SERVICE) as android.content.ClipboardManager
val clip = android.content.ClipData.newPlainText("droidclaw", text)
clipboard.setPrimaryClip(clip)
return ActionResult(true)
}
private fun executeClipboardGet(): ActionResult {
val clipboard = service.getSystemService(android.content.Context.CLIPBOARD_SERVICE) as android.content.ClipboardManager
val text = clipboard.primaryClip?.getItemAt(0)?.text?.toString() ?: ""
return ActionResult(true, data = text)
}
private fun executePaste(): ActionResult {
val focused = findFocusedNode()
if (focused != null) {
try {
if (focused.performAction(AccessibilityNodeInfo.ACTION_PASTE)) {
return ActionResult(true)
}
} finally {
focused.recycle()
}
}
return ActionResult(false, "No focused node to paste into")
}
private fun executeOpenUrl(url: String): ActionResult {
val intent = Intent(Intent.ACTION_VIEW, Uri.parse(url)).apply {
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
service.startActivity(intent)
return ActionResult(true)
}
private fun executeKeyEvent(code: Int): ActionResult {
// AccessibilityService doesn't have direct keyevent dispatch
// Use instrumentation or shell command via Runtime
return try {
Runtime.getRuntime().exec(arrayOf("input", "keyevent", code.toString()))
ActionResult(true)
} catch (e: Exception) {
ActionResult(false, "keyevent failed: ${e.message}")
}
}
private fun executeOpenSettings(): ActionResult {
val intent = Intent(android.provider.Settings.ACTION_SETTINGS).apply {
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
service.startActivity(intent)
return ActionResult(true)
}
private suspend fun executeWait(duration: Int): ActionResult {
kotlinx.coroutines.delay(duration.toLong())
return ActionResult(true)
}
// --- Gesture Helpers ---
private suspend fun dispatchTapGesture(x: Int, y: Int): ActionResult {
val path = Path().apply { moveTo(x.toFloat(), y.toFloat()) }
val stroke = GestureDescription.StrokeDescription(path, 0, 50)
val gesture = GestureDescription.Builder().addStroke(stroke).build()
return dispatchGesture(gesture)
}
private suspend fun dispatchSwipeGesture(
x1: Int, y1: Int, x2: Int, y2: Int, duration: Int
): ActionResult {
val path = Path().apply {
moveTo(x1.toFloat(), y1.toFloat())
lineTo(x2.toFloat(), y2.toFloat())
}
val stroke = GestureDescription.StrokeDescription(path, 0, duration.toLong())
val gesture = GestureDescription.Builder().addStroke(stroke).build()
return dispatchGesture(gesture)
}
private suspend fun dispatchGesture(gesture: GestureDescription): ActionResult =
suspendCancellableCoroutine { cont ->
service.dispatchGesture(
gesture,
object : AccessibilityService.GestureResultCallback() {
override fun onCompleted(gestureDescription: GestureDescription?) {
if (cont.isActive) cont.resume(ActionResult(true))
}
override fun onCancelled(gestureDescription: GestureDescription?) {
if (cont.isActive) cont.resume(ActionResult(false, "Gesture cancelled"))
}
},
null
)
}
private fun findFocusedNode(): AccessibilityNodeInfo? {
return service.rootInActiveWindow?.findFocus(AccessibilityNodeInfo.FOCUS_INPUT)
}
}
Step 2: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 3: Commit
git add android/app/src/main/java/com/thisux/droidclaw/accessibility/GestureExecutor.kt
git commit -m "feat(android): add GestureExecutor with node-first strategy"
Task 6: Screen Capture (MediaProjection)
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/capture/ScreenCaptureManager.kt
Step 1: Create ScreenCaptureManager.kt
package com.thisux.droidclaw.capture
import android.app.Activity
import android.content.Context
import android.content.Intent
import android.graphics.Bitmap
import android.graphics.PixelFormat
import android.hardware.display.DisplayManager
import android.hardware.display.VirtualDisplay
import android.media.ImageReader
import android.media.projection.MediaProjection
import android.media.projection.MediaProjectionManager
import android.util.DisplayMetrics
import android.util.Log
import android.view.WindowManager
import kotlinx.coroutines.flow.MutableStateFlow
import java.io.ByteArrayOutputStream
class ScreenCaptureManager(private val context: Context) {
companion object {
private const val TAG = "ScreenCapture"
const val REQUEST_CODE = 1001
val isAvailable = MutableStateFlow(false)
}
private var mediaProjection: MediaProjection? = null
private var virtualDisplay: VirtualDisplay? = null
private var imageReader: ImageReader? = null
private var screenWidth = 720
private var screenHeight = 1280
private var screenDensity = DisplayMetrics.DENSITY_DEFAULT
fun initialize(resultCode: Int, data: Intent) {
val mgr = context.getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
mediaProjection = mgr.getMediaProjection(resultCode, data)
val wm = context.getSystemService(Context.WINDOW_SERVICE) as WindowManager
val metrics = DisplayMetrics()
@Suppress("DEPRECATION")
wm.defaultDisplay.getRealMetrics(metrics)
screenWidth = metrics.widthPixels
screenHeight = metrics.heightPixels
screenDensity = metrics.densityDpi
// Scale down for capture
val scale = 720f / screenWidth
val captureWidth = 720
val captureHeight = (screenHeight * scale).toInt()
imageReader = ImageReader.newInstance(captureWidth, captureHeight, PixelFormat.RGBA_8888, 2)
virtualDisplay = mediaProjection?.createVirtualDisplay(
"DroidClaw",
captureWidth, captureHeight, screenDensity,
DisplayManager.VIRTUAL_DISPLAY_FLAG_AUTO_MIRROR,
imageReader!!.surface, null, null
)
mediaProjection?.registerCallback(object : MediaProjection.Callback() {
override fun onStop() {
Log.i(TAG, "MediaProjection stopped")
release()
}
}, null)
isAvailable.value = true
Log.i(TAG, "Screen capture initialized: ${captureWidth}x${captureHeight}")
}
fun capture(): ByteArray? {
val reader = imageReader ?: return null
val image = reader.acquireLatestImage() ?: return null
return try {
val planes = image.planes
val buffer = planes[0].buffer
val pixelStride = planes[0].pixelStride
val rowStride = planes[0].rowStride
val rowPadding = rowStride - pixelStride * image.width
val bitmap = Bitmap.createBitmap(
image.width + rowPadding / pixelStride,
image.height,
Bitmap.Config.ARGB_8888
)
bitmap.copyPixelsFromBuffer(buffer)
// Crop padding
val cropped = Bitmap.createBitmap(bitmap, 0, 0, image.width, image.height)
if (cropped != bitmap) bitmap.recycle()
// Check for secure window (all black)
if (isBlackFrame(cropped)) {
cropped.recycle()
Log.w(TAG, "Detected FLAG_SECURE (black frame)")
return null
}
// Compress to JPEG
val stream = ByteArrayOutputStream()
cropped.compress(Bitmap.CompressFormat.JPEG, 50, stream)
cropped.recycle()
stream.toByteArray()
} finally {
image.close()
}
}
private fun isBlackFrame(bitmap: Bitmap): Boolean {
// Sample 4 corners + center
val points = listOf(
0 to 0,
bitmap.width - 1 to 0,
0 to bitmap.height - 1,
bitmap.width - 1 to bitmap.height - 1,
bitmap.width / 2 to bitmap.height / 2
)
return points.all { (x, y) -> bitmap.getPixel(x, y) == android.graphics.Color.BLACK }
}
fun release() {
virtualDisplay?.release()
virtualDisplay = null
imageReader?.close()
imageReader = null
mediaProjection?.stop()
mediaProjection = null
isAvailable.value = false
}
}
Step 2: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 3: Commit
git add android/app/src/main/java/com/thisux/droidclaw/capture/
git commit -m "feat(android): add ScreenCaptureManager with MediaProjection"
Task 7: ReliableWebSocket (Ktor)
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/connection/ReliableWebSocket.kt
Step 1: Create ReliableWebSocket.kt
package com.thisux.droidclaw.connection
import android.util.Log
import com.thisux.droidclaw.model.AuthMessage
import com.thisux.droidclaw.model.ConnectionState
import com.thisux.droidclaw.model.DeviceInfoMsg
import com.thisux.droidclaw.model.ServerMessage
import io.ktor.client.HttpClient
import io.ktor.client.engine.cio.CIO
import io.ktor.client.plugins.websocket.WebSockets
import io.ktor.client.plugins.websocket.webSocket
import io.ktor.websocket.Frame
import io.ktor.websocket.close
import io.ktor.websocket.readText
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.isActive
import kotlinx.coroutines.launch
import kotlinx.serialization.encodeToString
import kotlinx.serialization.json.Json
class ReliableWebSocket(
private val scope: CoroutineScope,
private val onMessage: suspend (ServerMessage) -> Unit
) {
companion object {
private const val TAG = "ReliableWS"
private const val MAX_BACKOFF_MS = 30_000L
}
private val json = Json { ignoreUnknownKeys = true; encodeDefaults = true }
private val _state = MutableStateFlow(ConnectionState.Disconnected)
val state: StateFlow<ConnectionState> = _state
private val _errorMessage = MutableStateFlow<String?>(null)
val errorMessage: StateFlow<String?> = _errorMessage
private val outbound = Channel<String>(Channel.BUFFERED)
private var connectionJob: Job? = null
private var client: HttpClient? = null
private var backoffMs = 1000L
private var shouldReconnect = true
var deviceId: String? = null
private set
fun connect(serverUrl: String, apiKey: String, deviceInfo: DeviceInfoMsg) {
shouldReconnect = true
connectionJob?.cancel()
connectionJob = scope.launch {
while (shouldReconnect && isActive) {
try {
_state.value = ConnectionState.Connecting
_errorMessage.value = null
connectOnce(serverUrl, apiKey, deviceInfo)
} catch (e: CancellationException) {
throw e
} catch (e: Exception) {
Log.e(TAG, "Connection failed: ${e.message}")
_state.value = ConnectionState.Error
_errorMessage.value = e.message
}
if (shouldReconnect && isActive) {
Log.i(TAG, "Reconnecting in ${backoffMs}ms")
delay(backoffMs)
backoffMs = (backoffMs * 2).coerceAtMost(MAX_BACKOFF_MS)
}
}
}
}
private suspend fun connectOnce(serverUrl: String, apiKey: String, deviceInfo: DeviceInfoMsg) {
val httpClient = HttpClient(CIO) {
install(WebSockets) {
pingIntervalMillis = 30_000
}
}
client = httpClient
// Convert wss:// to proper URL path
val wsUrl = serverUrl.trimEnd('/') + "/ws/device"
httpClient.webSocket(wsUrl) {
// Auth handshake
val authMsg = AuthMessage(apiKey = apiKey, deviceInfo = deviceInfo)
send(Frame.Text(json.encodeToString(authMsg)))
Log.i(TAG, "Sent auth message")
// Wait for auth response
val authFrame = incoming.receive() as? Frame.Text
?: throw Exception("Expected text frame for auth response")
val authResponse = json.decodeFromString<ServerMessage>(authFrame.readText())
when (authResponse.type) {
"auth_ok" -> {
deviceId = authResponse.deviceId
_state.value = ConnectionState.Connected
_errorMessage.value = null
backoffMs = 1000L // Reset backoff on success
Log.i(TAG, "Authenticated, deviceId=$deviceId")
}
"auth_error" -> {
shouldReconnect = false // Don't retry auth errors
_state.value = ConnectionState.Error
_errorMessage.value = authResponse.message ?: "Authentication failed"
close()
return@webSocket
}
else -> {
throw Exception("Unexpected auth response: ${authResponse.type}")
}
}
// Launch outbound sender
val senderJob = launch {
for (msg in outbound) {
send(Frame.Text(msg))
}
}
// Read incoming messages
try {
for (frame in incoming) {
if (frame is Frame.Text) {
val text = frame.readText()
try {
val msg = json.decodeFromString<ServerMessage>(text)
onMessage(msg)
} catch (e: Exception) {
Log.e(TAG, "Failed to parse message: ${e.message}")
}
}
}
} finally {
senderJob.cancel()
}
}
httpClient.close()
client = null
_state.value = ConnectionState.Disconnected
}
fun send(message: String) {
outbound.trySend(message)
}
inline fun <reified T> sendTyped(message: T) {
send(json.encodeToString(message))
}
fun disconnect() {
shouldReconnect = false
connectionJob?.cancel()
connectionJob = null
client?.close()
client = null
_state.value = ConnectionState.Disconnected
_errorMessage.value = null
deviceId = null
}
}
Step 2: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 3: Commit
git add android/app/src/main/java/com/thisux/droidclaw/connection/ReliableWebSocket.kt
git commit -m "feat(android): add ReliableWebSocket with Ktor, reconnect, auth handshake"
Task 8: CommandRouter
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/connection/CommandRouter.kt
Step 1: Create CommandRouter.kt
package com.thisux.droidclaw.connection
import android.util.Base64
import android.util.Log
import com.thisux.droidclaw.accessibility.DroidClawAccessibilityService
import com.thisux.droidclaw.accessibility.GestureExecutor
import com.thisux.droidclaw.accessibility.ScreenTreeBuilder
import com.thisux.droidclaw.capture.ScreenCaptureManager
import com.thisux.droidclaw.model.AgentStep
import com.thisux.droidclaw.model.GoalStatus
import com.thisux.droidclaw.model.PongMessage
import com.thisux.droidclaw.model.ResultResponse
import com.thisux.droidclaw.model.ScreenResponse
import com.thisux.droidclaw.model.ServerMessage
import kotlinx.coroutines.flow.MutableStateFlow
class CommandRouter(
private val webSocket: ReliableWebSocket,
private val captureManager: ScreenCaptureManager?
) {
companion object {
private const val TAG = "CommandRouter"
}
val currentGoalStatus = MutableStateFlow(GoalStatus.Idle)
val currentSteps = MutableStateFlow<List<AgentStep>>(emptyList())
val currentGoal = MutableStateFlow("")
val currentSessionId = MutableStateFlow<String?>(null)
private var gestureExecutor: GestureExecutor? = null
fun updateGestureExecutor() {
val svc = DroidClawAccessibilityService.instance
gestureExecutor = if (svc != null) GestureExecutor(svc) else null
}
suspend fun handleMessage(msg: ServerMessage) {
Log.d(TAG, "Handling: ${msg.type}")
when (msg.type) {
"get_screen" -> handleGetScreen(msg.requestId!!)
"ping" -> webSocket.sendTyped(PongMessage())
// Action commands — all have requestId
"tap", "type", "enter", "back", "home", "notifications",
"longpress", "swipe", "launch", "clear", "clipboard_set",
"clipboard_get", "paste", "open_url", "switch_app",
"keyevent", "open_settings", "wait" -> handleAction(msg)
// Goal lifecycle
"goal_started" -> {
currentSessionId.value = msg.sessionId
currentGoal.value = msg.goal ?: ""
currentGoalStatus.value = GoalStatus.Running
currentSteps.value = emptyList()
Log.i(TAG, "Goal started: ${msg.goal}")
}
"step" -> {
val step = AgentStep(
step = msg.step ?: 0,
action = msg.action?.toString() ?: "",
reasoning = msg.reasoning ?: ""
)
currentSteps.value = currentSteps.value + step
Log.d(TAG, "Step ${step.step}: ${step.reasoning}")
}
"goal_completed" -> {
currentGoalStatus.value = if (msg.success == true) GoalStatus.Completed else GoalStatus.Failed
Log.i(TAG, "Goal completed: success=${msg.success}, steps=${msg.stepsUsed}")
}
else -> Log.w(TAG, "Unknown message type: ${msg.type}")
}
}
private fun handleGetScreen(requestId: String) {
updateGestureExecutor()
val svc = DroidClawAccessibilityService.instance
val elements = svc?.getScreenTree() ?: emptyList()
val packageName = try {
svc?.rootInActiveWindow?.packageName?.toString()
} catch (_: Exception) { null }
// Optionally include screenshot
var screenshot: String? = null
if (elements.isEmpty()) {
// Vision fallback: capture screenshot
val bytes = captureManager?.capture()
if (bytes != null) {
screenshot = Base64.encodeToString(bytes, Base64.NO_WRAP)
}
}
val response = ScreenResponse(
requestId = requestId,
elements = elements,
screenshot = screenshot,
packageName = packageName
)
webSocket.sendTyped(response)
}
private suspend fun handleAction(msg: ServerMessage) {
updateGestureExecutor()
val executor = gestureExecutor
if (executor == null) {
webSocket.sendTyped(
ResultResponse(
requestId = msg.requestId!!,
success = false,
error = "Accessibility service not running"
)
)
return
}
val result = executor.execute(msg)
webSocket.sendTyped(
ResultResponse(
requestId = msg.requestId!!,
success = result.success,
error = result.error,
data = result.data
)
)
}
fun reset() {
currentGoalStatus.value = GoalStatus.Idle
currentSteps.value = emptyList()
currentGoal.value = ""
currentSessionId.value = null
}
}
Step 2: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 3: Commit
git add android/app/src/main/java/com/thisux/droidclaw/connection/CommandRouter.kt
git commit -m "feat(android): add CommandRouter for dispatching server commands"
Task 9: ConnectionService (Foreground Service)
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/connection/ConnectionService.kt - Create:
android/app/src/main/java/com/thisux/droidclaw/util/DeviceInfo.kt
Step 1: Create DeviceInfo.kt
package com.thisux.droidclaw.util
import android.content.Context
import android.util.DisplayMetrics
import android.view.WindowManager
import com.thisux.droidclaw.model.DeviceInfoMsg
object DeviceInfoHelper {
fun get(context: Context): DeviceInfoMsg {
val wm = context.getSystemService(Context.WINDOW_SERVICE) as WindowManager
val metrics = DisplayMetrics()
@Suppress("DEPRECATION")
wm.defaultDisplay.getRealMetrics(metrics)
return DeviceInfoMsg(
model = android.os.Build.MODEL,
androidVersion = android.os.Build.VERSION.RELEASE,
screenWidth = metrics.widthPixels,
screenHeight = metrics.heightPixels
)
}
}
Step 2: Create ConnectionService.kt
package com.thisux.droidclaw.connection
import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.PendingIntent
import android.content.Context
import android.content.Intent
import android.os.Build
import android.os.IBinder
import android.os.PowerManager
import android.util.Log
import androidx.core.app.NotificationCompat
import androidx.lifecycle.LifecycleService
import androidx.lifecycle.lifecycleScope
import com.thisux.droidclaw.DroidClawApp
import com.thisux.droidclaw.MainActivity
import com.thisux.droidclaw.R
import com.thisux.droidclaw.capture.ScreenCaptureManager
import com.thisux.droidclaw.model.ConnectionState
import com.thisux.droidclaw.model.GoalMessage
import com.thisux.droidclaw.model.GoalStatus
import com.thisux.droidclaw.model.AgentStep
import com.thisux.droidclaw.util.DeviceInfoHelper
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.launch
class ConnectionService : LifecycleService() {
companion object {
private const val TAG = "ConnectionSvc"
private const val CHANNEL_ID = "droidclaw_connection"
private const val NOTIFICATION_ID = 1
val connectionState = MutableStateFlow(ConnectionState.Disconnected)
val currentSteps = MutableStateFlow<List<AgentStep>>(emptyList())
val currentGoalStatus = MutableStateFlow(GoalStatus.Idle)
val currentGoal = MutableStateFlow("")
val errorMessage = MutableStateFlow<String?>(null)
var instance: ConnectionService? = null
const val ACTION_CONNECT = "com.thisux.droidclaw.CONNECT"
const val ACTION_DISCONNECT = "com.thisux.droidclaw.DISCONNECT"
const val ACTION_SEND_GOAL = "com.thisux.droidclaw.SEND_GOAL"
const val EXTRA_GOAL = "goal_text"
}
private var webSocket: ReliableWebSocket? = null
private var commandRouter: CommandRouter? = null
private var captureManager: ScreenCaptureManager? = null
private var wakeLock: PowerManager.WakeLock? = null
override fun onCreate() {
super.onCreate()
instance = this
createNotificationChannel()
}
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
super.onStartCommand(intent, flags, startId)
when (intent?.action) {
ACTION_CONNECT -> {
startForeground(NOTIFICATION_ID, buildNotification("Connecting..."))
connect()
}
ACTION_DISCONNECT -> {
disconnect()
stopSelf()
}
ACTION_SEND_GOAL -> {
val goal = intent.getStringExtra(EXTRA_GOAL) ?: return START_NOT_STICKY
sendGoal(goal)
}
}
return START_NOT_STICKY
}
private fun connect() {
lifecycleScope.launch {
val app = application as DroidClawApp
val apiKey = app.settingsStore.apiKey.first()
val serverUrl = app.settingsStore.serverUrl.first()
if (apiKey.isBlank() || serverUrl.isBlank()) {
connectionState.value = ConnectionState.Error
errorMessage.value = "API key or server URL not configured"
stopSelf()
return@launch
}
captureManager = ScreenCaptureManager(this@ConnectionService)
val ws = ReliableWebSocket(lifecycleScope) { msg ->
commandRouter?.handleMessage(msg)
}
webSocket = ws
val router = CommandRouter(ws, captureManager)
commandRouter = router
// Forward state
launch {
ws.state.collect { state ->
connectionState.value = state
updateNotification(
when (state) {
ConnectionState.Connected -> "Connected to server"
ConnectionState.Connecting -> "Connecting..."
ConnectionState.Error -> "Connection error"
ConnectionState.Disconnected -> "Disconnected"
}
)
}
}
launch {
ws.errorMessage.collect { errorMessage.value = it }
}
launch {
router.currentSteps.collect { currentSteps.value = it }
}
launch {
router.currentGoalStatus.collect { currentGoalStatus.value = it }
}
launch {
router.currentGoal.collect { currentGoal.value = it }
}
// Acquire wake lock during active connection
acquireWakeLock()
val deviceInfo = DeviceInfoHelper.get(this@ConnectionService)
ws.connect(serverUrl, apiKey, deviceInfo)
}
}
private fun sendGoal(text: String) {
webSocket?.sendTyped(GoalMessage(text = text))
}
private fun disconnect() {
webSocket?.disconnect()
webSocket = null
commandRouter?.reset()
commandRouter = null
captureManager?.release()
captureManager = null
releaseWakeLock()
connectionState.value = ConnectionState.Disconnected
}
override fun onDestroy() {
disconnect()
instance = null
super.onDestroy()
}
override fun onBind(intent: Intent): IBinder? {
super.onBind(intent)
return null
}
// --- Notification ---
private fun createNotificationChannel() {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
val channel = NotificationChannel(
CHANNEL_ID,
"DroidClaw Connection",
NotificationManager.IMPORTANCE_LOW
).apply {
description = "Shows when DroidClaw is connected to the server"
}
val nm = getSystemService(NotificationManager::class.java)
nm.createNotificationChannel(channel)
}
}
private fun buildNotification(text: String): Notification {
val openIntent = PendingIntent.getActivity(
this, 0,
Intent(this, MainActivity::class.java),
PendingIntent.FLAG_IMMUTABLE
)
val disconnectIntent = PendingIntent.getService(
this, 1,
Intent(this, ConnectionService::class.java).apply {
action = ACTION_DISCONNECT
},
PendingIntent.FLAG_IMMUTABLE
)
return NotificationCompat.Builder(this, CHANNEL_ID)
.setContentTitle("DroidClaw")
.setContentText(text)
.setSmallIcon(R.drawable.ic_launcher_foreground)
.setOngoing(true)
.setContentIntent(openIntent)
.addAction(0, "Disconnect", disconnectIntent)
.build()
}
private fun updateNotification(text: String) {
val nm = getSystemService(NotificationManager::class.java)
nm.notify(NOTIFICATION_ID, buildNotification(text))
}
// --- Wake Lock ---
private fun acquireWakeLock() {
val pm = getSystemService(Context.POWER_SERVICE) as PowerManager
wakeLock = pm.newWakeLock(
PowerManager.PARTIAL_WAKE_LOCK,
"DroidClaw::ConnectionWakeLock"
).apply {
acquire(10 * 60 * 1000L) // 10 minutes max
}
}
private fun releaseWakeLock() {
wakeLock?.let {
if (it.isHeld) it.release()
}
wakeLock = null
}
}
Step 3: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 4: Commit
git add android/app/src/main/java/com/thisux/droidclaw/connection/ConnectionService.kt android/app/src/main/java/com/thisux/droidclaw/util/DeviceInfo.kt
git commit -m "feat(android): add ConnectionService foreground service and DeviceInfo helper"
Task 10: UI — Navigation + HomeScreen
Files:
- Modify:
android/app/src/main/java/com/thisux/droidclaw/MainActivity.kt - Create:
android/app/src/main/java/com/thisux/droidclaw/ui/screens/HomeScreen.kt
Step 1: Create HomeScreen.kt
package com.thisux.droidclaw.ui.screens
import android.content.Intent
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.lazy.items
import androidx.compose.foundation.shape.CircleShape
import androidx.compose.material3.Button
import androidx.compose.material3.Card
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.OutlinedTextField
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.draw.clip
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.unit.dp
import com.thisux.droidclaw.connection.ConnectionService
import com.thisux.droidclaw.model.ConnectionState
import com.thisux.droidclaw.model.GoalStatus
@Composable
fun HomeScreen() {
val context = LocalContext.current
val connectionState by ConnectionService.connectionState.collectAsState()
val goalStatus by ConnectionService.currentGoalStatus.collectAsState()
val steps by ConnectionService.currentSteps.collectAsState()
val currentGoal by ConnectionService.currentGoal.collectAsState()
val errorMessage by ConnectionService.errorMessage.collectAsState()
var goalInput by remember { mutableStateOf("") }
Column(
modifier = Modifier
.fillMaxSize()
.padding(16.dp)
) {
// Status Badge
Row(
verticalAlignment = Alignment.CenterVertically,
modifier = Modifier.fillMaxWidth()
) {
Box(
modifier = Modifier
.size(12.dp)
.clip(CircleShape)
.background(
when (connectionState) {
ConnectionState.Connected -> Color(0xFF4CAF50)
ConnectionState.Connecting -> Color(0xFFFFC107)
ConnectionState.Error -> Color(0xFFF44336)
ConnectionState.Disconnected -> Color.Gray
}
)
)
Text(
text = when (connectionState) {
ConnectionState.Connected -> "Connected to server"
ConnectionState.Connecting -> "Connecting..."
ConnectionState.Error -> errorMessage ?: "Connection error"
ConnectionState.Disconnected -> "Disconnected"
},
style = MaterialTheme.typography.bodyLarge,
modifier = Modifier.padding(start = 8.dp)
)
}
Spacer(modifier = Modifier.height(8.dp))
// Connect/Disconnect button
Button(
onClick = {
val intent = Intent(context, ConnectionService::class.java).apply {
action = if (connectionState == ConnectionState.Disconnected || connectionState == ConnectionState.Error) {
ConnectionService.ACTION_CONNECT
} else {
ConnectionService.ACTION_DISCONNECT
}
}
context.startForegroundService(intent)
},
modifier = Modifier.fillMaxWidth()
) {
Text(
when (connectionState) {
ConnectionState.Disconnected, ConnectionState.Error -> "Connect"
else -> "Disconnect"
}
)
}
Spacer(modifier = Modifier.height(16.dp))
// Goal Input
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.spacedBy(8.dp)
) {
OutlinedTextField(
value = goalInput,
onValueChange = { goalInput = it },
label = { Text("Enter a goal...") },
modifier = Modifier.weight(1f),
enabled = connectionState == ConnectionState.Connected && goalStatus != GoalStatus.Running,
singleLine = true
)
Button(
onClick = {
if (goalInput.isNotBlank()) {
val intent = Intent(context, ConnectionService::class.java).apply {
action = ConnectionService.ACTION_SEND_GOAL
putExtra(ConnectionService.EXTRA_GOAL, goalInput)
}
context.startService(intent)
goalInput = ""
}
},
enabled = connectionState == ConnectionState.Connected
&& goalStatus != GoalStatus.Running
&& goalInput.isNotBlank()
) {
Text("Run")
}
}
// Current goal
if (currentGoal.isNotEmpty()) {
Spacer(modifier = Modifier.height(8.dp))
Text(
text = "Goal: $currentGoal",
style = MaterialTheme.typography.titleSmall,
color = MaterialTheme.colorScheme.primary
)
}
Spacer(modifier = Modifier.height(16.dp))
// Step Log
LazyColumn(
modifier = Modifier.weight(1f),
verticalArrangement = Arrangement.spacedBy(8.dp)
) {
items(steps) { step ->
Card(
modifier = Modifier.fillMaxWidth()
) {
Column(modifier = Modifier.padding(12.dp)) {
Text(
text = "Step ${step.step}: ${step.action}",
style = MaterialTheme.typography.titleSmall
)
if (step.reasoning.isNotEmpty()) {
Text(
text = step.reasoning,
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
}
}
}
}
}
// Goal Status
if (goalStatus == GoalStatus.Completed || goalStatus == GoalStatus.Failed) {
Spacer(modifier = Modifier.height(8.dp))
Text(
text = if (goalStatus == GoalStatus.Completed) {
"Goal completed (${steps.size} steps)"
} else {
"Goal failed"
},
style = MaterialTheme.typography.titleMedium,
color = if (goalStatus == GoalStatus.Completed) {
Color(0xFF4CAF50)
} else {
MaterialTheme.colorScheme.error
}
)
}
}
}
Step 2: Rewrite MainActivity.kt with bottom nav
package com.thisux.droidclaw
import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.activity.enableEdgeToEdge
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.History
import androidx.compose.material.icons.filled.Home
import androidx.compose.material.icons.filled.Settings
import androidx.compose.material3.Icon
import androidx.compose.material3.NavigationBar
import androidx.compose.material3.NavigationBarItem
import androidx.compose.material3.Scaffold
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.getValue
import androidx.compose.ui.Modifier
import androidx.navigation.NavDestination.Companion.hierarchy
import androidx.navigation.NavGraph.Companion.findStartDestination
import androidx.navigation.compose.NavHost
import androidx.navigation.compose.composable
import androidx.navigation.compose.currentBackStackEntryAsState
import androidx.navigation.compose.rememberNavController
import com.thisux.droidclaw.ui.screens.HomeScreen
import com.thisux.droidclaw.ui.screens.LogsScreen
import com.thisux.droidclaw.ui.screens.SettingsScreen
import com.thisux.droidclaw.ui.theme.DroidClawTheme
sealed class Screen(val route: String, val label: String) {
data object Home : Screen("home", "Home")
data object Settings : Screen("settings", "Settings")
data object Logs : Screen("logs", "Logs")
}
class MainActivity : ComponentActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
enableEdgeToEdge()
setContent {
DroidClawTheme {
MainNavigation()
}
}
}
}
@Composable
fun MainNavigation() {
val navController = rememberNavController()
val screens = listOf(Screen.Home, Screen.Settings, Screen.Logs)
Scaffold(
modifier = Modifier.fillMaxSize(),
bottomBar = {
NavigationBar {
val navBackStackEntry by navController.currentBackStackEntryAsState()
val currentDestination = navBackStackEntry?.destination
screens.forEach { screen ->
NavigationBarItem(
icon = {
Icon(
when (screen) {
is Screen.Home -> Icons.Filled.Home
is Screen.Settings -> Icons.Filled.Settings
is Screen.Logs -> Icons.Filled.History
},
contentDescription = screen.label
)
},
label = { Text(screen.label) },
selected = currentDestination?.hierarchy?.any { it.route == screen.route } == true,
onClick = {
navController.navigate(screen.route) {
popUpTo(navController.graph.findStartDestination().id) {
saveState = true
}
launchSingleTop = true
restoreState = true
}
}
)
}
}
}
) { innerPadding ->
NavHost(
navController = navController,
startDestination = Screen.Home.route,
modifier = Modifier.padding(innerPadding)
) {
composable(Screen.Home.route) { HomeScreen() }
composable(Screen.Settings.route) { SettingsScreen() }
composable(Screen.Logs.route) { LogsScreen() }
}
}
}
Step 3: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 4: Commit
git add android/app/src/main/java/com/thisux/droidclaw/ui/screens/HomeScreen.kt android/app/src/main/java/com/thisux/droidclaw/MainActivity.kt
git commit -m "feat(android): add HomeScreen with goal input and bottom nav"
Task 11: UI — SettingsScreen
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/ui/screens/SettingsScreen.kt - Create:
android/app/src/main/java/com/thisux/droidclaw/util/BatteryOptimization.kt
Step 1: Create BatteryOptimization.kt
package com.thisux.droidclaw.util
import android.content.Context
import android.content.Intent
import android.net.Uri
import android.os.PowerManager
import android.provider.Settings
object BatteryOptimization {
fun isIgnoringBatteryOptimizations(context: Context): Boolean {
val pm = context.getSystemService(Context.POWER_SERVICE) as PowerManager
return pm.isIgnoringBatteryOptimizations(context.packageName)
}
fun requestExemption(context: Context) {
val intent = Intent(Settings.ACTION_REQUEST_IGNORE_BATTERY_OPTIMIZATIONS).apply {
data = Uri.parse("package:${context.packageName}")
}
context.startActivity(intent)
}
fun openAccessibilitySettings(context: Context) {
context.startActivity(Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS))
}
}
Step 2: Create SettingsScreen.kt
package com.thisux.droidclaw.ui.screens
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.verticalScroll
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.CheckCircle
import androidx.compose.material.icons.filled.Error
import androidx.compose.material3.Card
import androidx.compose.material3.CardDefaults
import androidx.compose.material3.Icon
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.OutlinedButton
import androidx.compose.material3.OutlinedTextField
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.text.input.PasswordVisualTransformation
import androidx.compose.ui.unit.dp
import com.thisux.droidclaw.DroidClawApp
import com.thisux.droidclaw.accessibility.DroidClawAccessibilityService
import com.thisux.droidclaw.capture.ScreenCaptureManager
import com.thisux.droidclaw.util.BatteryOptimization
import kotlinx.coroutines.launch
@Composable
fun SettingsScreen() {
val context = LocalContext.current
val app = context.applicationContext as DroidClawApp
val scope = rememberCoroutineScope()
val apiKey by app.settingsStore.apiKey.collectAsState(initial = "")
val serverUrl by app.settingsStore.serverUrl.collectAsState(initial = "wss://localhost:8080")
var editingApiKey by remember(apiKey) { mutableStateOf(apiKey) }
var editingServerUrl by remember(serverUrl) { mutableStateOf(serverUrl) }
val isAccessibilityEnabled by DroidClawAccessibilityService.isRunning.collectAsState()
val isCaptureAvailable by ScreenCaptureManager.isAvailable.collectAsState()
val isBatteryExempt = remember { BatteryOptimization.isIgnoringBatteryOptimizations(context) }
Column(
modifier = Modifier
.fillMaxSize()
.padding(16.dp)
.verticalScroll(rememberScrollState()),
verticalArrangement = Arrangement.spacedBy(16.dp)
) {
Text("Settings", style = MaterialTheme.typography.headlineMedium)
// API Key
OutlinedTextField(
value = editingApiKey,
onValueChange = { editingApiKey = it },
label = { Text("API Key") },
modifier = Modifier.fillMaxWidth(),
visualTransformation = PasswordVisualTransformation(),
singleLine = true
)
if (editingApiKey != apiKey) {
OutlinedButton(
onClick = { scope.launch { app.settingsStore.setApiKey(editingApiKey) } }
) {
Text("Save API Key")
}
}
// Server URL
OutlinedTextField(
value = editingServerUrl,
onValueChange = { editingServerUrl = it },
label = { Text("Server URL") },
modifier = Modifier.fillMaxWidth(),
singleLine = true
)
if (editingServerUrl != serverUrl) {
OutlinedButton(
onClick = { scope.launch { app.settingsStore.setServerUrl(editingServerUrl) } }
) {
Text("Save Server URL")
}
}
Spacer(modifier = Modifier.height(8.dp))
// Setup Checklist
Text("Setup Checklist", style = MaterialTheme.typography.titleMedium)
ChecklistItem(
label = "API key configured",
isOk = apiKey.isNotBlank(),
actionLabel = null,
onAction = {}
)
ChecklistItem(
label = "Accessibility service",
isOk = isAccessibilityEnabled,
actionLabel = "Enable",
onAction = { BatteryOptimization.openAccessibilitySettings(context) }
)
ChecklistItem(
label = "Screen capture permission",
isOk = isCaptureAvailable,
actionLabel = null,
onAction = {}
)
ChecklistItem(
label = "Battery optimization disabled",
isOk = isBatteryExempt,
actionLabel = "Disable",
onAction = { BatteryOptimization.requestExemption(context) }
)
}
}
@Composable
private fun ChecklistItem(
label: String,
isOk: Boolean,
actionLabel: String?,
onAction: () -> Unit
) {
Card(
modifier = Modifier.fillMaxWidth(),
colors = CardDefaults.cardColors(
containerColor = if (isOk) {
MaterialTheme.colorScheme.secondaryContainer
} else {
MaterialTheme.colorScheme.errorContainer.copy(alpha = 0.3f)
}
)
) {
Row(
modifier = Modifier
.fillMaxWidth()
.padding(12.dp),
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.SpaceBetween
) {
Row(
verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(8.dp)
) {
Icon(
imageVector = if (isOk) Icons.Filled.CheckCircle else Icons.Filled.Error,
contentDescription = if (isOk) "OK" else "Missing",
tint = if (isOk) Color(0xFF4CAF50) else MaterialTheme.colorScheme.error
)
Text(label)
}
if (!isOk && actionLabel != null) {
OutlinedButton(onClick = onAction) {
Text(actionLabel)
}
}
}
}
}
Step 3: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 4: Commit
git add android/app/src/main/java/com/thisux/droidclaw/ui/screens/SettingsScreen.kt android/app/src/main/java/com/thisux/droidclaw/util/BatteryOptimization.kt
git commit -m "feat(android): add SettingsScreen with checklist and BatteryOptimization util"
Task 12: UI — LogsScreen
Files:
- Create:
android/app/src/main/java/com/thisux/droidclaw/ui/screens/LogsScreen.kt
Step 1: Create LogsScreen.kt
package com.thisux.droidclaw.ui.screens
import androidx.compose.foundation.clickable
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.lazy.LazyColumn
import androidx.compose.foundation.lazy.items
import androidx.compose.material3.Card
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.collectAsState
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.unit.dp
import com.thisux.droidclaw.connection.ConnectionService
import com.thisux.droidclaw.model.GoalStatus
@Composable
fun LogsScreen() {
val steps by ConnectionService.currentSteps.collectAsState()
val goalStatus by ConnectionService.currentGoalStatus.collectAsState()
val currentGoal by ConnectionService.currentGoal.collectAsState()
Column(
modifier = Modifier
.fillMaxSize()
.padding(16.dp)
) {
Text("Logs", style = MaterialTheme.typography.headlineMedium)
if (currentGoal.isNotEmpty()) {
Row(
modifier = Modifier
.fillMaxWidth()
.padding(vertical = 8.dp),
horizontalArrangement = Arrangement.SpaceBetween
) {
Text(
text = currentGoal,
style = MaterialTheme.typography.titleSmall
)
Text(
text = when (goalStatus) {
GoalStatus.Running -> "Running"
GoalStatus.Completed -> "Completed"
GoalStatus.Failed -> "Failed"
GoalStatus.Idle -> "Idle"
},
color = when (goalStatus) {
GoalStatus.Running -> Color(0xFFFFC107)
GoalStatus.Completed -> Color(0xFF4CAF50)
GoalStatus.Failed -> MaterialTheme.colorScheme.error
GoalStatus.Idle -> Color.Gray
}
)
}
}
if (steps.isEmpty()) {
Text(
text = "No steps recorded yet. Submit a goal to see agent activity here.",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant,
modifier = Modifier.padding(top = 16.dp)
)
} else {
LazyColumn(
modifier = Modifier.weight(1f),
verticalArrangement = Arrangement.spacedBy(8.dp)
) {
items(steps) { step ->
var expanded by remember { mutableStateOf(false) }
Card(
modifier = Modifier
.fillMaxWidth()
.clickable { expanded = !expanded }
) {
Column(modifier = Modifier.padding(12.dp)) {
Text(
text = "Step ${step.step}: ${step.action}",
style = MaterialTheme.typography.titleSmall
)
if (expanded && step.reasoning.isNotEmpty()) {
Text(
text = step.reasoning,
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant,
modifier = Modifier.padding(top = 4.dp)
)
}
}
}
}
}
}
}
}
Step 2: Verify build compiles
Run: cd android && ./gradlew assembleDebug
Expected: BUILD SUCCESSFUL
Step 3: Commit
git add android/app/src/main/java/com/thisux/droidclaw/ui/screens/LogsScreen.kt
git commit -m "feat(android): add LogsScreen with expandable step cards"
Task 13: Final Integration & Build Verification
Files:
- All previously created files
Step 1: Full clean build
Run: cd android && ./gradlew clean assembleDebug
Expected: BUILD SUCCESSFUL with 0 errors
Step 2: Verify APK exists
Run: ls -la android/app/build/outputs/apk/debug/app-debug.apk
Expected: File exists
Step 3: Commit all remaining changes
cd android && git add -A && git status
git commit -m "feat(android): complete DroidClaw v1 companion app
- Accessibility service with ScreenTreeBuilder and GestureExecutor
- Ktor WebSocket with reliable reconnection and auth handshake
- Foreground ConnectionService with notification
- MediaProjection screen capture with vision fallback
- DataStore settings for API key, server URL
- Compose UI with bottom nav (Home, Settings, Logs)
- Home: connection status, goal input, live step log
- Settings: API key, server URL, setup checklist
- Logs: expandable step cards with reasoning"
Execution Notes
Build requirement: The Android project requires Android Studio or at minimum the Android SDK with API 36 installed. Gradle commands run via ./gradlew wrapper in the android/ directory.
Testing on device: After building, install via adb install android/app/build/outputs/apk/debug/app-debug.apk. Then:
- Open Settings > Accessibility > DroidClaw and enable the service
- Open the app > Settings tab > enter API key and server URL
- Go to Home tab > tap Connect
- Enter a goal and tap Run
Not covered in v1 (future work):
- Persistent session history (LogsScreen clears on restart)
- MediaProjection consent flow wired through UI (currently needs manual setup)
- Auto-connect on boot
- OEM-specific battery guidance (dontkillmyapp.com links)