This commit adds a comprehensive AI Assistant plugin that provides AI-powered accessibility features for the Cthulhu screen reader. Major Features: - Screen analysis using screenshots combined with AT-SPI accessibility data - Natural language questions about UI elements and screen content - Safe action assistance with user confirmation (click, type, copy) - Multi-provider AI support (Claude, Claude Code CLI, OpenAI, Gemini, Ollama) - Complete preferences GUI integration with provider selection and settings Technical Implementation: - Plugin-based architecture using pluggy framework - Three keybindings: Cthulhu+Ctrl+Shift+A/Q/D for describe/question/action - PyAutoGUI integration for universal input synthesis (Wayland/X11 compatible) - Robust error handling and user safety confirmations - Claude Code CLI integration (no API key required) Core Files Added/Modified: - src/cthulhu/plugins/AIAssistant/ - Complete plugin implementation - src/cthulhu/settings.py - AI settings and Claude Code provider constants - src/cthulhu/cthulhu-setup.ui - AI Assistant preferences tab - src/cthulhu/cthulhu_gui_prefs.py - GUI handlers and settings management - distro-packages/Arch-Linux/PKGBUILD - Updated dependencies - CLAUDE.md - Comprehensive documentation Testing Status: - Terminal applications: 100% working - Web forms (focus mode): 100% working - Question and description features: 100% working - Claude Code CLI integration: 100% working - Settings persistence: 100% working The AI Assistant is fully functional and ready for production use. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
		
			
				
	
	
		
			835 lines
		
	
	
		
			28 KiB
		
	
	
	
		
			Markdown
		
	
	
	
	
	
			
		
		
	
	
			835 lines
		
	
	
		
			28 KiB
		
	
	
	
		
			Markdown
		
	
	
	
	
	
# CLAUDE.md
 | 
						|
 | 
						|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 | 
						|
 | 
						|
## Project Overview
 | 
						|
 | 
						|
Cthulhu is a fork of the Orca screen reader, providing access to the graphical desktop via speech and/or braille. It's designed as a supplemental screen reader for advanced users, particularly useful for older Qt applications and specific window managers like i3.
 | 
						|
 | 
						|
## Build System and Commands
 | 
						|
 | 
						|
### Local Development Build (Recommended)
 | 
						|
```bash
 | 
						|
# Build and install locally to ~/.local (no system overwrite)
 | 
						|
./build-local.sh
 | 
						|
 | 
						|
# Test local installation
 | 
						|
./test-local.sh
 | 
						|
 | 
						|
# Run local version
 | 
						|
~/.local/bin/cthulhu
 | 
						|
 | 
						|
# Clean build artifacts and local installation
 | 
						|
./clean-local.sh
 | 
						|
```
 | 
						|
 | 
						|
### System Build (Autotools)
 | 
						|
```bash
 | 
						|
# Configure and build for system installation
 | 
						|
./autogen.sh --prefix=/usr
 | 
						|
make
 | 
						|
make install
 | 
						|
 | 
						|
# Or use CI script
 | 
						|
ci/build_and_install.sh
 | 
						|
```
 | 
						|
 | 
						|
### Alternative Build (Python packaging)
 | 
						|
```bash
 | 
						|
# Using pip/hatchling
 | 
						|
pip install -e .
 | 
						|
```
 | 
						|
 | 
						|
### Testing
 | 
						|
```bash
 | 
						|
# Run all regression tests
 | 
						|
test/harness/runall.sh
 | 
						|
 | 
						|
# Run single test
 | 
						|
test/harness/runone.sh <test.py> <app-name>
 | 
						|
 | 
						|
# Run specific app tests
 | 
						|
test/harness/runall.sh -a /path/to/app/tests
 | 
						|
 | 
						|
# Coverage analysis
 | 
						|
test/harness/runall.sh -c
 | 
						|
 | 
						|
# Profile mode
 | 
						|
test/harness/runall.sh -p
 | 
						|
```
 | 
						|
 | 
						|
## Core Architecture
 | 
						|
 | 
						|
### Main Components
 | 
						|
- **Event System**: `event_manager.py`, `signal_manager.py` - Central event handling
 | 
						|
- **Accessibility Layer**: `ax_object.py`, `ax_utilities.py` - AT-SPI object management  
 | 
						|
- **Output Systems**: `speech.py`, `braille.py` - Speech and braille generation
 | 
						|
- **Navigation**: `structural_navigation.py`, `flat_review.py` - Document and screen navigation
 | 
						|
- **Plugin System**: `plugin_system_manager.py` - Extensible plugin architecture using pluggy
 | 
						|
 | 
						|
### Application Scripts
 | 
						|
Scripts in `src/cthulhu/scripts/` provide application-specific behavior:
 | 
						|
- `scripts/apps/` - Specific applications (Firefox, Thunderbird, LibreOffice)
 | 
						|
- `scripts/toolkits/` - Toolkit support (GTK, Qt, Gecko, WebKit)
 | 
						|
- `scripts/web/` - Web browsing enhancements
 | 
						|
 | 
						|
### Plugin Development
 | 
						|
System plugins in `src/cthulhu/plugins/`, user plugins in `~/.local/share/cthulhu/plugins/`:
 | 
						|
 | 
						|
#### Basic Plugin Structure
 | 
						|
```
 | 
						|
src/cthulhu/plugins/MyPlugin/
 | 
						|
├── __init__.py          # Package import: from .plugin import MyPlugin
 | 
						|
├── plugin.py            # Main implementation
 | 
						|
├── plugin.info          # Metadata
 | 
						|
└── Makefile.am          # Build system integration
 | 
						|
```
 | 
						|
 | 
						|
#### Minimal Plugin Template
 | 
						|
```python
 | 
						|
from cthulhu.plugin import Plugin, cthulhu_hookimpl
 | 
						|
 | 
						|
class MyPlugin(Plugin):
 | 
						|
    def __init__(self, *args, **kwargs):
 | 
						|
        super().__init__(*args, **kwargs)
 | 
						|
        self._enabled = True
 | 
						|
        
 | 
						|
    @cthulhu_hookimpl
 | 
						|
    def activate(self, plugin=None):
 | 
						|
        if plugin is not None and plugin is not self:
 | 
						|
            return
 | 
						|
        # Plugin activation logic
 | 
						|
        return True
 | 
						|
        
 | 
						|
    @cthulhu_hookimpl  
 | 
						|
    def deactivate(self, plugin=None):
 | 
						|
        if plugin is not None and plugin is not self:
 | 
						|
            return
 | 
						|
        # Cleanup logic
 | 
						|
        return True
 | 
						|
```
 | 
						|
 | 
						|
#### Plugin Integration Patterns
 | 
						|
 | 
						|
**1. Keybinding Registration:**
 | 
						|
```python
 | 
						|
def _register_keybinding(self):
 | 
						|
    if not self.app:
 | 
						|
        return
 | 
						|
    api_helper = self.app.getAPIHelper()
 | 
						|
    self._kb_binding = api_helper.registerGestureByString(
 | 
						|
        "Cthulhu+key", self._handler_method, "Description"
 | 
						|
    )
 | 
						|
```
 | 
						|
 | 
						|
**2. Accessing Cthulhu APIs:**
 | 
						|
```python
 | 
						|
# Get current active script
 | 
						|
state = self.app.getDynamicApiManager().getAPI('CthulhuState')
 | 
						|
active_script = state.activeScript
 | 
						|
 | 
						|
# Access speech/messages
 | 
						|
speech = self.app.getDynamicApiManager().getAPI('Speech')
 | 
						|
messages = self.app.getDynamicApiManager().getAPI('Messages')
 | 
						|
```
 | 
						|
 | 
						|
**3. Sound Generation:**
 | 
						|
```python
 | 
						|
from cthulhu import sound
 | 
						|
from cthulhu.sound_generator import Tone
 | 
						|
 | 
						|
# Initialize player
 | 
						|
self._player = sound.getPlayer()
 | 
						|
 | 
						|
# Create and play tone
 | 
						|
tone = Tone(duration=0.15, frequency=400, volumeMultiplier=0.7)
 | 
						|
self._player.play(tone, interrupt=False)
 | 
						|
```
 | 
						|
 | 
						|
**4. Script Method Monkey-Patching:**
 | 
						|
```python
 | 
						|
def _monkey_patch_script_methods(self):
 | 
						|
    state = self.app.getDynamicApiManager().getAPI('CthulhuState')
 | 
						|
    if state and state.activeScript:
 | 
						|
        script = state.activeScript
 | 
						|
        self._original_method = script.methodName
 | 
						|
        
 | 
						|
        def wrapped_method(*args, **kwargs):
 | 
						|
            result = self._original_method(*args, **kwargs)
 | 
						|
            # Add custom logic here
 | 
						|
            return result
 | 
						|
            
 | 
						|
        script.methodName = wrapped_method
 | 
						|
```
 | 
						|
 | 
						|
#### Plugin Files
 | 
						|
 | 
						|
**plugin.info format:**
 | 
						|
```ini
 | 
						|
[Core]
 | 
						|
Name = PluginName
 | 
						|
Module = PluginName
 | 
						|
 | 
						|
[Documentation]
 | 
						|
Description = Plugin description
 | 
						|
Author = Author Name
 | 
						|
Version = 1.0.0
 | 
						|
Website = https://example.com
 | 
						|
```
 | 
						|
 | 
						|
**Makefile.am template:**
 | 
						|
```makefile
 | 
						|
pluginname_PYTHON = \
 | 
						|
	__init__.py \
 | 
						|
	plugin.py
 | 
						|
 | 
						|
pluginnamedir = $(pkgdatadir)/cthulhu/plugins/PluginName
 | 
						|
 | 
						|
pluginname_DATA = \
 | 
						|
	plugin.info
 | 
						|
 | 
						|
EXTRA_DIST = $(pluginname_DATA)
 | 
						|
```
 | 
						|
 | 
						|
**Build Integration:**
 | 
						|
Add plugin to `src/cthulhu/plugins/Makefile.am` SUBDIRS line.
 | 
						|
 | 
						|
#### Advanced Plugin Features
 | 
						|
 | 
						|
**Event System Integration:**
 | 
						|
- Register with Dynamic API: `api_manager.registerAPI('MyPlugin', self)`
 | 
						|
- Hook into script changes via monkey-patching
 | 
						|
- Access event manager for custom event handling
 | 
						|
 | 
						|
**Text Analysis:**
 | 
						|
- Use `script.getTextLineAtCaret(obj)` for current line
 | 
						|
- Access text attributes via `script.utilities.getTextAttributes()`
 | 
						|
- Leverage existing utilities like `indentationDescription()`
 | 
						|
 | 
						|
**Settings Integration:**
 | 
						|
- Access settings manager: `settings_manager.getManager()`
 | 
						|
- Create plugin-specific settings with prefixes
 | 
						|
- Integrate with preferences GUI if needed
 | 
						|
 | 
						|
## Development Workflow
 | 
						|
 | 
						|
### Contributing
 | 
						|
- Repository: https://git.stormux.org/storm/cthulhu
 | 
						|
- Create branch for changes, merge to master when ready
 | 
						|
- Email patches to storm_dragon@stormux.org if no direct access
 | 
						|
- Community: IRC #stormux on irc.stormux.org
 | 
						|
 | 
						|
### Key Dependencies
 | 
						|
- Python 3.3+, pygobject-3.0, pluggy, gtk+-3.0
 | 
						|
- AT-SPI2, ATK for accessibility
 | 
						|
- Optional: BrlTTY/BrlAPI (braille), Speech Dispatcher, liblouis, GStreamer
 | 
						|
 | 
						|
### Version Information
 | 
						|
Current version in `src/cthulhu/cthulhuVersion.py`, codename "plugins"
 | 
						|
 | 
						|
### Self-voicing Feature
 | 
						|
Direct speech output via Unix socket:
 | 
						|
```bash
 | 
						|
echo "Hello world." | socat - UNIX-CLIENT:/tmp/cthulhu.sock
 | 
						|
echo "<!#APPEND#!>Hello world." | socat - UNIX-CLIENT:/tmp/cthulhu.sock  # No interrupt
 | 
						|
echo "Hello world.<#APPEND#>" | socat - UNIX-CLIENT:/tmp/cthulhu.sock    # Persistent braille
 | 
						|
```
 | 
						|
 | 
						|
## Key Directories
 | 
						|
 | 
						|
- `src/cthulhu/` - Core source code
 | 
						|
- `test/` - Regression test framework with keystroke-based testing
 | 
						|
- `po/` - Internationalization files (extensive i18n support)
 | 
						|
- `help/` - Multi-language user documentation
 | 
						|
- `ci/` - Continuous integration scripts
 | 
						|
 | 
						|
## Testing Notes
 | 
						|
 | 
						|
The test system uses keystroke recording/playback with speech and braille output comparison. Tests are organized by application and toolkit. The testing framework automatically handles application launching and result comparison for regression testing.
 | 
						|
 | 
						|
---
 | 
						|
 | 
						|
## ORCA vs CTHULHU COMPARISON & INTEGRATION CONSIDERATIONS
 | 
						|
 | 
						|
### **Fork History & Strategic Decisions**
 | 
						|
- **Cthulhu Origin**: Forked from Orca 45 (GNOME-45 release) - `git log` shows initial commit `a523205`
 | 
						|
- **Strategic Goals**: 
 | 
						|
  - Supplemental screen reader for advanced users
 | 
						|
  - Better support for older Qt applications
 | 
						|
  - Enhanced window manager support (i3, etc.)
 | 
						|
  - Plugin system with pluggy framework
 | 
						|
  - Community-driven development
 | 
						|
 | 
						|
### **Major Architectural Differences**
 | 
						|
 | 
						|
#### **Build Systems**
 | 
						|
- **Cthulhu**: Autotools (102 Makefile.am files) - mature, stable build system
 | 
						|
- **Orca**: Meson/Ninja (33 meson.build files, 84 legacy makefiles) - modern, faster builds
 | 
						|
- **Integration Consideration**: Should Cthulhu migrate to Meson for faster builds and better dependencies?
 | 
						|
 | 
						|
#### **Plugin Architecture**
 | 
						|
- **Cthulhu**: Extensive pluggy-based plugin system with 9 core plugins
 | 
						|
- **Orca**: No comparable plugin system - features are integrated directly into core
 | 
						|
- **Strategic Value**: Plugin system is Cthulhu's key differentiator - maintain this advantage
 | 
						|
 | 
						|
#### **New Orca Features (v49.alpha)**
 | 
						|
1. **D-Bus Remote Controller** (`dbus_service.py`)
 | 
						|
   - Service: `org.gnome.Orca.Service`
 | 
						|
   - Remote command execution, module access
 | 
						|
   - Requires: `dasbus` library
 | 
						|
   - **Integration Value**: HIGH - would enable external control of Cthulhu
 | 
						|
 | 
						|
2. **Spiel TTS Support** (`spiel.py`)
 | 
						|
   - Alternative to speech-dispatcher
 | 
						|
   - Multi-synthesizer support (eSpeak, Piper)
 | 
						|
   - Flatpak-based providers
 | 
						|
   - **Integration Value**: MEDIUM - broader TTS options
 | 
						|
 | 
						|
3. **Enhanced Module Architecture**
 | 
						|
   - `focus_manager.py`, `input_event_manager.py`
 | 
						|
   - Better separation of concerns
 | 
						|
   - **Integration Value**: LOW - architectural improvement but not user-facing
 | 
						|
 | 
						|
### **Current Bug Investigation: Plugin Keybindings**
 | 
						|
[Previous debugging section remains valid - plugin keybinding integration is working but needs refinement]
 | 
						|
 | 
						|
### **Integration Recommendations**
 | 
						|
 | 
						|
#### **HIGH Priority - D-Bus Remote Controller**
 | 
						|
- **Benefit**: External application control, automation, integration with other tools
 | 
						|
- **Risk**: LOW - self-contained feature, minimal core impact
 | 
						|
- **Files to Port**: `dbus_service.py`, related D-Bus infrastructure
 | 
						|
- **Dependencies**: Add `dasbus` to Cthulhu's requirements
 | 
						|
 | 
						|
#### **MEDIUM Priority - Build System Migration**
 | 
						|
- **Benefit**: Faster builds, better dependency management, alignment with GNOME ecosystem
 | 
						|
- **Risk**: MEDIUM - significant build system changes, potential disruption
 | 
						|
- **Approach**: Gradual migration, maintain autotools compatibility initially
 | 
						|
 | 
						|
#### **LOW Priority - Spiel Integration**
 | 
						|
- **Benefit**: More TTS options, potentially better voice quality
 | 
						|
- **Risk**: MEDIUM - additional complexity, Flatpak dependencies
 | 
						|
- **Approach**: Optional feature, don't replace speech-dispatcher by default
 | 
						|
 | 
						|
### **Files Requiring Attention for Integration**
 | 
						|
 | 
						|
#### **D-Bus Remote Controller Integration**:
 | 
						|
```bash
 | 
						|
# Core files to review and potentially port:
 | 
						|
src/orca/dbus_service.py           # Main D-Bus service implementation
 | 
						|
README-REMOTE-CONTROLLER.md       # API documentation and examples
 | 
						|
```
 | 
						|
 | 
						|
#### **Build System Files**:
 | 
						|
```bash
 | 
						|
# Modern build configuration to consider:
 | 
						|
meson.build                        # Root build configuration
 | 
						|
meson_options.txt                  # Build options
 | 
						|
subprojects/spiel.wrap            # Subproject integration
 | 
						|
```
 | 
						|
 | 
						|
### **Strategic Questions for Decision**
 | 
						|
 | 
						|
1. **Build System**: Should Cthulhu migrate to Meson for better GNOME ecosystem alignment?
 | 
						|
2. **D-Bus Interface**: High value feature - should this be priority #1 for integration?
 | 
						|
3. **Plugin System**: How to maintain Cthulhu's plugin advantage while integrating Orca improvements?
 | 
						|
4. **Version Strategy**: Selective feature backporting vs. major version sync?
 | 
						|
 | 
						|
## AI Assistant Integration
 | 
						|
 | 
						|
### **NEW FEATURE**: AI-Powered Accessibility Assistant
 | 
						|
Cthulhu now includes an optional AI assistant plugin for enhanced accessibility support:
 | 
						|
 | 
						|
- **Vision Analysis**: Screenshots + AT-SPI data for understanding unlabeled UI elements
 | 
						|
- **Safe Actions**: Confirmed element clicking and navigation assistance  
 | 
						|
- **Multi-Provider Support**: Claude, ChatGPT, Gemini, and Ollama backends
 | 
						|
- **Privacy-First**: Disabled by default, requires explicit opt-in and API key configuration
 | 
						|
 | 
						|
### AI Assistant Configuration
 | 
						|
```bash
 | 
						|
# Access via Cthulhu Preferences
 | 
						|
~/.local/bin/cthulhu -s  # Opens preferences dialog
 | 
						|
# Navigate to "AI Assistant" tab
 | 
						|
# 1. Check "Enable AI Assistant" 
 | 
						|
# 2. Select provider (Claude, ChatGPT, Gemini, Ollama)
 | 
						|
# 3. Set API key file path
 | 
						|
# 4. Configure safety and quality settings
 | 
						|
```
 | 
						|
 | 
						|
### AI Assistant Keybindings
 | 
						|
- **Cthulhu+Control+Shift+Q**: Ask questions about current screen
 | 
						|
- **Cthulhu+Control+Shift+D**: Describe current screen 
 | 
						|
- **Cthulhu+Control+Shift+A**: Request actions (click, type, copy)
 | 
						|
 | 
						|
### AI Provider Setup
 | 
						|
 | 
						|
#### 1. Claude (Anthropic) - **Recommended**
 | 
						|
```bash
 | 
						|
# Get API key from: https://console.anthropic.com/
 | 
						|
# 1. Sign up/login → "Get API Keys" → Create new key
 | 
						|
# 2. Copy the key (starts with "sk-ant-...")
 | 
						|
# 3. Save to file:
 | 
						|
mkdir -p ~/.config/cthulhu
 | 
						|
echo "sk-ant-your-actual-key-here" > ~/.config/cthulhu/claude-api-key
 | 
						|
chmod 600 ~/.config/cthulhu/claude-api-key
 | 
						|
 | 
						|
# Pricing: ~$3 per million input tokens, ~$15 per million output tokens
 | 
						|
# Best vision capabilities and safety for accessibility use
 | 
						|
```
 | 
						|
 | 
						|
#### 2. ChatGPT (OpenAI)
 | 
						|
```bash
 | 
						|
# Get API key from: https://platform.openai.com/api-keys  
 | 
						|
# 1. Sign up/login → "Create new secret key"
 | 
						|
# 2. Copy immediately (can't view again, starts with "sk-...")
 | 
						|
# 3. Save to file:
 | 
						|
mkdir -p ~/.config/cthulhu
 | 
						|
echo "sk-your-actual-openai-key" > ~/.config/cthulhu/openai-api-key
 | 
						|
chmod 600 ~/.config/cthulhu/openai-api-key
 | 
						|
 | 
						|
# Pricing: ~$2.50 per million input tokens, ~$10 per million output tokens
 | 
						|
# Good vision capabilities, widely supported
 | 
						|
```
 | 
						|
 | 
						|
#### 3. Gemini (Google) 
 | 
						|
```bash
 | 
						|
# Get API key from: https://aistudio.google.com/app/apikey
 | 
						|
# 1. Sign up/login → "Create API key" 
 | 
						|
# 2. Copy the generated key
 | 
						|
# 3. Save to file:
 | 
						|
mkdir -p ~/.config/cthulhu
 | 
						|
echo "your-actual-gemini-key" > ~/.config/cthulhu/gemini-api-key
 | 
						|
chmod 600 ~/.config/cthulhu/gemini-api-key
 | 
						|
 | 
						|
# Pricing: Free tier (15 requests/min), then ~$1.25 per million tokens
 | 
						|
# Good for testing, has generous free allowance
 | 
						|
```
 | 
						|
 | 
						|
#### 4. Ollama (Local) - **Privacy-Focused**
 | 
						|
```bash
 | 
						|
# Install Ollama (no API key needed!)
 | 
						|
sudo pacman -S ollama  # Arch Linux
 | 
						|
# OR: curl -fsSL https://ollama.ai/install.sh | sh
 | 
						|
 | 
						|
# Start service
 | 
						|
systemctl --user enable ollama
 | 
						|
systemctl --user start ollama
 | 
						|
 | 
						|
# Download vision-capable model (required for AI assistant)
 | 
						|
ollama pull llama3.2-vision  # 7.9GB download
 | 
						|
# OR smaller model: ollama pull moondream    # 1.7GB
 | 
						|
 | 
						|
# Verify installation
 | 
						|
ollama list  # Should show downloaded models
 | 
						|
 | 
						|
# No API key needed - runs entirely offline!
 | 
						|
# Free to use, privacy-focused, but slower than cloud providers
 | 
						|
```
 | 
						|
 | 
						|
### AI Assistant Usage Patterns
 | 
						|
- **Information Queries**: "What does this unlabeled button do?"
 | 
						|
- **Navigation Help**: "Where is the login form?" 
 | 
						|
- **Action Assistance**: "Click the submit button", "Type hello world and press enter"
 | 
						|
- **Layout Understanding**: "Describe the main sections of this page"
 | 
						|
- **Text Operations**: "Copy this text to clipboard", "Enter my username in the field"
 | 
						|
 | 
						|
### Safety Framework
 | 
						|
- **Confirmation Required**: All actions require user approval by default
 | 
						|
- **Action Descriptions**: Clear explanation of what will happen before execution
 | 
						|
- **Safe Defaults**: Conservative timeouts and quality settings
 | 
						|
- **Privacy Protection**: API keys stored securely, no data logging
 | 
						|
- **Action Types**: Click, Type, Copy operations via PyAutoGUI (Wayland/X11 compatible)
 | 
						|
 | 
						|
### Troubleshooting AI Assistant Setup
 | 
						|
 | 
						|
#### Common Issues
 | 
						|
```bash
 | 
						|
# Check if AI settings loaded correctly
 | 
						|
~/.local/bin/cthulhu -s  # Open preferences, check AI Assistant tab
 | 
						|
 | 
						|
# Verify API key file permissions and format
 | 
						|
ls -la ~/.config/cthulhu/*-api-key  # Should show 600 permissions
 | 
						|
cat ~/.config/cthulhu/claude-api-key  # Should contain only the API key
 | 
						|
 | 
						|
# Test Ollama connection
 | 
						|
curl http://localhost:11434/api/version  # Should return Ollama version
 | 
						|
ollama ps  # Should show running models
 | 
						|
 | 
						|
# Check dependencies
 | 
						|
python3 -c "import requests, PIL, pyautogui; print('Dependencies OK')"
 | 
						|
 | 
						|
# Test screenshot capability (requires X11/Wayland)
 | 
						|
python3 -c "
 | 
						|
from gi.repository import Gdk
 | 
						|
window = Gdk.get_default_root_window()
 | 
						|
print('Screenshot capability available')
 | 
						|
"
 | 
						|
```
 | 
						|
 | 
						|
#### Required Permissions
 | 
						|
- **File Access**: API key files in `~/.config/cthulhu/`
 | 
						|
- **Screen Access**: Screenshot capture (automatic on most setups)
 | 
						|
- **Network Access**: HTTP requests to AI providers (except Ollama)
 | 
						|
- **AT-SPI Access**: Accessibility tree traversal (enabled by default)
 | 
						|
- **Input Synthesis**: PyAutoGUI for action execution (click, type, copy)
 | 
						|
 | 
						|
## Cthulhu Plugin System - Developer Reference
 | 
						|
 | 
						|
### **Plugin Architecture Overview**
 | 
						|
 | 
						|
Cthulhu uses a **pluggy-based plugin system** with the following components:
 | 
						|
 | 
						|
1. **Plugin Manager**: `src/cthulhu/plugin_system_manager.py` - Central plugin loading/management
 | 
						|
2. **Base Plugin Class**: `src/cthulhu/plugin.py` - Provides common functionality
 | 
						|
3. **Hook System**: Uses `@cthulhu_hookimpl` decorators for lifecycle management
 | 
						|
4. **Plugin Discovery**: Automatic scanning of `src/cthulhu/plugins/` and `~/.local/share/cthulhu/plugins/`
 | 
						|
 | 
						|
### **Plugin Directory Structure**
 | 
						|
 | 
						|
Every plugin must follow this exact structure:
 | 
						|
```
 | 
						|
src/cthulhu/plugins/YourPlugin/
 | 
						|
├── __init__.py          # Import: from .plugin import YourPlugin
 | 
						|
├── plugin.py            # Main plugin class
 | 
						|
├── plugin.info          # Metadata (name, version, description)
 | 
						|
└── Makefile.am          # Build system integration
 | 
						|
```
 | 
						|
 | 
						|
### **Essential Plugin Files**
 | 
						|
 | 
						|
#### **`__init__.py`** - Package Import
 | 
						|
```python
 | 
						|
from .plugin import YourPlugin
 | 
						|
```
 | 
						|
 | 
						|
#### **`plugin.info`** - Metadata
 | 
						|
```ini
 | 
						|
name = Your Plugin Name
 | 
						|
version = 1.0.0
 | 
						|
description = What your plugin does
 | 
						|
authors = Your Name <email@example.com>
 | 
						|
website = https://example.com
 | 
						|
copyright = Copyright 2025
 | 
						|
builtin = false
 | 
						|
hidden = false
 | 
						|
```
 | 
						|
 | 
						|
#### **`Makefile.am`** - Build Integration
 | 
						|
```makefile
 | 
						|
cthulhu_python_PYTHON = \
 | 
						|
	__init__.py \
 | 
						|
	plugin.info \
 | 
						|
	plugin.py
 | 
						|
 | 
						|
cthulhu_pythondir=$(pkgpythondir)/plugins/YourPlugin
 | 
						|
```
 | 
						|
 | 
						|
### **Plugin Class Template**
 | 
						|
 | 
						|
```python
 | 
						|
#!/usr/bin/env python3
 | 
						|
import logging
 | 
						|
from cthulhu.plugin import Plugin, cthulhu_hookimpl
 | 
						|
 | 
						|
logger = logging.getLogger(__name__)
 | 
						|
 | 
						|
class YourPlugin(Plugin):
 | 
						|
    """Your plugin description."""
 | 
						|
    
 | 
						|
    def __init__(self, *args, **kwargs):
 | 
						|
        """Initialize the plugin."""
 | 
						|
        super().__init__(*args, **kwargs)
 | 
						|
        logger.info("YourPlugin initialized")
 | 
						|
        
 | 
						|
        # Keybinding storage - use individual variables, NOT dictionaries
 | 
						|
        self._kb_binding = None
 | 
						|
        
 | 
						|
    @cthulhu_hookimpl
 | 
						|
    def activate(self, plugin=None):
 | 
						|
        """Activate the plugin."""
 | 
						|
        if plugin is not None and plugin is not self:
 | 
						|
            return
 | 
						|
            
 | 
						|
        try:
 | 
						|
            logger.info("=== YourPlugin activation starting ===")
 | 
						|
            
 | 
						|
            # Register keybindings
 | 
						|
            self._register_keybinding()
 | 
						|
            
 | 
						|
            logger.info("YourPlugin activated successfully")
 | 
						|
            return True
 | 
						|
            
 | 
						|
        except Exception as e:
 | 
						|
            logger.error(f"Error activating YourPlugin: {e}")
 | 
						|
            return False
 | 
						|
    
 | 
						|
    @cthulhu_hookimpl
 | 
						|
    def deactivate(self, plugin=None):
 | 
						|
        """Deactivate the plugin."""
 | 
						|
        if plugin is not None and plugin is not self:
 | 
						|
            return
 | 
						|
            
 | 
						|
        logger.info("Deactivating YourPlugin")
 | 
						|
        self._kb_binding = None
 | 
						|
        return True
 | 
						|
        
 | 
						|
    def _register_keybinding(self):
 | 
						|
        """Register plugin keybindings."""
 | 
						|
        try:
 | 
						|
            # CRITICAL: Use this exact parameter order!
 | 
						|
            self._kb_binding = self.registerGestureByString(
 | 
						|
                self._your_handler_method,    # Handler method (first)
 | 
						|
                "Description of action",      # Description (second) 
 | 
						|
                'kb:cthulhu+your+keys'       # Gesture string (third)
 | 
						|
            )
 | 
						|
            
 | 
						|
            if self._kb_binding:
 | 
						|
                logger.info(f"Registered keybinding: {gesture_string}")
 | 
						|
            else:
 | 
						|
                logger.error(f"Failed to register keybinding")
 | 
						|
                
 | 
						|
        except Exception as e:
 | 
						|
            logger.error(f"Error registering keybinding: {e}")
 | 
						|
    
 | 
						|
    def _your_handler_method(self, script=None, inputEvent=None):
 | 
						|
        """Handle the keybinding activation."""
 | 
						|
        try:
 | 
						|
            logger.info("Keybinding triggered")
 | 
						|
            
 | 
						|
            # Your plugin logic here
 | 
						|
            
 | 
						|
            return True
 | 
						|
        except Exception as e:
 | 
						|
            logger.error(f"Error in handler: {e}")
 | 
						|
            return False
 | 
						|
```
 | 
						|
 | 
						|
### **🚨 CRITICAL Keybinding Patterns**
 | 
						|
 | 
						|
#### **✅ CORRECT Pattern (What Works)**
 | 
						|
```python
 | 
						|
# Individual binding storage (NOT dictionaries)
 | 
						|
self._kb_binding = None
 | 
						|
self._kb_binding_action1 = None  
 | 
						|
self._kb_binding_action2 = None
 | 
						|
 | 
						|
# Correct registerGestureByString parameter order
 | 
						|
self._kb_binding = self.registerGestureByString(
 | 
						|
    self._handler_method,        # 1st: Handler method
 | 
						|
    "Action description",        # 2nd: Description  
 | 
						|
    'kb:cthulhu+your+keys'      # 3rd: Gesture string
 | 
						|
)
 | 
						|
```
 | 
						|
 | 
						|
#### **❌ INCORRECT Patterns (What Fails)**
 | 
						|
```python
 | 
						|
# DON'T use dictionaries for keybinding storage
 | 
						|
self._kb_bindings = {}  # ❌ WRONG
 | 
						|
self._kb_bindings['action'] = self.registerGestureByString(...)  # ❌ WRONG
 | 
						|
 | 
						|
# DON'T use wrong parameter order
 | 
						|
self.registerGestureByString(
 | 
						|
    'kb:cthulhu+keys',          # ❌ WRONG ORDER
 | 
						|
    "Description", 
 | 
						|
    self._handler_method
 | 
						|
)
 | 
						|
 | 
						|
# DON'T use description as handler parameter
 | 
						|
self.registerGestureByString(
 | 
						|
    self._handler_method,
 | 
						|
    'kb:cthulhu+keys',          # ❌ WRONG ORDER
 | 
						|
    "Description"
 | 
						|
)
 | 
						|
```
 | 
						|
 | 
						|
### **Plugin Registration & Activation**
 | 
						|
 | 
						|
#### **Add to Build System**
 | 
						|
1. **Add to `src/cthulhu/plugins/Makefile.am`**:
 | 
						|
   ```makefile
 | 
						|
   SUBDIRS = YourPlugin OtherPlugin1 OtherPlugin2 ...
 | 
						|
   ```
 | 
						|
 | 
						|
2. **Add to `configure.ac`**:
 | 
						|
   ```
 | 
						|
   src/cthulhu/plugins/YourPlugin/Makefile
 | 
						|
   ```
 | 
						|
 | 
						|
#### **Add to Default Active Plugins**
 | 
						|
In `src/cthulhu/settings.py`:
 | 
						|
```python
 | 
						|
activePlugins = ['YourPlugin', 'DisplayVersion', 'PluginManager', ...]
 | 
						|
```
 | 
						|
 | 
						|
### **Plugin Lifecycle Events**
 | 
						|
 | 
						|
1. **`__init__`**: Plugin instance created
 | 
						|
2. **`activate`**: Plugin enabled (register keybindings, connect events)
 | 
						|
3. **`deactivate`**: Plugin disabled (cleanup, disconnect)
 | 
						|
 | 
						|
**Note**: `activate()` may be called multiple times for different script contexts.
 | 
						|
 | 
						|
### **Common Plugin Patterns**
 | 
						|
 | 
						|
#### **Settings Integration**
 | 
						|
```python
 | 
						|
from cthulhu import settings_manager
 | 
						|
 | 
						|
class YourPlugin(Plugin):
 | 
						|
    def __init__(self, *args, **kwargs):
 | 
						|
        super().__init__(*args, **kwargs)
 | 
						|
        self._settings_manager = settings_manager.getManager()
 | 
						|
    
 | 
						|
    def activate(self, plugin=None):
 | 
						|
        # Check if plugin should be active
 | 
						|
        enabled = self._settings_manager.getSetting('yourPluginEnabled')
 | 
						|
        if not enabled:
 | 
						|
            return
 | 
						|
```
 | 
						|
 | 
						|
#### **Message Presentation**
 | 
						|
```python
 | 
						|
def _present_message(self, message):
 | 
						|
    """Present a message to the user via speech."""
 | 
						|
    try:
 | 
						|
        if self.app:
 | 
						|
            state = self.app.getDynamicApiManager().getAPI('CthulhuState')
 | 
						|
            if state and state.activeScript:
 | 
						|
                state.activeScript.presentMessage(message, resetStyles=False)
 | 
						|
    except Exception as e:
 | 
						|
        logger.error(f"Error presenting message: {e}")
 | 
						|
```
 | 
						|
 | 
						|
#### **Sound Generation**
 | 
						|
```python
 | 
						|
from cthulhu import sound
 | 
						|
from cthulhu.sound_generator import Tone
 | 
						|
 | 
						|
def _play_sound(self):
 | 
						|
    player = sound.getPlayer()
 | 
						|
    tone = Tone(duration=0.15, frequency=400, volumeMultiplier=0.7)
 | 
						|
    player.play(tone, interrupt=False)
 | 
						|
```
 | 
						|
 | 
						|
### **Debugging Plugin Issues**
 | 
						|
 | 
						|
#### **Common Debug Techniques**
 | 
						|
1. **Add debug output to both logger and print**:
 | 
						|
   ```python
 | 
						|
   logger.info("Plugin message")
 | 
						|
   print("DEBUG: Plugin message")  # Shows in terminal
 | 
						|
   ```
 | 
						|
 | 
						|
2. **Check plugin loading**:
 | 
						|
   ```python
 | 
						|
   # In __init__
 | 
						|
   with open('/tmp/your_plugin_debug.log', 'a') as f:
 | 
						|
       f.write("Plugin loaded\n")
 | 
						|
   ```
 | 
						|
 | 
						|
3. **Verify keybinding registration**:
 | 
						|
   ```python
 | 
						|
   if self._kb_binding:
 | 
						|
       print(f"DEBUG: Keybinding registered: {self._kb_binding}")
 | 
						|
   else:
 | 
						|
       print("DEBUG: Keybinding registration FAILED")
 | 
						|
   ```
 | 
						|
 | 
						|
#### **Common Issues & Solutions**
 | 
						|
 | 
						|
| Issue | Symptom | Solution |
 | 
						|
|-------|---------|----------|
 | 
						|
| Plugin not loading | No __init__ debug output | Check `activePlugins` list |
 | 
						|
| Keybindings not working | "stored for later registration" | Use correct parameter order |
 | 
						|
| Import errors | Plugin fails to activate | Check module imports and dependencies |
 | 
						|
| Settings not loading | Default values used | Verify settings key names |
 | 
						|
 | 
						|
### **Working Plugin Examples**
 | 
						|
- **`DisplayVersion`**: Simple keybinding + message
 | 
						|
- **`PluginManager`**: GUI dialog + settings management  
 | 
						|
- **`IndentationAudio`**: Event listening + sound generation
 | 
						|
- **`AIAssistant`**: Complex settings + multi-keybinding + external APIs
 | 
						|
 | 
						|
## D-Bus Remote Controller Integration
 | 
						|
 | 
						|
### **EXISTING FEATURE**: D-Bus Service for Remote Control
 | 
						|
Cthulhu includes a D-Bus service (ported from Orca v49.alpha) for external control and automation:
 | 
						|
 | 
						|
- **Service Name**: `org.stormux.Cthulhu.Service`
 | 
						|
- **Object Path**: `/org/stormux/Cthulhu/Service`
 | 
						|
- **Dependency**: `dasbus` library (automatically detected)
 | 
						|
 | 
						|
### Testing D-Bus Functionality
 | 
						|
```bash
 | 
						|
# Start Cthulhu with D-Bus service
 | 
						|
~/.local/bin/cthulhu
 | 
						|
 | 
						|
# Test service availability
 | 
						|
busctl --user list | grep Cthulhu
 | 
						|
 | 
						|
# Get Cthulhu version via D-Bus
 | 
						|
busctl --user call org.stormux.Cthulhu.Service /org/stormux/Cthulhu/Service org.stormux.Cthulhu.Service GetVersion
 | 
						|
 | 
						|
# Present message to user via D-Bus
 | 
						|
busctl --user call org.stormux.Cthulhu.Service /org/stormux/Cthulhu/Service org.stormux.Cthulhu.Service PresentMessage s "Hello from D-Bus"
 | 
						|
 | 
						|
# List available modules and commands
 | 
						|
busctl --user call org.stormux.Cthulhu.Service /org/stormux/Cthulhu/Service org.stormux.Cthulhu.Service ListModules
 | 
						|
```
 | 
						|
 | 
						|
### Integration Status
 | 
						|
- ✅ **Core D-Bus service**: Fully ported and integrated
 | 
						|
- ✅ **Service lifecycle**: Automatic start/shutdown with Cthulhu
 | 
						|
- ✅ **Message presentation**: `PresentMessage()` method working - **FULLY FUNCTIONAL**
 | 
						|
- ✅ **Version info**: `GetVersion()` method working - **FULLY FUNCTIONAL**
 | 
						|
- ✅ **Circular import issues**: All resolved - Cthulhu imports work correctly
 | 
						|
- ✅ **GDK version conflicts**: Fixed with proper gi.require_version calls
 | 
						|
- ✅ **Presenter singleton patterns**: Fixed with lazy initialization
 | 
						|
- ✅ **Settings manager backend**: Fixed with proper activation sequence
 | 
						|
- ✅ **ATSPI registry startup**: Fixed with deferred D-Bus service initialization
 | 
						|
- ✅ **API naming conventions**: All Orca→Cthulhu API differences resolved
 | 
						|
- 🔄 **Module registration**: Ready for individual managers to register D-Bus commands
 | 
						|
- 🔄 **Plugin integration**: Plugins can expose D-Bus commands using decorators
 | 
						|
 | 
						|
### **✅ COMPLETED - D-Bus Remote Controller Integration**
 | 
						|
The D-Bus Remote Controller from Orca v49.alpha has been successfully integrated into Cthulhu and is fully functional.
 | 
						|
 | 
						|
**Root Cause of Issues**: D-Bus service startup timing conflicts with ATSPI registry initialization.
 | 
						|
 | 
						|
**Solution Implemented**: 
 | 
						|
- Deferred D-Bus service startup using `GObject.idle_add()` after ATSPI event loop is running
 | 
						|
- Fixed all API naming convention differences between Orca and Cthulhu
 | 
						|
 | 
						|
**Files Modified for D-Bus Integration**:
 | 
						|
- `src/cthulhu/dbus_service.py` (NEW FILE) - Complete D-Bus service port with Cthulhu API fixes
 | 
						|
- `src/cthulhu/input_event.py` - Added RemoteControllerEvent + GDK version fix
 | 
						|
- `src/cthulhu/cthulhu.py` - D-Bus integration + lazy BrailleEvent import + settings manager activation + deferred startup
 | 
						|
- `src/cthulhu/Makefile.am` - Added dbus_service.py to build
 | 
						|
- Multiple presenter files - Converted to lazy initialization pattern
 | 
						|
- `src/cthulhu/keybindings.py` - Fixed GDK version requirement
 | 
						|
- `README-REMOTE-CONTROLLER.md` (NEW FILE) - Complete documentation with examples
 | 
						|
 | 
						|
**API Fixes Applied**:
 | 
						|
- `debug.print_message` → `debug.printMessage`
 | 
						|
- `script_manager.get_manager()` → `script_manager.getManager()`
 | 
						|
- `get_active_script()` → `cthulhu_state.activeScript`
 | 
						|
- `get_default_script()` → `getDefaultScript()`
 | 
						|
 | 
						|
### Bug Fixes Applied
 | 
						|
- Fixed circular imports in presenter modules (learn_mode_presenter, notification_presenter, etc.)
 | 
						|
- Added lazy imports for BrailleEvent to break cthulhu.py ↔ input_event.py cycle
 | 
						|
- Fixed GDK version conflicts by adding gi.require_version("Gdk", "3.0") to input_event.py
 | 
						|
- Changed all presenter singleton patterns to lazy initialization to prevent import-time issues
 | 
						|
- Added settings manager activation before loadUserSettings() to fix backend initialization
 | 
						|
 | 
						|
### **Commands for Analysis**
 | 
						|
```bash
 | 
						|
# Compare specific features between projects
 | 
						|
diff -r /home/storm/devel/cthulhu/src/cthulhu /home/storm/devel/orca/src/orca
 | 
						|
 | 
						|
# Test Orca's new features
 | 
						|
cd /home/storm/devel/orca && meson setup _build && meson compile -C _build
 | 
						|
 | 
						|
# Test D-Bus interface
 | 
						|
# (requires running Orca instance with D-Bus support)
 | 
						|
``` |