# OCR Plugin for Cthulhu Screen Reader

A powerful OCR (Optical Character Recognition) plugin that enables Cthulhu users to extract text from visual content and interact with it through precise coordinate mapping. Originally based on the ocrdesktop project by Chrys, this plugin has been enhanced with interactive features and comprehensive settings integration.

## Features

### 🔍 **OCR Operations**
- **Window OCR**: Extract text from the currently active window
- **Desktop OCR**: Extract text from the entire desktop screen
- **Clipboard OCR**: Extract text from images copied to the clipboard
- **Interactive OCR**: OCR with coordinate mapping for clicking and navigation

### 🎯 **Interactive Coordinate Mapping**
- **Precise Clicking**: Click any text found in OCR results using exact coordinates
- **Dual View Modes**: Toggle between text view and interactive coordinate table
- **Safety Confirmation**: Preview click coordinates before executing
- **Real-time Navigation**: Browse OCR results and click immediately

### ⚙️ **Comprehensive Settings**
- **Language Configuration**: Support for all Tesseract language packs
- **Image Processing**: Grayscale, invert, black/white, and scaling options
- **Clipboard Integration**: Automatic copying of OCR results to clipboard
- **Quality Tuning**: Adjustable parameters for optimal OCR accuracy

### 🖥️ **Accessibility Integration**
- **Voice Announcements**: Clear audio feedback for all operations
- **Keyboard Navigation**: Full keyboard control of interactive features
- **Settings GUI**: Integrated settings tab in Cthulhu preferences
- **Non-blocking Processing**: Multi-threaded operation with progress tracking

## Keybindings

| Key Combination | Action | Description |
|----------------|--------|-------------|
| `Cthulhu+Control+W` | OCR Active Window | Performs OCR on the currently focused window |
| `Cthulhu+Control+D` | OCR Desktop | Performs OCR on the entire desktop screen |
| `Cthulhu+Control+Shift+C` | OCR Clipboard | Performs OCR on image data from clipboard |
| `Cthulhu+Control+F` | **Interactive OCR** | **Opens OCR results window with coordinate mapping** |

### Interactive OCR Window Controls
| Key | Action | Description |
|-----|--------|-------------|
| `Alt+V` | Toggle View | Switch between text view and coordinate table |
| `Enter` | Click Selected | Click the text at the selected coordinates |
| `Escape` | Close Window | Close the OCR results window |
| `Arrow Keys` | Navigate | Move through OCR results in table view |

## Dependencies

### Required Dependencies
- **python3-pillow** (PIL) - Image processing library
- **python-pytesseract** - Python wrapper for Tesseract OCR
- **tesseract** - OCR engine (with language packs)
- **GTK3/GDK/Wnck** - For screenshot capture (usually pre-installed)

### Installation Commands

#### Arch Linux
```bash
sudo pacman -S python-pillow python-pytesseract tesseract tesseract-data-eng
```

#### Ubuntu/Debian
```bash
sudo apt install python3-pil python3-pytesseract tesseract-ocr tesseract-ocr-eng
```

#### Fedora
```bash
sudo dnf install python3-pillow python3-pytesseract tesseract tesseract-langpack-eng
```

### Additional Language Support
To add support for other languages, install additional Tesseract language packs:

```bash
# Examples for different distributions:
# Arch: sudo pacman -S tesseract-data-fra tesseract-data-deu tesseract-data-spa
# Ubuntu: sudo apt install tesseract-ocr-fra tesseract-ocr-deu tesseract-ocr-spa
# Fedora: sudo dnf install tesseract-langpack-fra tesseract-langpack-deu tesseract-langpack-spa
```

## Usage

1. **Enable the Plugin**: The OCR plugin is enabled by default in Cthulhu. If disabled, you can enable it through:
   - Cthulhu Preferences → Plugins → Check "OCR"
   - Or ensure `'OCR'` is in the `activePlugins` list in settings.py

2. **Basic OCR Workflow**:
   - Navigate to content you want to OCR
   - Press the appropriate key combination
   - Listen for "Performing OCR on [window/desktop/clipboard]"
   - Wait for processing to complete
   - OCR results will be announced via speech

3. **Interactive OCR Workflow**:
   - Press `Cthulhu+Control+F` to open OCR results window
   - Wait for "Performing OCR on window for interactive results"
   - Use `Alt+V` to toggle between text and coordinate table views
   - Navigate with arrow keys in table view to find desired text
   - Press `Enter` to click on the selected text location
   - Confirm the click action in the safety dialog

4. **Best Practices**:
   - Ensure good contrast between text and background for better results
   - Use window OCR for focused content (faster processing)
   - Use desktop OCR for content spanning multiple windows
   - Use clipboard OCR for images from web browsers or image viewers
   - Enable "Copy Results to Clipboard" for easy text retrieval
   - Adjust scale factor for small or blurry text (try 5-7)

## Configuration

### OCR Settings GUI
Access comprehensive OCR settings through Cthulhu Preferences:

1. **Open Cthulhu Preferences**: `~/.local/bin/cthulhu -s`
2. **Navigate to OCR Tab**: Use keyboard navigation to find the OCR settings tab
3. **Configure Settings**: Adjust all OCR parameters through the accessible interface

### Available Settings

#### **Language Configuration**
- **Language Code**: Tesseract language pack to use (default: 'eng')
  - Examples: 'fra' (French), 'deu' (German), 'spa' (Spanish)
  - Use '+' for multiple languages: 'eng+fra' for English and French

#### **Image Processing**
- **Scale Factor**: Image scaling multiplier (1-10, default: 3)
  - Higher values improve OCR accuracy for small text
  - Lower values process faster but may miss details
- **Grayscale Image**: Convert to grayscale for better text recognition
- **Invert Image**: Invert colors (useful for white text on dark backgrounds)
- **Black and White Image**: Convert to pure black/white with threshold
- **Black/White Threshold**: Threshold value for black/white conversion (0-255, default: 200)

#### **Advanced Features**
- **Analyze Colors**: Extract color information from OCR regions (requires scipy/webcolors)
- **Copy Results to Clipboard**: Automatically copy all OCR results to system clipboard

### Configuration File
Settings are automatically stored in Cthulhu's configuration system:
- **Global Settings**: `~/.local/share/cthulhu/user-settings.conf`
- **Profile Settings**: `~/.local/share/cthulhu/app-settings/[profile]/`

### Example Configuration Values
```json
{
  "ocrLanguageCode": "eng",
  "ocrScaleFactor": 3,
  "ocrGrayscaleImg": false,
  "ocrInvertImg": false, 
  "ocrBlackWhiteImg": false,
  "ocrBlackWhiteImgValue": 200,
  "ocrColorCalculation": false,
  "ocrCopyToClipboard": true
}
```

## Troubleshooting

### Common Issues

#### "No text found in OCR scan"
- **Cause**: Poor image quality, unsupported language, or no text in captured area
- **Solutions**:
  - Try different OCR mode (window vs desktop)
  - Ensure text has good contrast
  - Check if correct language pack is installed
  - Verify text is actually visible in the captured area

#### "Missing dependencies" message
- **Cause**: Required Python packages or Tesseract not installed
- **Solution**: Install missing packages using commands above

#### OCR taking too long
- **Cause**: Large desktop screenshots or complex images
- **Solutions**:
  - Use window OCR instead of desktop OCR when possible
  - Close unnecessary windows before desktop OCR
  - Consider adjusting `_scaleFactor` (lower = faster)

#### No speech output
- **Cause**: Cthulhu speech settings or audio issues
- **Solutions**:
  - Check Cthulhu speech settings
  - Test other Cthulhu speech functions
  - Verify audio system is working

#### Interactive OCR window doesn't open
- **Cause**: GTK dependencies missing or display issues
- **Solutions**:
  - Ensure GTK3 development packages are installed
  - Check display/Wayland/X11 compatibility
  - Verify Cthulhu GUI components are working

#### Click coordinates are inaccurate
- **Cause**: Window movement, scaling, or coordinate calculation errors
- **Solutions**:
  - Ensure window hasn't moved since OCR capture
  - Try recapturing with `Cthulhu+Control+F`
  - Check display scaling settings
  - Verify no window decoration changes occurred

#### Clipboard copy not working
- **Cause**: Clipboard setting disabled or GTK clipboard issues
- **Solutions**:
  - Enable "Copy Results to Clipboard" in OCR settings
  - Test clipboard functionality with other applications
  - Check GTK clipboard permissions

### Debug Information
OCR plugin debug messages are logged to Cthulhu's debug output. To enable debug logging:

```bash
cthulhu --debug > ocr_debug.log 2>&1
```

Look for messages starting with "OCRDesktop:" in the log file.

## Technical Details

### Architecture
- **Base Class**: Extends `cthulhu.plugin.Plugin`
- **Threading**: Uses Python threading for non-blocking OCR processing
- **Image Processing**: PIL/Pillow for image manipulation and enhancement
- **OCR Engine**: Tesseract via pytesseract wrapper
- **Integration**: Uses Cthulhu's speech system for output

### Image Processing Pipeline
1. **Capture**: Screenshot via GDK pixbuf system
2. **Scale**: Enlarge image by scale factor (default 3x)
3. **Transform**: Apply filters (grayscale, invert, etc.) if enabled
4. **OCR**: Process with Tesseract OCR engine
5. **Cleanup**: Remove extra whitespace and format text
6. **Present**: Announce results via Cthulhu speech

### Text Post-Processing
The plugin automatically cleans OCR output by:
- Removing multiple consecutive spaces
- Eliminating empty lines
- Trimming leading/trailing whitespace
- Removing trailing newlines

## Development

### Plugin Structure
```
src/cthulhu/plugins/OCR/
├── __init__.py          # Package import
├── plugin.py            # Main plugin implementation
├── plugin.info          # Plugin metadata
├── meson.build          # Build system integration
└── README.md           # This documentation
```

### Key Methods
- `_ocrActiveWindow()`: Captures and OCRs active window
- `_ocrDesktop()`: Captures and OCRs entire desktop
- `_ocrClipboard()`: OCRs image from clipboard
- `_showOCRResultsWindow()`: **NEW** - Interactive OCR with coordinate mapping
- `_performOCR()`: Core OCR processing with coordinate extraction
- `_presentOCRResult()`: Announces results via speech and clipboard
- `_createOCRResultsWindow()`: **NEW** - Creates interactive GTK results window
- `_clickSelectedText()`: **NEW** - Executes click at OCR coordinates

### Interactive Features Architecture
- **Coordinate Mapping**: Uses `pytesseract.image_to_data()` to extract word positions
- **Screen Transformation**: Converts OCR coordinates to actual screen coordinates
- **GTK Interface**: Accessible results window with text and table views
- **Click Safety**: Confirmation dialogs before executing click actions
- **Settings Integration**: Full integration with Cthulhu's preferences system

### Extending the Plugin
To add new OCR modes or features:

1. Add new keybinding in `_registerKeybindings()`
2. Create handler method following pattern `_ocrNewMode()`
3. Implement image capture logic for new mode
4. Use existing `_performOCR()` and `_presentOCRResult()` methods
5. For interactive features, extend `_createOCRResultsWindow()` functionality

## Version History

### Version 2.0 (Enhanced Interactive Features)
- **Interactive OCR Window**: `Cthulhu+Control+F` for coordinate mapping
- **Precise Clicking**: Click any text found in OCR results
- **Settings Integration**: Full GUI settings tab in Cthulhu preferences
- **Clipboard Integration**: Automatic copying with toggle setting
- **Dual View Modes**: Text view and coordinate table with Alt+V toggle
- **Safety Features**: Click confirmation dialogs
- **Enhanced Processing**: Coordinate extraction with quality metrics

### Version 1.0 (Original Implementation)
- Basic OCR for window, desktop, and clipboard
- Text extraction and speech output
- Multi-threading support
- Text cleanup and formatting

## Credits

- **Original ocrdesktop**: Created by Chrys (chrys87@users.noreply.github.com)
- **Cthulhu Integration**: Adapted by Storm Dragon for Cthulhu plugin system
- **Interactive Features**: Enhanced coordinate mapping and GUI integration
- **Cthulhu Screen Reader**: https://git.stormux.org/storm/cthulhu
- **Tesseract OCR**: https://github.com/tesseract-ocr/tesseract

## License

This plugin is distributed under the GNU Lesser General Public License (LGPL) version 2.1 or later, consistent with the Cthulhu screen reader project.

## Support

For issues, questions, or contributions:
- **Cthulhu Repository**: https://git.stormux.org/storm/cthulhu
- **Community**: IRC #stormux on irc.stormux.org
- **Email**: storm_dragon@stormux.org

---

*Part of the Cthulhu Screen Reader project - Making the desktop accessible for everyone.*