5 Commits
10 changed files with 892 additions and 91 deletions
@@ -0,0 +1,117 @@
# Guarded Mouse Review Wayland Backport
## Summary
Backport Orca's modern mouse review split into Cthulhu in a conservative way. Keep the current X11 `Wnck` plus `mouse:abs` implementation as the existing path. Add a second path that uses `Atspi.Device` pointer monitoring only when the local AT-SPI version is new enough and the compositor/device actually grants `POINTER_MONITOR`. If either condition fails, Cthulhu stays on the current behavior.
This is explicitly an X-safe design. The X11 path remains first-class and unchanged in behavior. Wayland support is opportunistic and must fail closed.
## Current State
- Cthulhu mouse review currently depends on `Wnck` window lookup and `mouse:abs` events in [mouse_review.py](/home/storm/devel/cthulhu/src/cthulhu/mouse_review.py).
- Recent local changes already avoid importing `Wnck` on Wayland to suppress the terminal warning, but they do not add functional Wayland mouse review.
- Cthulhu already has an `Atspi.Device` for keyboard handling in [input_event_manager.py](/home/storm/devel/cthulhu/src/cthulhu/input_event_manager.py), which is the required foundation for an Orca-style pointer-monitor path.
## Goals
- Preserve existing X11 mouse review behavior.
- Add a guarded Wayland-capable path modeled on Orca's `Atspi.Device` pointer monitoring.
- Avoid startup warnings from `Wnck` on Wayland.
- Avoid enabling any new path unless support is positively confirmed at runtime.
## Non-Goals
- No full Orca mouse review rewrite beyond the pointer-monitor split.
- No removal or semantic change of the current X11 `Wnck` path.
- No OCR refactor beyond existing `Wnck` gating already added.
- No assumption that Wayland mouse review will work everywhere; unsupported compositors must remain a no-op.
## Design
### Capability Detection
Mouse review will select its backend at runtime:
1. If AT-SPI version is at least `2.60`, or the pre-release threshold Orca used (`2.59.90`), Cthulhu may try the new backend.
2. The new backend must call `set_capabilities(... | POINTER_MONITOR)` on the existing `Atspi.Device`.
3. The new backend is considered available only if `POINTER_MONITOR` is present in the returned capability set.
4. If any of those checks fail, Cthulhu uses the existing X11 path.
This means X11 systems on older stacks continue to use the current implementation. Wayland systems with insufficient AT-SPI support do not partially enable mouse review.
### InputEventManager Extension
Extend [input_event_manager.py](/home/storm/devel/cthulhu/src/cthulhu/input_event_manager.py) with narrow pointer-monitor wrappers:
- `enable_pointer_monitoring() -> bool`
- `start_pointer_watcher(callback) -> None`
- `stop_pointer_watcher() -> None`
These wrappers should mirror Orca's error handling:
- return `False` on missing device or `GLib.GError`
- never raise into callers
- leave existing keyboard handling unchanged
No unrelated device-manager refactor is included in this pass.
### Mouse Review Backend Split
Update [mouse_review.py](/home/storm/devel/cthulhu/src/cthulhu/mouse_review.py) to maintain two internal paths:
- Legacy path:
- existing `Wnck` window tracking
- existing `mouse:abs` listener
- existing absolute-coordinate flow
- New path:
- register a `pointer-moved` watcher on the `Atspi.Device`
- use the accessible object and local coordinates delivered by AT-SPI
- if the event source is an application, resolve the containing window from the app's accessible children
- otherwise use the event object directly
The common object-presentation logic should stay shared as much as possible. Only the event source and coordinate acquisition differ.
### Safety Rules
- Do not change the old X11 path except for harmless factoring needed to share code.
- Do not force the new path on X11 systems that are currently using the legacy path successfully.
- If pointer monitoring disconnects, errors, or cannot be enabled, mouse review must behave as unavailable rather than falling into a broken mixed state.
- Existing `Wnck` suppression on Wayland remains in place.
## Testing
### Automated
Add focused regressions for:
- version/capability gating chooses the new backend only when allowed
- `enable_pointer_monitoring()` returns `False` on capability failure
- pointer watcher start/stop are no-ops when there is no device
- mouse review activation chooses the legacy backend when AT-SPI support is not available
- mouse review activation chooses the pointer-monitor backend when support is available
Tests should mock AT-SPI rather than requiring a compositor.
### Manual
X11:
- mouse review still enables and behaves as before
- no regressions in pointer routing or reviewed object presentation
Wayland:
- no `Wnck` warning on startup
- mouse review only enables when AT-SPI pointer monitoring is actually available
- if supported, pointer hover review works without breaking keyboard handling
- if unsupported, failure is clean and explicit rather than noisy or partially broken
## Risks
- The local environment here is still on AT-SPI `2.58.4`, so the new backend cannot be exercised live in this workspace.
- The guarded design limits that risk by preserving the existing path and enabling the new path only under the same runtime conditions Orca used.
- The main regression risk is accidental interaction with Cthulhu's existing keyboard `Atspi.Device`; that is why the change is limited to small wrappers plus mouse-review-specific watcher usage.
## Recommendation
Implement the guarded dual-path backport and stop there. Do not port broader Orca mouse review refactors in the same pass.
+78 -5
View File
@@ -44,6 +44,7 @@ from typing import TYPE_CHECKING, Optional, Union, Tuple, List, Dict
import gi
gi.require_version("Atspi", "2.0")
from gi.repository import Atspi
from gi.repository import GLib
from . import debug
from . import focus_manager
@@ -64,21 +65,36 @@ class InputEventManager:
self._last_input_event: Optional[input_event.InputEvent] = None
self._last_non_modifier_key_event: Optional[input_event.KeyboardEvent] = None
self._device: Optional[Atspi.Device] = None
self._pointer_moved_id: int = 0
self._mapped_keycodes: List[int] = []
self._mapped_keysyms: List[int] = []
self._grabbed_bindings: Dict[int, keybindings.KeyBinding] = {}
self._paused: bool = False
def activate_device(self) -> Atspi.Device:
"""Creates and returns the AT-SPI device used by this manager."""
if self._device is not None:
return self._device
if Atspi.get_version() >= (2, 55, 90):
self._device = Atspi.Device.new_full("org.stormux.Cthulhu")
else:
self._device = Atspi.Device.new()
return self._device
def get_device(self) -> Optional[Atspi.Device]:
"""Returns the active AT-SPI device, if any."""
return self._device
def start_key_watcher(self) -> None:
"""Starts the watcher for keyboard input events."""
msg = "INPUT EVENT MANAGER: Starting key watcher."
debug.print_message(debug.LEVEL_INFO, msg, True)
if Atspi.get_version() >= (2, 55, 90):
self._device = Atspi.Device.new_full("org.stormux.Cthulhu")
else:
self._device = Atspi.Device.new()
self._device.add_key_watcher(self.process_keyboard_event)
self.activate_device().add_key_watcher(self.process_keyboard_event)
def stop_key_watcher(self) -> None:
"""Starts the watcher for keyboard input events."""
@@ -87,6 +103,63 @@ class InputEventManager:
debug.print_message(debug.LEVEL_INFO, msg, True)
self._device = None
def enable_pointer_monitoring(self) -> bool:
"""Enables pointer monitoring on the current device, if possible."""
device = self.get_device()
if device is None:
return False
deviceCapability = getattr(Atspi, "DeviceCapability", None)
if deviceCapability is None:
return False
pointerMonitor = getattr(deviceCapability, "POINTER_MONITOR", None)
if pointerMonitor is None:
return False
setCapabilities = getattr(device, "set_capabilities", None)
if not callable(setCapabilities):
return False
currentCapabilities = 0
getCapabilities = getattr(device, "get_capabilities", None)
if callable(getCapabilities):
currentCapabilities = getCapabilities()
try:
grantedCapabilities = setCapabilities(currentCapabilities | pointerMonitor)
except GLib.GError:
return False
if isinstance(grantedCapabilities, bool):
return grantedCapabilities
try:
return bool(int(grantedCapabilities) & int(pointerMonitor))
except (TypeError, ValueError):
return False
def start_pointer_watcher(self, callback) -> None:
"""Starts the watcher for pointer movement events."""
device = self.get_device()
if device is None:
return
self._pointer_moved_id = device.connect("pointer-moved", callback)
def stop_pointer_watcher(self) -> None:
"""Stops the watcher for pointer movement events."""
device = self.get_device()
if device is None or not self._pointer_moved_id:
self._pointer_moved_id = 0
return
device.disconnect(self._pointer_moved_id)
self._pointer_moved_id = 0
def pause_key_watcher(self, pause: bool = True, reason: str = "") -> None:
"""Pauses processing of keyboard input events."""
+1
View File
@@ -108,6 +108,7 @@ cthulhu_python_sources = files([
'translation_manager.py',
'tutorialgenerator.py',
'typing_echo_presenter.py',
'wnck_support.py',
'where_am_i_presenter.py',
])
+204 -79
View File
@@ -32,30 +32,31 @@ __copyright__ = "Copyright (c) 2008 Eitan Isaacson" \
"Copyright (c) 2016 Igalia, S.L."
__license__ = "LGPL"
from collections import deque
import gi
gi.require_version("Atspi", "2.0")
from gi.repository import Atspi
from gi.repository import Atspi, GLib
import math
import time
from gi.repository import Gdk
try:
gi.require_version("Wnck", "3.0")
from gi.repository import Wnck
_mouseReviewCapable = True
except Exception:
_mouseReviewCapable = False
from .wnck_support import load_wnck
Wnck = load_wnck()
from . import cmdnames
from . import debug
from . import keybindings
from . import input_event
from . import input_event_manager
from . import messages
from . import cthulhu
from . import cthulhu_state
from . import script_manager
from . import settings_manager
from .ax_component import AXComponent
from .ax_object import AXObject
from .ax_text import AXText
from .ax_utilities import AXUtilities
@@ -82,8 +83,9 @@ class _StringContext:
self._start = start
self._end = end
self._boundingBox = 0, 0, 0, 0
if script:
self._boundingBox = script.utilities.getTextBoundingBox(obj, start, end)
if AXObject.supports_text(obj):
rect = AXText.get_range_rect(obj, start, end)
self._boundingBox = rect.x, rect.y, rect.width, rect.height
def __eq__(self, other):
return other is not None \
@@ -183,8 +185,9 @@ class _ItemContext:
self._string = self._getStringContext()
self._time = time.time()
self._boundingBox = 0, 0, 0, 0
if script:
self._boundingBox = script.utilities.getBoundingBox(obj)
if AXObject.supports_component(obj):
rect = AXComponent.get_rect(obj)
self._boundingBox = rect.x, rect.y, rect.width, rect.height
def __eq__(self, other):
return other is not None \
@@ -237,7 +240,12 @@ class _ItemContext:
return _StringContext(self._obj, self._script)
string, start, end = self._script.utilities.textAtPoint(
self._obj, self._x, self._y, boundary=self._boundary)
self._obj,
self._x,
self._y,
coordType=Atspi.CoordType.WINDOW,
boundary=self._boundary,
)
if string:
string = self._script.utilities.expandEOCs(self._obj, start, end)
@@ -351,31 +359,65 @@ class MouseReviewer:
self._handlerIds = {}
self._eventListener = Atspi.EventListener.new(self._listener)
self.inMouseEvent = False
self._eventQueue = deque()
self._mouseReviewCapable = False
self._useAtspi = False
self._handlers = self._setup_handlers()
self._bindings = self._setup_bindings()
if not _mouseReviewCapable:
msg = "MOUSE REVIEW ERROR: Wnck is not available"
debug.printMessage(debug.LEVEL_INFO, msg, True)
return
atspiVersion = Atspi.get_version()
capabilityEnum = getattr(Atspi, "DeviceCapability", None)
pointerMonitor = getattr(capabilityEnum, "POINTER_MONITOR", 0) if capabilityEnum else 0
atspiSupported = pointerMonitor and (
atspiVersion[0] > 2
or atspiVersion[1] >= 60
or (atspiVersion[0] == 2 and atspiVersion[1] == 59 and atspiVersion[2] >= 90)
)
display = Gdk.Display.get_default()
try:
seat = Gdk.Display.get_default_seat(display)
self._pointer = seat.get_pointer()
except AttributeError:
msg = "MOUSE REVIEW ERROR: Gtk+ 3.20 is not available"
debug.printMessage(debug.LEVEL_INFO, msg, True)
return
if atspiSupported:
manager = input_event_manager.get_manager()
manager.activate_device()
if manager.enable_pointer_monitoring():
self._useAtspi = True
self._mouseReviewCapable = True
else:
self._mouseReviewCapable = Wnck is not None
else:
self._mouseReviewCapable = Wnck is not None
except Exception:
msg = "MOUSE REVIEW ERROR: Exception getting pointer for default seat."
self._mouseReviewCapable = False
if not self._mouseReviewCapable:
msg = (
"MOUSE REVIEW ERROR: Not supported by AT-SPI device"
if atspiSupported
else "MOUSE REVIEW ERROR: Wnck or at-spi2-core >= 2.60 required"
)
debug.printMessage(debug.LEVEL_INFO, msg, True)
return
if not self._pointer:
msg = "MOUSE REVIEW ERROR: No pointer for default seat."
debug.printMessage(debug.LEVEL_INFO, msg, True)
return
if not self._useAtspi:
display = Gdk.Display.get_default()
try:
seat = Gdk.Display.get_default_seat(display)
self._pointer = seat.get_pointer()
except AttributeError:
msg = "MOUSE REVIEW ERROR: Gtk+ 3.20 is not available"
debug.printMessage(debug.LEVEL_INFO, msg, True)
self._mouseReviewCapable = False
return
except Exception:
msg = "MOUSE REVIEW ERROR: Exception getting pointer for default seat."
debug.printMessage(debug.LEVEL_INFO, msg, True)
self._mouseReviewCapable = False
return
if not self._pointer:
msg = "MOUSE REVIEW ERROR: No pointer for default seat."
debug.printMessage(debug.LEVEL_INFO, msg, True)
self._mouseReviewCapable = False
return
if not self._active:
return
@@ -421,9 +463,14 @@ class MouseReviewer:
def activate(self):
"""Activates mouse review."""
if not _mouseReviewCapable:
msg = "MOUSE REVIEW ERROR: Wnck is not available"
if not self._mouseReviewCapable:
msg = (
"MOUSE REVIEW ERROR: Not supported by AT-SPI device"
if self._useAtspi
else "MOUSE REVIEW ERROR: Wnck or at-spi2-core >= 2.60 required"
)
debug.printMessage(debug.LEVEL_INFO, msg, True)
self._active = False
return
# Set up the initial object as the one with the focus to avoid
@@ -437,6 +484,11 @@ class MouseReviewer:
frame = script.utilities.topLevelObject(obj)
self._currentMouseOver = _ItemContext(obj=obj, frame=frame, script=script)
if self._useAtspi:
input_event_manager.get_manager().start_pointer_watcher(self._on_pointer_moved)
self._active = True
return
self._eventListener.register("mouse:abs")
screen = Wnck.Screen.get_default()
if screen:
@@ -462,20 +514,29 @@ class MouseReviewer:
def deactivate(self):
"""Deactivates mouse review."""
self._eventListener.deregister("mouse:abs")
for key, value in self._handlerIds.items():
value.disconnect(key)
self._handlerIds = {}
self._workspace = None
self._windows = []
self._all_windows = []
if self._useAtspi:
input_event_manager.get_manager().stop_pointer_watcher()
else:
try:
self._eventListener.deregister("mouse:abs")
except GLib.GError as error:
msg = f"MOUSE REVIEW: Exception deregistering 'mouse:abs' listener: {error}"
debug.printMessage(debug.LEVEL_INFO, msg, True)
for key, value in self._handlerIds.items():
value.disconnect(key)
self._handlerIds = {}
self._workspace = None
self._windows = []
self._all_windows = []
self._eventQueue.clear()
self._active = False
def getCurrentItem(self):
"""Returns the accessible object being reviewed."""
if not _mouseReviewCapable:
if not self._mouseReviewCapable:
return None
if not self._active:
@@ -493,7 +554,7 @@ class MouseReviewer:
def toggle(self, script=None, event=None):
"""Toggle mouse reviewing on or off."""
if not _mouseReviewCapable:
if not self._mouseReviewCapable:
return
self._active = not self._active
@@ -570,52 +631,65 @@ class MouseReviewer:
return [extents.x, extents.y, extents.width, extents.height] == list(bounds)
def _accessible_window_at_point(self, pX, pY):
"""Returns the accessible window at the specified coordinates."""
def _accessible_window_at_point_deprecated(self, pX, pY):
"""Returns the accessible window and local coordinates for screen coordinates."""
window = None
for w in self._windows:
if w.is_minimized():
continue
x, y, width, height = w.get_geometry()
x, y, width, height = w.get_client_window_geometry()
if x <= pX <= x + width and y <= pY <= y + height:
window = w
break
if not window:
return None
return None, -1, -1
windowApp = window.get_application()
if not windowApp:
return None
app = AXUtilities.get_application_with_pid(windowApp.get_pid())
app = AXUtilities.get_application_with_pid(windowApp.get_pid()) if windowApp else None
if not app:
return None
return None, -1, -1
windowX = pX - x
windowY = pY - y
candidates = [o for o in AXObject.iter_children(
app, lambda x: self._contains_point(x, pX, pY))]
app, lambda obj: self._contains_point(obj, windowX, windowY, Atspi.CoordType.WINDOW))]
if len(candidates) == 1:
return candidates[0]
return candidates[0], windowX, windowY
name = window.get_name()
matches = [o for o in candidates if AXObject.get_name(o) == name]
if len(matches) == 1:
return matches[0]
return matches[0], windowX, windowY
bbox = window.get_client_window_geometry()
matches = [o for o in candidates if self._has_bounds(o, bbox)]
matches = [o for o in candidates if AXUtilities.is_active(o)]
if len(matches) == 1:
return matches[0]
return matches[0], windowX, windowY
return None
return None, -1, -1
def _on_mouse_moved(self, event):
"""Callback for mouse:abs events."""
def _accessible_window_at_point(self, app, pX, pY):
"""Returns the accessible window and local coordinates for pointer-moved events."""
screen, pX, pY = self._pointer.get_position()
window = self._accessible_window_at_point(pX, pY)
def getTuple(obj, x, y):
rect = AXComponent.get_rect(obj)
return obj, x - rect.x, y - rect.y
candidates = [o for o in AXObject.iter_children(
app, lambda obj: self._contains_point(obj, pX, pY, Atspi.CoordType.WINDOW))]
if len(candidates) == 1:
return getTuple(candidates[0], pX, pY)
matches = [o for o in candidates if AXUtilities.is_active(o)]
if len(matches) == 1:
return getTuple(matches[0], pX, pY)
return None, -1, -1
def _mouse_moved_common(self, window, pX, pY):
tokens = [f"MOUSE REVIEW: Window at ({pX}, {pY}) is", window]
debug.printTokens(debug.LEVEL_INFO, tokens, True)
if not window:
@@ -632,16 +706,16 @@ class MouseReviewer:
else:
menu = AXObject.find_ancestor(cthulhu_state.locusOfFocus, AXUtilities.is_menu)
screen, nowX, nowY = self._pointer.get_position()
if (pX, pY) != (nowX, nowY):
msg = f"MOUSE REVIEW: Pointer moved again: ({nowX}, {nowY})"
debug.printMessage(debug.LEVEL_INFO, msg, True)
return
obj = None
if menu:
obj = script.utilities.descendantAtPoint(menu, pX, pY, Atspi.CoordType.WINDOW)
tokens = ["MOUSE REVIEW: Object in", menu, f"at ({pX}, {pY}) is", obj]
debug.printTokens(debug.LEVEL_INFO, tokens, True)
obj = script.utilities.descendantAtPoint(menu, pX, pY) \
or script.utilities.descendantAtPoint(window, pX, pY)
tokens = [f"MOUSE REVIEW: Object at ({pX}, {pY}) is", obj]
debug.printTokens(debug.LEVEL_INFO, tokens, True)
if obj is None:
obj = script.utilities.descendantAtPoint(window, pX, pY, Atspi.CoordType.WINDOW)
tokens = ["MOUSE REVIEW: Object in", window, f"at ({pX}, {pY}) is", obj]
debug.printTokens(debug.LEVEL_INFO, tokens, True)
script = self.app.getScriptManager().get_script(AXObject.get_application(window), obj)
if menu and obj and not AXObject.find_ancestor(obj, AXUtilities.is_menu):
@@ -658,9 +732,8 @@ class MouseReviewer:
debug.printTokens(debug.LEVEL_INFO, tokens, True)
return
screen, nowX, nowY = self._pointer.get_position()
if (pX, pY) != (nowX, nowY):
msg = f"MOUSE REVIEW: Pointer moved again: ({nowX}, {nowY})"
if len(self._eventQueue):
msg = "MOUSE REVIEW: Mouse moved again."
debug.printMessage(debug.LEVEL_INFO, msg, True)
return
@@ -679,22 +752,76 @@ class MouseReviewer:
if new.present(self._currentMouseOver):
self._currentMouseOver = new
def _listener(self, event):
"""Generic listener, mainly to output debugging info."""
def _on_mouse_moved_deprecated(self, event):
"""Callback for mouse:abs events."""
pX, pY = event.detail1, event.detail2
window, windowX, windowY = self._accessible_window_at_point_deprecated(pX, pY)
self._mouse_moved_common(window, windowX, windowY)
def _on_mouse_moved(self, obj, pX, pY):
"""Callback for pointer-moved events."""
if AXObject.get_role(obj) == Atspi.Role.APPLICATION:
window, windowX, windowY = self._accessible_window_at_point(obj, pX, pY)
self._mouse_moved_common(window, windowX, windowY)
return
self._mouse_moved_common(obj, pX, pY)
def _process_event_deprecated(self):
if not self._eventQueue:
return
event = self._eventQueue.popleft()
if len(self._eventQueue):
return
startTime = time.time()
tokens = ["\nvvvvv PROCESS OBJECT EVENT", event.type, "vvvvv"]
debug.printTokens(debug.LEVEL_INFO, tokens, False)
if event.type.startswith("mouse:abs"):
self.inMouseEvent = True
self._on_mouse_moved(event)
self.inMouseEvent = False
self.inMouseEvent = True
self._on_mouse_moved_deprecated(event)
self.inMouseEvent = False
msg = f"TOTAL PROCESSING TIME: {time.time() - startTime:.4f}\n"
msg += f"^^^^^ PROCESS OBJECT EVENT {event.type} ^^^^^\n"
debug.printMessage(debug.LEVEL_INFO, msg, False)
def _process_pointer_event(self):
if not self._eventQueue:
return
obj, x, y = self._eventQueue.popleft()
if len(self._eventQueue):
return
startTime = time.time()
tokens = ["\nvvvvv PROCESS POINTER-MOVED EVENT", "vvvvv"]
debug.printTokens(debug.LEVEL_INFO, tokens, False)
self.inMouseEvent = True
self._on_mouse_moved(obj, x, y)
self.inMouseEvent = False
msg = f"TOTAL PROCESSING TIME: {time.time() - startTime:.4f}\n"
msg += "^^^^^ PROCESS POINTER-MOVED EVENT ^^^^^\n"
debug.printMessage(debug.LEVEL_INFO, msg, False)
def _listener(self, event):
"""Generic listener, mainly to output debugging info."""
if event.type.startswith("mouse:abs"):
self._eventQueue.append(event)
GLib.timeout_add(50, self._process_event_deprecated)
def _on_pointer_moved(self, _device, obj, x, y):
"""Listener for pointer-moved events from devices."""
self._eventQueue.append([obj, x, y])
GLib.timeout_add(50, self._process_pointer_event)
_reviewer = None
def getReviewer():
"""Returns the Mouse Reviewer"""
@@ -704,5 +831,3 @@ def getReviewer():
from . import cthulhu
_reviewer = MouseReviewer(cthulhu.cthulhuApp)
return _reviewer
+13 -7
View File
@@ -27,6 +27,7 @@ from gi.repository import Atspi
from cthulhu.plugin import Plugin, cthulhu_hookimpl
from cthulhu import debug
from cthulhu import settings_manager
from cthulhu.wnck_support import load_wnck
# Note: Removed complex beep system - simple announcements work perfectly!
@@ -68,17 +69,18 @@ try:
except ImportError:
WEBCOLORS_AVAILABLE = False
# GTK/GDK/Wnck
# GTK/GDK
try:
import gi
gi.require_version("Gtk", "3.0")
gi.require_version("Gdk", "3.0")
gi.require_version("Wnck", "3.0")
from gi.repository import Gtk, Gdk, Wnck
from gi.repository import Gtk, Gdk
GTK_AVAILABLE = True
except ImportError:
except Exception:
GTK_AVAILABLE = False
Wnck = load_wnck()
WNCK_AVAILABLE = Wnck is not None
logger = logging.getLogger(__name__)
class OCRDesktop(Plugin):
@@ -151,7 +153,7 @@ class OCRDesktop(Plugin):
if not PYTESSERACT_AVAILABLE:
missing_deps.append("python-pytesseract")
if not GTK_AVAILABLE:
missing_deps.append("GTK3/GDK/Wnck")
missing_deps.append("GTK3/GDK")
if missing_deps:
debug.printMessage(debug.LEVEL_INFO,
@@ -359,6 +361,10 @@ class OCRDesktop(Plugin):
if not GTK_AVAILABLE:
debug.printMessage(debug.LEVEL_INFO, "OCRDesktop: GTK not available for screenshots", True)
return False
if not WNCK_AVAILABLE:
debug.printMessage(debug.LEVEL_INFO, "OCRDesktop: Wnck not available for active window screenshots", True)
return False
try:
time.sleep(0.3) # Brief delay
@@ -869,4 +875,4 @@ class OCRDesktop(Plugin):
return True
except Exception as e:
debug.printMessage(debug.LEVEL_INFO, f"OCRDesktop: Error copying to clipboard: {e}", True)
return False
return False
@@ -44,6 +44,8 @@ class Utilities(ChromiumUtilities):
def clearSteamVirtualizedListCaches(self) -> None:
self.clearContentCache()
self._steamInferredButtonLabels = {}
self._isUselessImage = {}
self._shouldFilter = {}
def isSteamVirtualizedList(self, obj) -> bool:
if not (obj and self.inDocumentContent(obj)):
@@ -76,6 +78,21 @@ class Utilities(ChromiumUtilities):
cache[obj] = inferredLabel
return inferredLabel
def isUselessImage(self, obj) -> bool:
if not (obj and self.inDocumentContent(obj)):
return super().isUselessImage(obj)
cached = self._isUselessImage.get(hash(obj))
if cached is not None:
return cached
rv = super().isUselessImage(obj)
if not rv:
rv = self._isRedundantSteamImage(obj)
self._isUselessImage[hash(obj)] = rv
return rv
def _shouldInferSteamButtonLabel(self, obj) -> bool:
if not (obj and self.inDocumentContent(obj)):
return False
@@ -113,6 +130,34 @@ class Utilities(ChromiumUtilities):
return "Add Friend"
return ""
def _isRedundantSteamImage(self, obj) -> bool:
if not AXUtilities.is_image_or_canvas(obj):
return False
if AXObject.get_name(obj) or AXObject.get_description(obj):
return False
if AXObject.get_child_count(obj):
return False
if AXUtilities.is_focusable(obj):
return False
if not AXObject.has_action(obj, "click-ancestor"):
return False
roleDescription = self._normalizeSteamLabelText(AXObject.get_role_description(obj) or "")
if roleDescription and roleDescription.casefold() not in ["unlabeled image", "image"]:
return False
nearbyLabel = self._getSteamNearbyImageLabel(obj)
if not self._isUsefulSteamLabel(nearbyLabel):
return False
tokens = ["STEAM: Treating redundant image as useless:", obj, "(label:", nearbyLabel, ")"]
debug.printTokens(debug.LEVEL_INFO, tokens, True)
return True
def _getSteamNearbyButtonLabel(self, obj) -> str:
parent = AXObject.get_parent(obj)
if parent is None:
@@ -132,6 +177,25 @@ class Utilities(ChromiumUtilities):
return self._getSteamLabelFromChildren(grandParent, ignore=parent)
def _getSteamNearbyImageLabel(self, obj) -> str:
parent = AXObject.get_parent(obj)
if parent is None:
return ""
siblingLabel = self._getSteamLabelFromChildren(parent, ignore=obj)
if siblingLabel:
return siblingLabel
parentLabel = self._getSteamReadableText(parent)
if self._isUsefulSteamLabel(parentLabel):
return parentLabel
grandParent = AXObject.get_parent(parent)
if grandParent is None:
return ""
return self._getSteamLabelFromChildren(grandParent, ignore=parent)
def _getSteamLabelFromChildren(self, obj, ignore=None) -> str:
for child in AXObject.iter_children(obj):
if child == ignore:
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env python3
#
# Copyright (c) 2026 Stormux
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
"""Helpers for loading Wnck only when the session can support it."""
from __future__ import annotations
import importlib
import os
from types import ModuleType
def get_session_type() -> str:
sessionType = (os.environ.get("XDG_SESSION_TYPE") or "").strip().lower()
if sessionType:
return sessionType
if os.environ.get("WAYLAND_DISPLAY"):
return "wayland"
if os.environ.get("DISPLAY"):
return "x11"
return "unknown"
def can_use_wnck(session_type: str | None = None) -> bool:
return (session_type or get_session_type()).strip().lower() == "x11"
def load_wnck(session_type: str | None = None) -> ModuleType | None:
if not can_use_wnck(session_type):
return None
gi = importlib.import_module("gi")
gi.require_version("Wnck", "3.0")
return importlib.import_module("gi.repository.Wnck")
@@ -0,0 +1,224 @@
import sys
import types
import unittest
from pathlib import Path
from unittest import mock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from cthulhu import input_event_manager
from cthulhu import mouse_review
class FakeDevice:
def __init__(self):
self.add_key_watcher_calls = []
self.connect_calls = []
self.disconnect_calls = []
self.set_capabilities_calls = []
self.capabilities = 0
self.next_handler_id = 17
def add_key_watcher(self, callback, user_data=None):
self.add_key_watcher_calls.append((callback, user_data))
def connect(self, signalName, callback):
self.connect_calls.append((signalName, callback))
return self.next_handler_id
def disconnect(self, handlerId):
self.disconnect_calls.append(handlerId)
def get_capabilities(self):
return self.capabilities
def set_capabilities(self, capabilities):
self.set_capabilities_calls.append(capabilities)
self.capabilities = capabilities
return capabilities
class FakeDeviceFactory:
def __init__(self, device):
self.device = device
self.new_calls = 0
self.new_full_calls = 0
self.app_ids = []
def new(self):
self.new_calls += 1
return self.device
def new_full(self, appId):
self.new_full_calls += 1
self.app_ids.append(appId)
return self.device
class InputEventManagerPointerMonitorTests(unittest.TestCase):
def setUp(self):
self.manager = input_event_manager.InputEventManager()
def test_activate_device_creates_the_device_only_once(self):
device = FakeDevice()
deviceFactory = FakeDeviceFactory(device)
fakeAtspi = types.SimpleNamespace(
get_version=lambda: (2, 58, 4),
Device=deviceFactory,
)
with mock.patch.object(input_event_manager, "Atspi", fakeAtspi):
firstDevice = self.manager.activate_device()
secondDevice = self.manager.activate_device()
self.assertIs(firstDevice, device)
self.assertIs(secondDevice, device)
self.assertEqual(deviceFactory.new_full_calls, 1)
self.assertEqual(deviceFactory.new_calls, 0)
self.assertEqual(deviceFactory.app_ids, ["org.stormux.Cthulhu"])
def test_enable_pointer_monitoring_returns_false_without_a_device(self):
fakeAtspi = types.SimpleNamespace()
with mock.patch.object(input_event_manager, "Atspi", fakeAtspi):
self.assertFalse(self.manager.enable_pointer_monitoring())
def test_enable_pointer_monitoring_returns_false_when_device_capability_is_missing(self):
device = FakeDevice()
deviceFactory = FakeDeviceFactory(device)
fakeAtspi = types.SimpleNamespace(
get_version=lambda: (2, 58, 4),
Device=deviceFactory,
)
with mock.patch.object(input_event_manager, "Atspi", fakeAtspi):
self.manager.activate_device()
self.assertFalse(self.manager.enable_pointer_monitoring())
self.assertEqual(device.set_capabilities_calls, [])
def test_enable_pointer_monitoring_returns_true_when_pointer_monitor_is_granted(self):
device = FakeDevice()
deviceFactory = FakeDeviceFactory(device)
fakeDeviceCapability = types.SimpleNamespace(POINTER_MONITOR=8)
fakeAtspi = types.SimpleNamespace(
get_version=lambda: (2, 58, 4),
Device=deviceFactory,
DeviceCapability=fakeDeviceCapability,
)
with mock.patch.object(input_event_manager, "Atspi", fakeAtspi):
self.manager.activate_device()
self.assertTrue(self.manager.enable_pointer_monitoring())
self.assertEqual(device.set_capabilities_calls, [8])
self.assertEqual(device.capabilities, 8)
def test_start_and_stop_pointer_watcher_connect_and_disconnect_pointer_moved(self):
device = FakeDevice()
deviceFactory = FakeDeviceFactory(device)
fakeAtspi = types.SimpleNamespace(
get_version=lambda: (2, 58, 4),
Device=deviceFactory,
)
callback = mock.Mock()
with mock.patch.object(input_event_manager, "Atspi", fakeAtspi):
self.manager.activate_device()
self.manager.start_pointer_watcher(callback)
self.manager.stop_pointer_watcher()
self.assertEqual(device.connect_calls, [("pointer-moved", callback)])
self.assertEqual(device.disconnect_calls, [17])
class MouseReviewBackendSelectionTests(unittest.TestCase):
@staticmethod
def _make_app(enabled=False):
app = mock.Mock()
settingsManager = mock.Mock()
settingsManager.getSetting.return_value = enabled
app.getSettingsManager.return_value = settingsManager
app.getScriptManager.return_value = mock.Mock()
return app
def test_prefers_atspi_backend_when_version_and_capability_are_available(self):
listener = mock.Mock()
deviceManager = mock.Mock()
deviceManager.enable_pointer_monitoring.return_value = True
with (
mock.patch.object(mouse_review.Atspi.EventListener, "new", return_value=listener),
mock.patch.object(mouse_review.Atspi, "get_version", return_value=(2, 60, 0)),
mock.patch.object(
mouse_review.Atspi,
"DeviceCapability",
new=types.SimpleNamespace(POINTER_MONITOR=8),
create=True,
),
mock.patch.object(mouse_review.input_event_manager, "get_manager", return_value=deviceManager),
mock.patch.object(mouse_review, "Wnck", None),
):
reviewer = mouse_review.MouseReviewer(self._make_app())
self.assertTrue(reviewer._useAtspi)
self.assertTrue(reviewer._mouseReviewCapable)
deviceManager.activate_device.assert_called_once_with()
deviceManager.enable_pointer_monitoring.assert_called_once_with()
def test_activate_uses_pointer_watcher_for_atspi_backend(self):
listener = mock.Mock()
deviceManager = mock.Mock()
deviceManager.enable_pointer_monitoring.return_value = True
with (
mock.patch.object(mouse_review.Atspi.EventListener, "new", return_value=listener),
mock.patch.object(mouse_review.Atspi, "get_version", return_value=(2, 60, 0)),
mock.patch.object(
mouse_review.Atspi,
"DeviceCapability",
new=types.SimpleNamespace(POINTER_MONITOR=8),
create=True,
),
mock.patch.object(mouse_review.input_event_manager, "get_manager", return_value=deviceManager),
mock.patch.object(mouse_review.cthulhu_state, "locusOfFocus", None),
mock.patch.object(mouse_review, "Wnck", None),
):
reviewer = mouse_review.MouseReviewer(self._make_app(enabled=False))
reviewer.activate()
reviewer.deactivate()
deviceManager.start_pointer_watcher.assert_called_once_with(reviewer._on_pointer_moved)
deviceManager.stop_pointer_watcher.assert_called_once_with()
listener.register.assert_not_called()
def test_keeps_legacy_backend_when_atspi_pointer_monitor_is_unavailable(self):
listener = mock.Mock()
pointer = mock.Mock()
seat = mock.Mock()
seat.get_pointer.return_value = pointer
screen = mock.Mock()
screen.get_windows_stacked.return_value = []
screen.get_active_workspace.return_value = None
fakeWnck = mock.Mock()
fakeWnck.Screen.get_default.return_value = screen
with (
mock.patch.object(mouse_review.Atspi.EventListener, "new", return_value=listener),
mock.patch.object(mouse_review.Atspi, "get_version", return_value=(2, 58, 4)),
mock.patch.object(mouse_review.input_event_manager, "get_manager"),
mock.patch.object(mouse_review.cthulhu_state, "locusOfFocus", None),
mock.patch.object(mouse_review.Gdk.Display, "get_default", return_value=mock.Mock()),
mock.patch.object(mouse_review.Gdk.Display, "get_default_seat", return_value=seat),
mock.patch.object(mouse_review, "Wnck", fakeWnck),
):
reviewer = mouse_review.MouseReviewer(self._make_app(enabled=False))
reviewer.activate()
self.assertFalse(reviewer._useAtspi)
self.assertTrue(reviewer._mouseReviewCapable)
listener.register.assert_called_once_with("mouse:abs")
if __name__ == "__main__":
unittest.main()
+85
View File
@@ -267,5 +267,90 @@ class SteamLabelRecoveryTests(unittest.TestCase):
self.assertEqual(utilities.displayedLabel(button), "Add Friend")
class SteamRedundantImageTests(unittest.TestCase):
def test_is_useless_image_treats_unlabeled_click_ancestor_image_with_nearby_text_as_useless(self):
testScript = mock.Mock(generatorCache={})
utilities = steam_script_utilities.Utilities(testScript)
image = object()
parent = object()
entry = object()
utilities.inDocumentContent = mock.Mock(return_value=True)
def get_parent(obj):
if obj in (image, entry):
return parent
return None
def get_name(obj):
if obj is entry:
return "Search for games or profiles..."
return ""
def get_child_count(obj):
if obj is parent:
return 2
return 0
def has_action(obj, actionName):
return obj is image and actionName == "click-ancestor"
def iter_children(obj, pred=None):
children = [image, entry] if obj is parent else []
if pred is not None:
children = [child for child in children if pred(child)]
return iter(children)
with (
mock.patch.object(steam_script_utilities.ChromiumUtilities, "isUselessImage", return_value=False),
mock.patch.object(steam_script_utilities.AXObject, "get_parent", side_effect=get_parent),
mock.patch.object(steam_script_utilities.AXObject, "get_name", side_effect=get_name),
mock.patch.object(steam_script_utilities.AXObject, "get_description", return_value=""),
mock.patch.object(steam_script_utilities.AXObject, "get_child_count", side_effect=get_child_count),
mock.patch.object(steam_script_utilities.AXObject, "has_action", side_effect=has_action),
mock.patch.object(steam_script_utilities.AXObject, "iter_children", side_effect=iter_children),
mock.patch.object(steam_script_utilities.AXObject, "supports_text", return_value=False),
mock.patch.object(
steam_script_utilities.AXObject,
"get_role_description",
side_effect=lambda obj: "Unlabeled image" if obj is image else "",
),
mock.patch.object(
steam_script_utilities.AXUtilities,
"is_image_or_canvas",
side_effect=lambda obj: obj is image,
),
mock.patch.object(steam_script_utilities.AXUtilities, "is_focusable", return_value=False),
mock.patch.object(steam_script_utilities.AXUtilities, "is_button", return_value=False),
mock.patch.object(steam_script_utilities.AXUtilities, "is_push_button", return_value=False),
):
self.assertTrue(utilities.isUselessImage(image))
def test_is_useless_image_defers_to_generic_logic_without_nearby_text(self):
testScript = mock.Mock(generatorCache={})
utilities = steam_script_utilities.Utilities(testScript)
image = object()
utilities.inDocumentContent = mock.Mock(return_value=True)
with (
mock.patch.object(steam_script_utilities.ChromiumUtilities, "isUselessImage", return_value=False),
mock.patch.object(steam_script_utilities.AXObject, "get_parent", return_value=None),
mock.patch.object(steam_script_utilities.AXObject, "get_name", return_value=""),
mock.patch.object(steam_script_utilities.AXObject, "get_description", return_value=""),
mock.patch.object(steam_script_utilities.AXObject, "get_child_count", return_value=0),
mock.patch.object(steam_script_utilities.AXObject, "has_action", return_value=True),
mock.patch.object(steam_script_utilities.AXObject, "supports_text", return_value=False),
mock.patch.object(steam_script_utilities.AXObject, "get_role_description", return_value=""),
mock.patch.object(
steam_script_utilities.AXUtilities,
"is_image_or_canvas",
side_effect=lambda obj: obj is image,
),
mock.patch.object(steam_script_utilities.AXUtilities, "is_focusable", return_value=False),
):
self.assertFalse(utilities.isUselessImage(image))
if __name__ == "__main__":
unittest.main()
+66
View File
@@ -0,0 +1,66 @@
import sys
import unittest
from pathlib import Path
from unittest import mock
import gi
gi.require_version("Gdk", "3.0")
gi.require_version("Gtk", "3.0")
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from cthulhu import wnck_support
from cthulhu.plugins.OCR import plugin as ocr_plugin
class WnckSupportTests(unittest.TestCase):
def test_load_wnck_skips_import_when_session_is_wayland(self):
with mock.patch.object(
wnck_support.importlib,
"import_module",
side_effect=AssertionError("import_module should not be called"),
):
self.assertIsNone(wnck_support.load_wnck(session_type="wayland"))
def test_load_wnck_imports_wnck_when_session_is_x11(self):
fakeGi = mock.Mock()
fakeWnck = object()
def importModule(name):
if name == "gi":
return fakeGi
if name == "gi.repository.Wnck":
return fakeWnck
raise AssertionError(f"Unexpected import: {name}")
with mock.patch.object(wnck_support.importlib, "import_module", side_effect=importModule):
self.assertIs(wnck_support.load_wnck(session_type="x11"), fakeWnck)
fakeGi.require_version.assert_called_once_with("Wnck", "3.0")
class OCRWnckHandlingTests(unittest.TestCase):
def test_check_dependencies_does_not_fail_when_wnck_is_unavailable(self):
testPlugin = ocr_plugin.OCRDesktop.__new__(ocr_plugin.OCRDesktop)
with (
mock.patch.object(ocr_plugin, "PIL_AVAILABLE", True),
mock.patch.object(ocr_plugin, "PYTESSERACT_AVAILABLE", True),
mock.patch.object(ocr_plugin, "GTK_AVAILABLE", True),
mock.patch.object(ocr_plugin, "WNCK_AVAILABLE", False),
):
self.assertTrue(testPlugin._checkDependencies())
def test_screen_shot_window_returns_false_when_wnck_is_unavailable(self):
testPlugin = ocr_plugin.OCRDesktop.__new__(ocr_plugin.OCRDesktop)
with (
mock.patch.object(ocr_plugin, "GTK_AVAILABLE", True),
mock.patch.object(ocr_plugin, "WNCK_AVAILABLE", False),
):
self.assertFalse(testPlugin._screenShotWindow())
if __name__ == "__main__":
unittest.main()