QuickNav Text paragraph navigation (#16031)

Closes #15998

Summary of the issue:
Add QuickNav text paragraph navigation

Description of user facing changes
Added QuickNav gesture p to jump to next/previous text paragraph in browse mode.
Description of development approach
In BrowseModeTreeInterceptor added function _iterSimilarParagraph that finds next/previous paragraph that satisfies condition defined by a lambda function. I will reuse this function in later PRs to implement vertical navigation.
Added a new clause in BrowseModeTreeInterceptor._quickNavScript() that handles text paragraph case and calls function defined in the previous bullet.
For text criteria I ended up implementing a user configurable regex.
Its initial value is defined in DEFAULT_TEXT_PARAGRAPH_REGEX variable in configSpec.py:11.
It is user-configurable in Browse Mode page in NVDA options.
The rationale for using a regex instead of plain-text search is to reduce the number of false positives.
For example, for period character we check that it follows a word character \w and is followed by space character or end of string. There are a few more rules for edge cases in the code.
I don't include comma, colon and semicolon punctuation marks in the regex due to high number of false positives.
The regex also accounts for CJK languages by checking for full-width punctuation marks.

source/browseMode.py

@@ -10,19 +10,22 @@
 	Union,
 	cast,
 )
+from collections.abc import Generator
 import os
 import itertools
 import collections
 import winsound
 import time
 import weakref
+import re
 
 import wx
 import core
 import winUser
 import mouseHandler
 from logHandler import log
 import documentBase
+from documentBase import _Movement
 import review
 import inputCore
 import scriptHandler
@@ -440,9 +443,56 @@ def _iterNodesByType(self,itemType,direction="next",pos=None):
 	def _iterNotLinkBlock(self, direction="next", pos=None):
 		raise NotImplementedError
 
+	MAX_ITERATIONS_FOR_SIMILAR_PARAGRAPH = 100_000
+
+	def _iterSimilarParagraph(
+			self,
+			kind: str,
+			paragraphFunction: Callable[[textInfos.TextInfo], Optional[Any]],
+			desiredValue: Optional[Any],
+			direction: _Movement,
+			pos: textInfos.TextInfo,
+	) -> Generator[TextInfoQuickNavItem, None, None]:
+		if direction not in [_Movement.NEXT, _Movement.PREVIOUS]:
+			raise RuntimeError
+		info = pos.copy()
+		info.collapse()
+		info.expand(textInfos.UNIT_PARAGRAPH)
+		if desiredValue is None:
+			desiredValue = paragraphFunction(info)
+		for i in range(self.MAX_ITERATIONS_FOR_SIMILAR_PARAGRAPH):
+			# move by one paragraph in the desired direction
+			info.collapse(end=direction == _Movement.NEXT)
+			if direction == _Movement.PREVIOUS:
+				if info.move(textInfos.UNIT_CHARACTER, -1) == 0:
+					return
+			info.expand(textInfos.UNIT_PARAGRAPH)
+			if info.isCollapsed:
+				return
+			value = paragraphFunction(info)
+			if value == desiredValue:
+				yield TextInfoQuickNavItem(kind, self, info.copy())
+
+
 	def _quickNavScript(self,gesture, itemType, direction, errorMessage, readUnit):
 		if itemType=="notLinkBlock":
 			iterFactory=self._iterNotLinkBlock
+		elif itemType == "textParagraph":
+			punctuationMarksRegex = re.compile(
+				config.conf["virtualBuffers"]["textParagraphRegex"],
+			)
+
+			def paragraphFunc(info: textInfos.TextInfo) -> bool:
+				return punctuationMarksRegex.search(info.text) is not None
+
+			def iterFactory(direction: str, pos: textInfos.TextInfo) -> Generator[TextInfoQuickNavItem, None, None]:
+				return self._iterSimilarParagraph(
+					kind="textParagraph",
+					paragraphFunction=paragraphFunc,
+					desiredValue=True,
+					direction=_Movement(direction),
+					pos=pos,
+				)
 		else:
 			iterFactory=lambda direction,info: self._iterNodesByType(itemType,direction,info)
 		info=self.selection
@@ -949,6 +999,19 @@ def _get_disableAutoPassThrough(self):
 	# Translators: Message presented when the browse mode element is not found.
 	prevError=_("no previous tab")
 )
+qn(
+	"textParagraph",
+	key="p",
+	# Translators: Input help message for a quick navigation command in browse mode.
+	nextDoc=_("moves to the next text paragraph"),
+	# Translators: Message presented when the browse mode element is not found.
+	nextError=_("no next text paragraph"),
+	# Translators: Input help message for a quick navigation command in browse mode.
+	prevDoc=_("moves to the previous text paragraph"),
+	# Translators: Message presented when the browse mode element is not found.
+	prevError=_("no previous text paragraph"),
+	readUnit=textInfos.UNIT_PARAGRAPH,
+)
 del qn
 
 

source/config/configDefaults.py

@@ -0,0 +1,33 @@
+# A part of NonVisual Desktop Access (NVDA)
+# Copyright (C) 2024 NV Access Limited
+# This file is covered by the GNU General Public License.
+# See the file COPYING for more details.
+
+DEFAULT_TEXT_PARAGRAPH_REGEX = (
+	r"({lookBehind}{optQuote}{punc}{optQuote}{optWiki}{lookAhead}|{punc2}|{cjk})".format(
+		# Look behind clause ensures that we have a text character before text punctuation mark.
+		# We have a positive lookBehind \w that resolves to a text character in any language,
+		# coupled with negative lookBehind \d that excludes digits.
+		lookBehind=r'(?<=\w)(?<!\d)',
+		# In some cases quote or closing parenthesis might appear right before or right after text punctuation.
+		# For example:
+		# > He replied, "That's wonderful."
+		optQuote=r'["”»)]?',
+		# Actual punctuation marks that suggest end of sentence.
+		# We don't include symbols like comma and colon, because of too many false positives.
+		# We include question mark and exclamation mark below in punc2.
+		punc=r'[.…]{1,3}',
+		# On Wikipedia references appear right after period in sentences, the following clause takes this
+		# into account. For example:
+		# > On his father's side, he was a direct descendant of John Churchill.[3]
+		optWiki=r'(\[\d+\])*',
+		# LookAhead clause checks that punctuation mark must be followed by either space,
+		# or newLine symbol or end of string.
+		lookAhead=r'(?=[\r\n  ]|$)',
+		# Include question mark and exclamation mark with no extra conditions,
+		# since they don't trigger as many false positives.
+		punc2=r'[?!]',
+		# We also check for CJK full-width punctuation marks without any extra rules.
+		cjk=r'[．！？：；]',
+	)
+)

source/config/configSpec.py

@@ -7,6 +7,7 @@
 
 from io import StringIO
 from configobj import ConfigObj
+from . import configDefaults
 
 #: The version of the schema outlined in this file. Increment this when modifying the schema and
 #: provide an upgrade step (@see profileUpgradeSteps.py). An upgrade step does not need to be added when
@@ -183,6 +184,7 @@
 	enableOnPageLoad = boolean(default=true)
 	autoFocusFocusableElements = boolean(default=False)
 	loadChromiumVBufOnBusyState = featureFlag(optionsEnum="BoolFlag", behaviorOfDefault="enabled")
+	textParagraphRegex = string(default="{configDefaults.DEFAULT_TEXT_PARAGRAPH_REGEX}")
 
 [touch]
 	enabled = boolean(default=true)

source/gui/settingsDialogs.py

@@ -13,7 +13,7 @@
 import os
 from enum import IntEnum
 from locale import strxfrm
-
+import re
 import typing
 import wx
 from NVDAState import WritePaths
@@ -3295,6 +3295,34 @@ def __init__(self, parent):
 
 		self.Layout()
 
+		# Translators: This is the label for a textfield in the
+		# browse mode settings panel.
+		textParagraphRegexLabelText = _("Regular expression for text paragraph navigation")
+		self.textParagraphRegexEdit = sHelper.addLabeledControl(
+			textParagraphRegexLabelText,
+			wxCtrlClass=wx.TextCtrl,
+			size=(self.Parent.scaleSize(300), -1),
+		)
+		self.textParagraphRegexEdit.SetValue(config.conf["virtualBuffers"]["textParagraphRegex"])
+		self.bindHelpEvent("TextParagraphRegexEdit", self.textParagraphRegexEdit)
+
+	def isValid(self) -> bool:
+		regex = self.textParagraphRegexEdit.GetValue()
+		try:
+			re.compile(regex)
+		except re.error as e:
+			log.debugWarning("Failed to compile text paragraph regex", exc_info=True)
+			gui.messageBox(
+				# Translators: Message shown when invalid text paragraph regex entered
+				_("Failed to compile text paragraph regular expression: %s") % str(e),
+				# Translators: The title of the message box
+				_("Error"),
+				wx.OK | wx.ICON_ERROR,
+				self,
+			)
+			return False
+		return super().isValid()
+
 	def onOpenScratchpadDir(self,evt):
 		path=config.getScratchpadDir(ensureExists=True)
 		os.startfile(path)
@@ -3394,6 +3422,9 @@ def onSave(self):
 		for index,key in enumerate(self.logCategories):
 			config.conf['debugLog'][key]=self.logCategoriesList.IsChecked(index)
 		config.conf["featureFlag"]["playErrorSound"] = self.playErrorSoundCombo.GetSelection()
+		config.conf["virtualBuffers"]["textParagraphRegex"] = (
+			self.textParagraphRegexEdit.GetValue()
+		)
 
 
 class AdvancedPanel(SettingsPanel):

tests/system/robot/chromeTests.py

@@ -2473,3 +2473,52 @@ def test_i13307():
 		]),
 		message="jumping into region with aria-labelledby should speak label",
 	)
+
+
+def test_textParagraphNavigation():
+	_chrome.prepareChrome("""
+		<!-- First a bunch of paragraphs that don't match text regex -->
+		<p>Header</p>
+		<p>Liberal MP: 1904–1908</p>
+		<p>.</p>
+		<p>…</p>
+		<p>5.</p>
+		<p>test....</p>
+		<p>a.b</p>
+		<p></p>
+		<!-- Now a bunch of matching paragraphs -->
+		<p>Hello, world!</p>
+		<p>He replied, "That's wonderful."</p>
+		<p>He replied, "That's wonderful".</p>
+		<p>He replied, "That's wonderful."[4]</p>
+		<p>Предложение по-русски.</p>
+		<p>我不会说中文！</p>
+		<p>Bye-bye, world!</p>
+	""")
+
+	expectedParagraphs = [
+		# Tests exclamation sign
+		"Hello, world!",
+		# Tests Period with preceding quote
+		"He replied,  That's wonderful.",
+		# Tests period with trailing quote
+		"He replied,  That's wonderful .",
+		# Tests wikipedia-style reference
+		"He replied,  That's wonderful.  4",
+		# Tests compatibility with Russian Cyrillic script
+		"Предложение по-русски.",
+		# Tests regex condition for CJK full width character terminators
+		"我不会说中文！",
+		"Bye-bye, world!",
+	]
+	for p in expectedParagraphs:
+		actualSpeech = _chrome.getSpeechAfterKey("p")
+		_asserts.strings_match(actualSpeech, p)
+	actualSpeech = _chrome.getSpeechAfterKey("p")
+	_asserts.strings_match(actualSpeech, "no next text paragraph")
+
+	for p in expectedParagraphs[-2::-1]:
+		actualSpeech = _chrome.getSpeechAfterKey("shift+p")
+		_asserts.strings_match(actualSpeech, p)
+	actualSpeech = _chrome.getSpeechAfterKey("shift+p")
+	_asserts.strings_match(actualSpeech, "no previous text paragraph")

tests/system/robot/chromeTests.robot

@@ -156,3 +156,6 @@ ARIA switch role
 i13307
 	[Documentation]	ensure aria-labelledby on a landmark or region is automatically spoken when jumping inside from outside using focus in browse mode
 	test_i13307
+textParagraphNavigation
+	[Documentation]	Text paragraph navigation
+	test_textParagraphNavigation

user_docs/en/changes.t2t

@@ -6,6 +6,9 @@ What's New in NVDA
 = 2024.2 =
 
 == New Features ==
+- New key commands:
+  - New Quick Navigation command ``p`` for jumping to next/previous text paragraph in browse mode. (#15998, @mltony)
+  -
 - Reporting row and column headers is now supported in contenteditable HTML elements. (#14113)
 - In Windows 11, NVDA will announce alerts from voice typing and suggested actions including the top suggestion when copying data such as phone numbers to the clipboard (Windows 11 2022 Update and later). (#16009, @josephsl)
 - Added support for the BrailleEdgeS2 braille device. (#16033)

user_docs/en/userGuide.t2t

@@ -828,6 +828,7 @@ The following keys by themselves jump to the next available element, while addin
 - o: embedded object (audio and video player, application, dialog, etc.)
 - 1 to 6: headings at levels 1 to 6 respectively
 - a: annotation (comment, editor revision, etc.)
+- ``p``: text paragraph
 - w: spelling error
 -
 To move to the beginning or end of containing elements such as lists and tables:
@@ -842,6 +843,26 @@ If you want to use these while still being able to use your cursor keys to read
 To toggle single letter navigation on and off for the current document, press NVDA+shift+space.
 %kc:endInclude
 
++++ Text paragraph navigation command +++[TextNavigationCommand]
+
+You can jump to the next or previous text paragraph by pressing ``p`` or ``shift+p``.
+Text paragraphs are defined by a group of text that appears to be written in complete sentences.
+This can be useful to find the beginning of readable content on various webpages, such as:
+- News websites
+- Forums
+- Blog posts
+- 
+
+These commands can also be helpful for skipping certain kinds of clutter, such as:
+- Ads
+- Menus
+- Headers
+- 
+
+Please note, however, that while NVDA tries its best to identify text paragraphs, the algorithm is not perfect and at times can make mistakes.
+Additionally, this command is different from paragraph navigation commands ``control+downArrow/upArrow``.
+Text paragraph navigation only jumps between text paragraphs, while paragraph navigation commands take the cursor to the previous/next paragraphs regardless of whether they contain text or not.
+
 +++ Other navigation commands +++[OtherNavigationCommands]
 
 In addition to the quick navigation commands listed above, NVDA has commands that have no default keys assigned.
@@ -2594,6 +2615,11 @@ This option allows you to specify if NVDA will play an error sound in case an er
 Choosing Only in test versions (default) makes NVDA play error sounds only if the current NVDA version is a test version (alpha, beta or run from source).
 Choosing Yes allows to enable error sounds whatever your current NVDA version is.
 
+==== Regular expression for text paragraph quick navigation commands ====[TextParagraphRegexEdit]
+
+This field allows users to customize regular expression for detecting text paragraphs in browse mode.
+The [text paragraph navigation command #TextNavigationCommand] searches for paragraphs matched by this regular expression.
+
 ++ miscellaneous Settings ++[MiscSettings]
 Besides the [NVDA Settings #NVDASettings] dialog, The Preferences sub-menu of the NVDA Menu contains several other items which are outlined below.