The position of the square root symbol is incorrect.

[crazymxm]

Description of the bug

Get text by page.get_text("dict")["blocks"], the positions （span's bbox ） of regular text and normal symbols are correct. However, the position of the square root symbol is noticeably incorrect. Please refer to the following debug information and the rectangles I redrew on the original PDF page based on the obtained bbox. It can be observed that the bbox positions are off by nearly 1 line-height to the actual positions.

Debug message:
.....
DEBUG: PDF Span:(query with all keys, divide each by) bbox:[108.0, 391.57, 247.56, 401.57] fonts:NimbusRomNo9L-Regu
DEBUG: PDF Span:(√ ) bbox:[250.05, 383.49, 258.35, 393.46] fonts:CMSY10
DEBUG: PDF Span:(d) bbox:[258.35, 391.36, 263.54, 401.33] fonts:CMMI10
DEBUG: PDF Span:(k) bbox:[263.54, 395.19, 267.77, 402.17] fonts:CMMI7
.....

Image : bbox of span and original pdf

How to reproduce the bug

TEST PDF File : https://arxiv.org/pdf/1706.03762
Test Page： P.4

PyMuPDF version

1.25.1

Operating system

MacOS

Python version

3.10

[julian-smith-artifex-com]

I have a reproduced the problem. It looks like it is not the font ascender/descender issue that has just been fixed on main.

[julian-smith-artifex-com]

See MuPDF bug: https://bugs.ghostscript.com/show_bug.cgi?id=708269

[julian-smith-artifex-com]

Thanks for reporting this.

It looks like the font bbox ascender/descender information for the square root symbol is wildly inaccurate.

However there is a partial workaround that you could try, which is in two parts:

1. Call pymupdf.TOOLS.unset_quad_corrections(True). This is a global setting.
2. Pass flags=pymupdf.TEXT_ACCURATE_BBOXES to Page.get_text().

Number 1 stops PyMuPDF trying to do its own patching up of bboxes, which allows number 2 to work - TEXT_ACCURATE_BBOXES makes MuPDF calculate the exact bboxes of characters by looking at the actual drawing operations. This will probably make things slower, but it's probably the only way of reliably coping with a broken font.

Using these accurate bboxes for individual characters isn't very good for highlighting whole spans or lines, hence the description as being only a partial workaround.

[We're looking at making number 1 be the default in future releases, but this will need some more testing and isn't certain yet.]

[westNeighbor]

@julian-smith-artifex-com
When I deal with math fonts, there are many cases (for big operators), the bboxes are not correctly cover the exact shape of the chars. The accurate option does solve my desired exact bboxes problem. I attached the codes for visualize the problems.

But for the square root problem, it looks one more step:

• the bbox position is wrong, but only for the normal size square root, the enlarged size ones are indeed correct.
• when accurate option added, the bbox covers the shape correctly, but the origin position doesn't move, please also look into the origin position problem.

You can see picture 1 without accurate bbox option, some bboxes are not cover the shapes, square root's bboxes are not positioning correctly except the enlarged one. The red points are the origin positions.

The picture 2 shows that with accurate option solves the actual size problem, but not the square root's position problem.

Also, for the composed big math {,} symbols, the decomposes have extra {,} symbols, this may be not the pymupdf problem
You can see the actual chars for the matrix, the ( and [ are right, the { has extra
Char: ⎡, Unicode: U+23A1, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎢, Unicode: U+23A2, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎣, Unicode: U+23A3, Font: LatinModernMath-Regular, FontSize: 9.96

Char: ⎤, Unicode: U+23A4, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎥, Unicode: U+23A5, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎦, Unicode: U+23A6, Font: LatinModernMath-Regular, FontSize: 9.96

Char: ⎛, Unicode: U+239B, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎜, Unicode: U+239C, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎝, Unicode: U+239D, Font: LatinModernMath-Regular, FontSize: 9.96

Char: ⎞, Unicode: U+239E, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎟, Unicode: U+239F, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎠, Unicode: U+23A0, Font: LatinModernMath-Regular, FontSize: 9.96

Char: ⎧, Unicode: U+23A7, Font: LatinModernMath-Regular, FontSize: 9.96
Char: {, Unicode: U+007B, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎨, Unicode: U+23A8, Font: LatinModernMath-Regular, FontSize: 9.96
Char: {, Unicode: U+007B, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎩, Unicode: U+23A9, Font: LatinModernMath-Regular, FontSize: 9.96

Char: ⎫, Unicode: U+23AB, Font: LatinModernMath-Regular, FontSize: 9.96
Char: }, Unicode: U+007D, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎬, Unicode: U+23AC, Font: LatinModernMath-Regular, FontSize: 9.96
Char: }, Unicode: U+007D, Font: LatinModernMath-Regular, FontSize: 9.96
Char: ⎭, Unicode: U+23AD, Font: LatinModernMath-Regular, FontSize: 9.96

The visual debug code:

import pymupdf
import os
def math_text_info(pdf_path, visual_box=True, accurate_box=False):
    # Open the PDF file
    if accurate_box:
        pymupdf.TOOLS.unset_quad_corrections(True)
    pdf_document = pymupdf.open(pdf_path)

    # Iterate through pages
    math_texts = []
    for page_num in range(pdf_document.page_count):
        page = pdf_document[page_num]
        text = ''
        if accurate_box:
            text = page.get_text("rawdict", flags= pymupdf.TEXTFLAGS_RAWDICT | pymupdf.TEXT_ACCURATE_BBOXES)  # Extract text in dictionary format
        else:
            text = page.get_text("rawdict")  # Extract text in dictionary format
        #print(text)
        #print('\n')

        m_chars = []
        for block in text["blocks"]:
            for line in block["lines"]:
                for span in line["spans"]:
                    font = span["font"]
                    font_size = span["size"]

                    # Extract each character
                    chars = span["chars"]
                    for char in chars:
                        if char["c"].strip():  # Ignore whitespace characters
                            # Get character Unicode value
                            unicode_value = ord(char["c"])

                            # Get character position
                            origin_xy = char["origin"]
                            y = origin_xy[1]
                            x = origin_xy[0]
                            height = char['bbox'][3] - char['bbox'][1]
                            if visual_box:
                                page.draw_circle(char['origin'], color=(1.0,0.0,0.0), radius=0.5)
                                page.draw_rect(char['bbox'], color=(0.0,0.6,0.6), width=0.2)

                            m_chars.append({"Char": f"{char['c']}", "Unicode": f"U+{unicode_value:04X}", "Font": f"{font}", "FontSize": f"{font_size:.6f}", "Location": f"({x:.6f}, {y:.6f})"})

                            print(f"Char: {char['c']}, Unicode: U+{unicode_value:04X}, Font: {font}, FontSize: {font_size:.2f}, Height: {height:.2f}, Location: {char['origin']}, bbox: {char['bbox']}")
        math_texts.append(m_chars)
    #print(math_texts)
    if visual_box:
        file_path = os.path.dirname(pdf_path)
        file_name, _ = os.path.splitext(os.path.basename(pdf_path))
        save_file = f"{file_path}"+ os.sep + f"{file_name}_visualbox.pdf"
        if accurate_box:
            save_file = f"{file_path}"+ os.sep + f"{file_name}_visualbox_accurate.pdf"
        pdf_document.save(save_file)

# Replace 'your_pdf.pdf' with the path to your PDF file
math_text_info("./op-demo.pdf", True, True)

op-demo.pdf

op-demo_visualbox.pdf

op-demo_visualbox_accurate.pdf

[JorjMcKie]

We do not intend to change the origin value. When deviations happen, the font after all has errors. While - when ACCURATE BBOXES is requested - we are capable of computing a boundary box by following the graphics operators of the glyph, there is no way to derive a non-disputable value for the origin.

So you will have to live with whatever the font is telling you or simply assume the bboxe's bottom-left point as a fallback or whatever.