Typometer

Typometer is a tool to measure and analyze the visual latency of text editors.

Editor latency is the delay between an input event and a corresponding screen update — in particular, the delay between keystroke and character appearance. While there are many kinds of delays (caret movement, line editing, etc.), typing latency is a major predictor of editor usability.

Check the article typing with pleasure to learn more about editor latency and its effects on typing performance.

Download

Check the releases section.

Java 8 or later is required to run the program. You can download Java from the official site.

Build

You will need a JDK and Maven. Once you have them set up, you're ready to build Typometer.

Open your terminal of choice, and set your working directory to this project's root (where pom.xml is located). Then, build with Maven:

mvn clean package

If the compilation is successful, you will get a target directory, with the executable typometer-<version>.jar available inside.

Features

• Cross-platform (Windows, Mac, Linux).
• Native API calls for faster screen access.
• Synchronous and asynchronous modes.
• Import and export of CSV data.
• Summary statistics, frequency distribution.
• Line and bar charts (including comparative ones).
• Chart image export (with legends).

Screenshots

Main window:

Frequency distribution chart:

Principle

The program generates OS input events (key presses) and uses screen capture to fully automate the test process.

At first, a predefined pattern (a series of "." characters) is inserted in the editor window in order to detect screen metrics (start position, step, background, etc.).

After that, the program types a predefined number of "." characters into the editor (with given periodicity), measuring delays between key presses and corresponding character drawings.

To achieve high accuracy of measurement, only a single pixel is queried for each symbol. Moreover, the program can use fast native API (WinAPI, XLib, and Cocoa) calls on supported platforms, offering AWT robot as a fallback option.

There are two modes of testing available:

• Synchronous – the program always waits for a typed character to appear before making a pause and typing the next character. It's the most accurate method (because there's no threading overhead).
• Asynchronous – the program types and recognizes characters independently. This method is slightly less accurate, but it's useful for testing rapid typing when editor drawing might easily lag by multiple characters.

Usage

To register only essential editor latency, the text must be rendered directly to the framebuffer, without intermediate image processing that might introduce additional delay. Prefer stacking window managers to compositing window managers for the testing purposes, particularly:

• Switch to Classic theme in Windows. Windows Aero enforces internal vertical synchronization, which leads to a minimum of 1 frame lag (about 17 ms for 60 Hz monitor refresh rate) and to delay discretization. It's also possible to disable the compositing directly in Windows 7 and in Windows 8.
• Use a Linux distribution with a lightweight window manager, like Lubuntu (Openbox). Complex, 3D-based windows managers might substantially increase system rendering latency, for example, on my hardware, Ubuntu's Compiz, adds ~10 ms unavoidable lag.

Close all programs that add system-wide keyboard hooks, as they might process the keyboard events synchronously and affect the results (for example, Workrave is known to noticeable increase the typing latency).

You may consider switching your machine in a particular hardware mode (power scheme, integrated/discrete graphics, etc.). In power save mode (and on battery), for example, editor responsiveness is usually much lower, so it's possible to detect significant performance glitches that are less frequently observable otherwise.

Before you start benchmarking, make sure that other applications are not placing a noticeable load on your system. It's up to you whether to warm-up VM-based editors, so they can pre-compile performance-critical parts of their code before proceeding.

If possible, enable non-block caret (i.e. underline or vertical bar instead of a rectangle) in the editor. This might increase measurement accuracy.

A typical action sequence:

1. Specify a measurement title, like "HTML in Vim" (optional, can be set later).
2. Configure test parameters (optional).
3. Launch an editor, maximize its window.
4. Open some data in the editor, for instance, a large HTML file (optional).
5. Place editor caret in the desired context (e.g. a comment), at the end of a short or empty line.
6. Start benchmarking process in the program.
7. After a corresponding prompt, transfer focus to the editor window.
8. Wait for test completion, don't interfere with the process.

You can always interrupt the testing process simply by transferring the focus back to the program window.

After the test result is acquired, you may either analyze the singular data by itself or perform additional tests (different editors and conditions) to do a comparative analysis.

Both source and aggregate data are easily accessible. You can:

• Copy the table content as text.
• Save the chart to PNG file (with legend and summary stats).
• Export raw data in CSV format (for Calc or R, if you fancy).

It's possible to merge the results either by inserting data from an existing CSV file or by appending data to a CSV file on saving.

Recipes

Here are a few tips on how you can use the tool to detect performance bottlenecks in text editors:

• Check whether the number of lines influences the latency. If so, typing might "lag" inside a large file.
• Check whether editor window size influences the latency. If so, the editor probably does excessive repainting instead of drawing only a newly inserted symbol.
• Check whether latency depends on the horizontal symbol position. Ideally, that correlation should be undetectable.
• Try to enable highlighting, autocomplete, folding, spellchecker, etc. Those features should be processed asynchronously, without affecting the typing as such.
• Try to run the test in power-saving mode. Ideally, typing should be handled decently even on less powerful hardware.

If you're implementing a text editor, take a look at the programming techniques to significantly reduce the drawing latency.

Troubleshooting

To make benchmarking possible, correct screen metrics must be detected at the initial step. The program attempts to recognize a custom pattern (5 new dots) to determine the following parameters:

• Starting position.
• Horizontal step.
• Background color.
• Line length.
• Caret type.

Because there are many editors (and multiple versions of each editor), which look different on different platforms, and there are many possible color schemes and fonts, the metrics recognition algorithm has to be very flexible. While the program sources contain many test cases, some glitches are still probable.

Here's a list of typical problems and corresponding solutions:

• Editor background is non-uniform (gradient, picture) – set solid color background.
• Characters are too low-contrast and obscure – use a crisp color scheme.
• Dot characters merge with the caret – increase font size.
• The editor replaces multiple dots with an ellipsis – disable that auto-correction.
• Spaces between dots are uneven – use monospaced fonts in the editor.
• The editor has a left panel that melds with the text area – hide the panel.

Feel free to contribute by creating additional test case images (check src/test/resources).

Pavel Fatin, https://pavelfatin.com