bad ansi escape sequences parsing?

[name-snrl]

In some cases escape sequences are not converted to colors.

---

to reproduce:

nvimpager -p part.log

part.log

nvimpager full.log

full.log

I use fish, many of its man pages have the same problem, e.g. man fish-doc

---

I have tried several setups (with alacritty 0.11.0):

• NVIM v0.5.1 +nvimpager-0.10.4 (nixpkgs 21.11)
• NVIM v0.7.0 +nvimpager-0.10.4 (nixpkgs 22.05)
• VIM 8.2 (2019 Dec 12, compiled Jan 01 1980 00:00:00) +vimpager 2.06 (nixpkgs 22.11
  nix shell nixos/nixpkgs/nixos-21.11#vim nixpkgs/nixos-21.11#vimpager-latest)

Also, I tried changing TERM to xterm-256color. Everywhere the same result.

[lucc]

What is the \e[K sequence supposed to do? If I cat part.log they don't seem to do anything.

PS: vimpager is unrelated.

[lucc]

The full log is not highlighted because nvimpager just checks the first 100 lines to decide if we should highlight terminal escape codes (function check_escape_sequences() in pager_mode())

[name-snrl]

What is the \e[K sequence supposed to do?

I don't really understand what it does. But often "less" is used with -R, this option ignores all sequences except color. I think we need implement this option.

[name-snrl]

The full log is not highlighted because nvimpager just checks the first 100 lines to decide if we should highlight terminal escape codes (function check_escape_sequences() in pager_mode())

Can we use io.read() instead of vim.api.nvim_buf_get_lines()?

[lucc]

about: io.read: why? I think it reads stdin or we need to open the file again with io.open. But nvim has opened it already so why not use nvim_buf_get_lines?

about ignoring unknown sequences: there are quite some of them: https://en.wikipedia.org/wiki/ANSI_escape_code

When stripping them from the buffer we match much more than when concealing them:

nvimpager/lua/nvimpager.lua

Lines 372 to 379 in 2251f29

	local function strip_ansi_escape_sequences_from_current_buffer()
	local modifiable = nvim.nvim_buf_get_option(0, "modifiable")
	nvim.nvim_buf_set_option(0, "modifiable", true)
	nvim.nvim_command(
	[=[keepjumps silent %substitute/\v\e\[[;?]*[0-9.;]*[a-z]//egi]=])
	nvim.nvim_win_set_cursor(0, {1, 0})
	nvim.nvim_buf_set_option(0, "modifiable", modifiable)
	end

nvimpager/lua/nvimpager.lua

Lines 635 to 637 in 2251f29

	local function ansi2highlight()
	nvim.nvim_command(
	"syntax match NvimPagerEscapeSequence conceal '\\e\\[[0-9;]*m'")

I have to read through the wiki article a bit and improve these regexes I think (help obviously apreciated :)

[name-snrl]

io.read: why? I think it reads stdin or we need to open the file again with io.open. But nvim has opened it already so why not use nvim_buf_get_lines?

I don't like this limitation of a hundred lines. If we check the whole file, io.read will be faster.

simple test:

local function with_io_read()
  local start = os.clock()
  local file = io.open(vim.fn.expand('%'), "r"):read("*all")

  file:find('\27%[[;?]*[0-9.;]*[A-Za-z]')
  return os.clock() - start
end

local function with_for_loop()
  local start = os.clock()

  for _, line in ipairs(vim.api.nvim_buf_get_lines(0, 0, -1, false)) do
    if line:find('\27%[[;?]*[0-9.;]*[A-Za-z]') ~= nil then
      return os.clock() - start
    end
  end

  return os.clock() - start
end

local function test()
  local msg = string.format('io.read  = %s\nfor-loop = %s\n-----',
    with_io_read(), with_for_loop())
  vim.api.nvim_echo({ { msg } }, true, {})
end

vim.keymap.set('n', 't', test)

[name-snrl]

I don't like this limitation of a hundred lines

OR we can simply create an option for this

[lucc]

I introduced the limit because somebody might use nvimpager with a large log file. At work I have log files that are easily 1-10 GB in size. No idea how long it would take with these.

I tested your example with the full.log file you provided and io.read is indead faster. I will have to see with a bigger log file.

[bqv]

What is the \e[K sequence supposed to do? If I cat part.log they don't seem to do anything.

I don't know what it does, but like I mentioned, it needs to be at least hidden.

often "less" is used with -R, this option ignores all sequences except color. I think we need implement this option.

good idea

[illfygli]

Seems like RGB colors are not parsed either.
Try e.g.

printf 'normal\e[48:2:0:162:31mgreen\n'

and then open that.

[lucc]

@illfygli currently only color sequences constructed with semicolon are recognized.

Reading Wikipedia it is not clear to me if the colon syntax is officially defined by the ansi standard and if so if it is the same as the semicolon syntax.

I tested xfce-termial, terminator, konsole, allacritty, kitty and xterm and all of them support your example. st does not.

[illfygli]

I tested xfce-termial, terminator, konsole, allacritty, kitty and xterm and all of them support your example. st does not.

Interesting, I didn't notice that it was not using ; when I copied an example. 😅

With ;, the colors are filtered out, but it would be nice if they were supported, like the basic colors.

[lucc]

@illfygli thanks for the heads up. I was under the impression that the escape sequences should be concealed but the colors displayed. This is a bug, I think that this did work at some point.

[lucc]

@illfygli I checked again and it depends on the value of 'termguicolors'. So please try set tgc.

[illfygli]

@illfygli I checked again and it depends on the value of 'termguicolors'. So please try set tgc.

Ah yes, that works. Thanks!