Embrace Standardized Color Control

[mitsuhiko]

I was trying to get people to embrace this crate before as a unified way to control colors but it did not get far: https://crates.io/crates/clicolors-control (that crate also turns on ansi colors for the windows console as an added bonus)

The idea was that you have one common crate that turns on and off colors for the entire cli app.

I'm fine finding a different mechanism for this but right now controlling colored output is really not a particularly fun experience.

Anything got some thoughts on this matter?

[killercup]

Relevant internals thread: https://internals.rust-lang.org/t/terminal-platform-abstraction/6746

[BurntSushi]

I've looked at clicolors-control before, but didn't use it for a variety of reasons on which reasonable people may disagree:

• It doesn't look at the TERM environment variable. I personally don't really care about the clicolors standard, so I think it's fine to use it, but it shouldn't be used to the exclusion of other things. When people set TERM=dumb, for example, then people expect that to be respected. (I've even found that some environments that don't support colors will do this automatically, and if you don't respect it, you end up emitting a bunch of escape codes.)
• Its tty detection on Windows is insufficient for my needs. I need it to work in MSYS2 terminals. The atty crate does this using a hack.
• My termcolor crate (which predates clicolors-control, AFAIK) already did everything that clicolors-control does, and assimilates it with common end users inputs via the ColorChoice type. (With one exception: only recently did it acquire the ability to enable ANSI escapes on Windows 10.)

I would in principle support coalescing around one crate that just did color detection, but on that same token, I'd also like to see people coalesce around one crate for dealing with a Windows console for coloring, e.g., wincolor.

[mitsuhiko]

@BurntSushi for what it's worth I do not really want to defend either the crate nor the defaults. I mostly just want a crate that holds the state of color on/off with a sensible default.

My termcolor crate (which predates clicolors-control, AFAIK) already did everything that clicolors-control does

The reason I never ended up using termcolor was that it does not actually (to the best of my knowledge) store the state of color on/off. It just has the auto color choice which derives the default for the writers. You cannot query it and ask: does the user want colors.

(My goal effectively is that I can parse --color=never and then tell the entire ecosystem of crates I use to please not emit colors)

[richard-uk1]

The way I would like to see terminal colours work is a wrapper around std(out/err) that interperets ansi escape sequences. If we are on windows before windows 10, we have to use the win32 api to process escape sequences, but on most platforms we can just pass them through. There is also terminfo/cap that should be respected.

If we do this, everyone just uses ansi escape sequences, and the wrapper can choose to discard them based on anything (TERM, configuration, etc.).

Idea from this post.

(My goal effectively is that I can parse --color=never and then tell the entire ecosystem of crates I use to please not emit colors)

Using this option, disabling colors would happen in the consuming crate, all crates further down the chain just write their ansi escape sequences as usual.

A bit of code:

struct AnsiPolyfillWrapper<W: io::Write>(W);

impl<W: io::Write> io::Write for AnsiPolyfillWrapper<W> {
    fn write(&mut self, buf: &[u8]) -> Result<usize> {
        // read buf and if necessary modify the escape codes, with many write calls, otherwise
        self.0.write(buf)
    }
}

[BurntSushi]

There is also terminfo/cap that should be respected.

I explicitly chose to not respect this, at least for things like colors or other simplistic styling like underlining or bold. The reason why is because when I did use a library that respected terminfo, I got numerous bug reports where the wrong escape codes were being emitted. Instead, I chose to follow in the footsteps of GNU grep: just stick to the basics. GNU grep is widely deployed, and I never hear anyone complaining about whether its colors work or not, which is compelling evidence from my perspective that it has the right approach here.

For CLI tools that need to do more extensive manipulation of the terminal, then this may be the wrong calculus.

In general, I agree that an ANSI interpreter would be great, and it would, for example, slide nicely into termcolor's existing API (while simultaneously removing the need for the buffer and buffer writer types). It's a nice-to-have though IMO, and we can make progress without it. :-) In particular, the kind of support you're talking about would require a dedicated maintainer.

[richard-uk1]

@BurntSushi what do you think are the most pressing issues in this area? Sorry if this question is answered elsewhere.

[mitsuhiko]

@derekdreery you still need to know if colors are on or off for other reasons. For instance I shell out to other processes which will typically not detect color correctly so I want to propagate my understanding of if color is on or off to that subprocess.

WRT to the wrapping stream: I agree that would be nice. Sadly right now there is only an undocumented API used by rusttest to intercept prints.

(Also now that windows 10 supports ansi colors after enabling i'm hoping we can just pretend ansi color is a thing and get rid of more complex wrappers around it)

[BurntSushi]

@derekdreery The hardest possible problem: getting people to agree on one library. It may be the case that you need an ANSI interpreter to do that, because termcolor's API today is not nearly as simple as crates that only handle ANSI formatting, and that annoys people enough to go out and write their own library.

@mitsuhiko's idea about at least getting all of them to agree on whether colors are enabled or not (and permit the caller to control) is a really nice middle ground and possibly much easier to achieve. I would certainly be on board with doing that for termcolor (subject to my concerns listed above).

[Screwtapello]

The idea of interpreting ANSI codes alarms me for a number of reasons:

• The number of potential ANSI codes somebody might conceivably want to send is very large, and drawing any hard line will rule out some number of reasonble use-cases
• What happens with things that look like ANSI codes but this crate doesn't know how to handle?
  • If they're passed through, you're likely to get output that works perfectly on some terminals and blurts ANSI goo in others
  • If they're silently ignored, that makes formatting problems difficult to debug
  • The library could raise an error, but is it worth killing the entire operation over an output formatting failure? Even if the culprit is just improperly-escaped user-input?
• "half assed ANSI code handling" has caused problems before (although in that case the Rust program was the producer of such codes, not the consumer)
• Even with helper functions with friendly names like make_green_text(), if your library's officially-supported API includes "put raw ANSI codes in your string literals", then people will do that, and raw ANSI codes are not an API surface that existing API tools like rustdoc are equipped to handle gracefully
• The reason that raw ANSI codes are a popular API is because it's so easy to bang strings together, and so much more difficult to work with a more structured API. This is also the reason for the popularity of cross-site-scripting attacks and SQL injection vulnerabilities. ANSI codes probably aren't as security sensitive as those other things, but still worth considering.

All that said, it may still be the case that the ease and flexibility of a custom ANSI parser outweighs all the downsides, in which case fair enough.

However, taking a cue from the "cross site scripting" analogy, I'd be interested to see/use a terminal colour API based on the ideas of HTML templating libraries like Elm's HTML DSL, ScalaTags or rust-tags. That is, building up a type-safe data-structure that represents text and its formatting, which can be stored and later serialized to ANSI-codes or Windows console API calls or HTML with inline CSS or whatever. That would hopefully have all the thread-safety advantages of ripgrep, with a more ergonomic and teachable API.

[BurntSushi]

@Screwtapello you raise a lot of good points I hadn't considered! Briefly, I will just respond to one thing here, and that's the idea of building up something more structured. In particular, for tools like ripgrep, printing output is a performance sensitive aspect of execution. It is likely that any overhead associated with emitting colors would need to be avoided, and I imagine execution templates probably wouldn't be acceptable. With that said, a convenient API doesn't need to solve every use case, so long as there are lower level APIs for when that is necessary.

[mitsuhiko]

The wider discussions a bout ansi, terminal abstractions etc. are quite complex which is why i thought about reducing this problem to initial color detection and control.

I think it might make sense to have a crate like clicolors control that just has a flippable global and then maybe various compile features to pick a sensible default (on, off, CLICOLOR, term detection etc.)

Then it should be easy for everybody to use it as a base and the rest of the discussion can be held separately.

[richard-uk1]

@Screwtapello Thanks for explaining all that! I had not considered ideas like cross-site scripting.

What I'm trying to define is the most general, minimal level, and fast API that can do everything that's required.

The number of potential ANSI codes somebody might conceivably want to send is very large, and drawing any hard line will rule out some number of reasonble use-cases.

We would draw a line somewhere, probably text styling and cursor movement to start with. This line could be moved in future, and if people used the library it would show where the gaps are and what features are most missed. These could be prioritised.

What happens with things that look like ANSI codes but this crate doesn't know how to handle?

I think there are a few options here. I believe that in production (release mode) the correct thing to do is to strip out these sequences, but in debug mode it may be useful to log their presence.

If they're silently ignored, that makes formatting problems difficult to debug

If someone comes to you with a formatting issue, you can hand them a version of the program compiled to write out a warning when it comes across an unexpected sequence.

"half assed ANSI code handling" has caused problems before (although in that case the Rust program was the producer of such codes, not the consumer)

I'd argue this would just be a bug that should be fixed. Any implementation may have bugs.

Even with helper functions with friendly names like make_green_text(), if your library's officially-supported API includes "put raw ANSI codes in your string literals", then people will do that, and raw ANSI codes are not an API surface that existing API tools like rustdoc are equipped to handle gracefully

I agree. This library should not be for user consumption. An alternative library should be advertised that implements a programmatic API, using the type system to prevent invalid input. But we still need the implementation of that API, and I propose that this is the best way to do it, or at least that it is worth investigation.

This is also the reason for the popularity of cross-site-scripting attacks and SQL injection vulnerabilities.

I need to think more about this to fully understand the risks. I think it would be the role of the higher-level API to sanitize any input, so that escape sequences are only emitted from the API calls. This would need careful auditing.

So, in summary you would have 2 levels

1. (lower) escape sequences polyfill

• fast (this is a guess because I imagine you could write a cache-friendly wrapper around the write calls, would need to check if its true or not)
• flexible (can be sent over stream before being converted into platform-specific, pre-serialized)
• complicated, and error prone if you write your own escape sequences

2. (higher) programmatic API
   • safe (type system and sanitization)
   • simpler, more ergonomic
   • slower (maybe, or will api calls just get inlined into string operations)
   • risky (sanitation needs to work, is there a difinitive list of escape sequences, or patterns of escape sequences that need to be stripped).

If you went straight for a higher level API you still have the issue of what to do with escape sequences in the input. Should you remove them, or leave them in? If you remove them, how do you know you are getting them all?

@mitsuhiko what are the specific things that are missing now that you would like to see as a minimum?

[mitsuhiko]

@derekdreery there needs to be an agreed upon way to turn color on and off. clicolors-control exists but nobody uses it and @BurntSushi outlined some of his reasons for not doing it. So what I can propose right now is to take that crate, incorporate his suggestions but I won't be able to force the rest of the community to use it :)