Support or document localization of messages and flags

[kamalmarhubi]

Make it easy for developers to support translations of their projects.

---

Current steps:

• Run cargo add clap -F string
• If using derive API
  • Replace field names being used as value names via #[arg(value_name = ...)]
• If using subcommands:
  • Replace Commands: header with a localized value via Command::subcommand_help_heading
  • Replace <COMMAND> value name with a localized value via Command::subcommand_value_name
• Replace Arguments and Options: headers with localized values via Command::next_help_heading and/or Arg::help_heading
• Disable the built-in flags (Command::disable_help_flag(false) and Command::disable_version_flag(false)) and provide your own versions (with ArgAction::Help, ArgAction::Version) with localized Arg::help
• Replace hard coded strings in help template (e.g. Usage) with a localized version via Command::help_template
  • Requires styling Usage manually
• Remove hard coded [] messages with Arg::hide_default_value, Arg::hide_possible_values, Arg::hide_env
• Fork clap::error::RichFormatter, replacing any hard coded strings with localized values. Use Error::apply to swap the formatter (e.g. use Parser::try_parse to get the error, call apply, and then err.exit())
• Replace in usages the collapsed optional named arguments placeholder [OPTIONS] by providing a custom usage with Command::override_usage

Unresolved:

• Getting the suggestion to run --help in errors, see How to make custom -h achieve the same effect as default -h #5409
• Manually writing usage and losing out on the adaptive usage in errors

Past examples:

• Migrate to clap, and localize usage & help text str4d/rage#442

[kbknapp]

This is something I'm interested in doing, but it will take some thought on how to best go about this. I'm putting it on the issue tracker, but let's get some discussion going on ways about this!

[kbknapp]

@kamalmarhubi I was thinking about this today, and would the YAML feature allow this? I.e. I'm thinking having a differente yaml file for each translation, and depending on which language/translation is compiled via features, you pick a different yaml file.

Example:

# Cargo.toml

[features]
default = ["en_US"]
en_US = []
es_ES = []

# en_US.yml
name: My Application
version: 1.0
about: An example using many languages
author: Kevin K. <[email protected]>
args:
    - opt:
        help: an example option
        short: o
        long: option
        multiple: true
        takes_value: true
    - pos:
        help: example positional with possible values
        index: 1
        possible_values:
            - fast
            - slow

# es_ES.yml
name: Mi Aplicacion
version: 1.0
about: Un ejamlo util muy idiomas
author: Kevin K. <[email protected]>
args:
    - opt:
        help: un ejamplo de un opcion
        short: o
        long: opcion
        multiple: true
        takes_value: true
    - pos:
        help: un ejamplo de un posicion con valores posible
        index: 1
        possible_values:
            - rapido
            - lento

// main.rs
#[cfg(feature = "en_US")]
const TRANSLATION: &'static str = "en_US.yml";

#[cfg(feature = "es_ES"]
const TRANSLATION: &'static str = "es_ES.yml";

fn main() {
    let yml = load_yaml!(TRANSLATION);
    let m = App::from_yaml(yml).get_matches();
}

Then depending on if you compile with --features en_US or --features es_ES it will set the translation. Does this work for you?

[kamalmarhubi]

Then depending on if you compile with --features en_US or --features es_ES it will set the translation. Does this work for you?

It's not me I'm thinking about, just something a user-facing system should allow for. :-)

And as for the solution, it would be much better to allow the user's locale settings to dictate the language in help. Requiring users to compile the tool with their language is non-ideal. Hooking into gettext or l20n would be a nice approach, as there's tooling for people to work on translations, and they can live outside of the binary.

[kamalmarhubi]

Though I guess there's a question of whether the options should be translated, or just the help... which comes up in your example. Thanks for that by the way, concrete is always good to look at!

[joshtriplett]

I've never seen a program that translates long option names; those are API, much like the names of functions in a library. Translating the help is a great idea, though.

I'd definitely agree with hooking into gettext, to work with the massive amount of translation infrastructure and tooling built around that, rather than creating a new translation format.

[hoodie]

I am currently trying to localize my app using crowbook-intl which is pretty nice. Unfortunately, the fact that AppMeta etc don't want to own the description strings makes it extremely hard to use dynamic strings in .about() and .help() methods. Do you have an idea to work around that? Could the next implementation perhaps own these strings?

[kbknapp]

@hoodie the strings only need to live as long as the App instance, so long as the localized strings are owned somewhere it shouldn't be an issue.

Can you paste some code that's giving issues and I can either see if there's a workaround or consider fixes if appropriate.

[hoodie]

infact you are right, it's just a bit tricky dealing with the lifetime constraints. I built so that app is immediately passed to a closure now.

[pksunkara]

This would also include localization of clap strings such Usage, Flags, etc..

[alerque]

I've come very close to achieving a localized CLI interface using Clap & Fluent. However there are a couple points that are a bit hackish and could use some first class support.

The main ugly bit about my solution is I have to parse args twice. I have a -l/--language parameter that can override any detected locale (not appropriate for most CLIs, useful for my case because my CLI generated other resources that have to be localized and uses that flag. In order to get any help messages localized to match I have to pre-parse to scrape out the language, then re-parse with Clap to get everything else.

The most useful improvement I can see would be for the derive variant of Clap (formerly StructOpt), where the documentation strings are relied on. Adding top level support for a macro that would call a function to get strings rather than using the doc strings would allow both pretty and functional setup.

[kbknapp]

@alerque See #1880 and if something like that would fit your use case if we implement it.

The most useful improvement I can see would be for the derive variant of Clap (formerly StructOpt), where the documentation strings are relied on. Adding top level support for a macro that would call a function to get strings rather than using the doc strings would allow both pretty and functional setup.

This might be useful outside of i18n context. I'm imagining a help_with(Fn). However, a few non-trivial questions would be need to be ironed out to make it general enough for inclusion:

• Should the Fn accept a context? I.e. an ArgMatches, Id, or a String perhaps? No context makes it easier to implement, but then means consumers are relying on out of band global state to make decisions.
• Would this be App level (applied to all Args), or Arg level?
• Should the short/long help be different, or the same Fn (kinda goes back to the Context question above).

I personally think it might be a little too much to bite off on by the 3.0 release, but there isn't any reasons we couldn't address it post 3.0 as a feature to add if everything can be fleshed out and the complexity/maint burden isn't too high.

[alerque]

@kbknapp

1. I'm not sure my understanding of Rust is developed enough to meaningfully address this question. I think the answer is having a context would introduce many more possibilities, but it would be useful even without — but I'm still struggling with Rust's variable lifetimes (I'm about 4 days in teaching myself through a pet project that I'm RIIR).

   Note that for my usage I've had to move the #[derive(Clap)] struct Cli {} and all related definitions out of the top level scope and into my fn main(). This is so that I can do a a few early initialization things like make a guess at the locale and pre-parse some arguments¹ before I can properly setup and run using Clap.

2. Arg probably. Of course you have to set it on the app too (#[clap(about = "<my value>")] needs to be set), but there might be reasons to use different functions or otherwise customize how an argument setup. An App level default callback function with Arg level customization would be my preference.

3. I'm of the opinion that the short/long options should be fixed because scripting with a program that changes it's API based on locale is a nightmare, only the descriptive about, help, etc would change in my implementation. I know there are people that feel like option names should be localized too, but it wouldn't suit my use case.

No worries about the 3.0 release, I want to see that go out the door with the baked in derive stuff sooner rather than later. But of course I'd love to see a follow up with the tooling for localized interfaces.

¹ _{Thanks for the link to #1880, that is exactly what I've needed and was hacking up by hand and messing with lighter weight argp parsing crates to handle a first pass to extract things needed for a full instantiation with Clap.}

[kbknapp]

@alerque no worres, I should have been more clear. I wasn't posing those questions for you to have to answer. They were more abstract thoughts, primarily for the other maintainers of clap 😉

[alerque]

The recent Rust 1.54.0 release includes a bit I think might have a huge impact on how hard this is to implement:

You can now use macros for values in built-in attribute macros.

[luojia65]

Is it possible to read and use current operating system's working locale settings (languages, money unit, etc.) in clap generated help messages? For example, on given operating system, users use --help and it shows help messages in current locale. Clap can provide a default way to define locales in command line, or remains to users to define them. It would be more nice if this can be implemented, as users in different culture could use the clap based applications more freely.
In this way we should (maybe) provide a way to write localized strings by locale setting input. Clap only provided as-is, it does not parse the locale setting string, but provide them to user to choose which locale page clap should use.
There would be two ways to implement them:

1. Use localization index in current clap help messages, and translate them using another table. For example, the desctiption of clap app may be myapp.ui.description, but there is another table to translate myapp.ui.description and en_US into 'My Awesome App!'.
2. Use a page based locale strings. There could be multiple ways to achieve this, but it requires large scale of code reformat to clap project.

[epage]

In discussing this with someone on reddit, I think users can get something working, even if it isn't streamlined yet

1. Replace Commands: header with a localized value via Command::subcommand_help_heading
2. Replace <COMMAND> value name with a localized value via Command::subcommand_value_name
3. Replace Arguments and Options: headers with localized values via Command::next_help_heading and/or Arg::help_heading
4. Disable the built-in flags (Command::disable_help_flag(false) and Command::disable_version_flag(false)) and provide your own versions with localized Arg::help
5. If the help template has any hard coded strings, replace them with a localized version via Command::help_template
6. Fork clap::error::RichFormatter, replacing any hard coded strings with localized values. Use Error::apply to swap the formatter (e.g. use Parser::try_parse to get the error, call apply, and then err.exit())

Would some users be willing to experiment with this within their applications and report on the results and what can be changed / improved?

[saona-raimundo]

I just tried it in one of my small projects and everything worked well :)

I was happy when I found that, on top of the listed methods for Command, there are special action variants ArgAction::Help and ArgAction::Version. This came in handy so I did not have to code the functionality of these flags again.

In my case, I had to include the string feature of clap, because I got Strings out of fluent.
It was not hard to find, but it would be useful to have in mind when writing docs about this use case.

If I get to do more experiments, I will comment on them.

One thing that clap does well and I ended up doing by hand is formatting the (localized) help message.
Hinting at some resources on command-line templating would be great.

[mleduque]

Hi, I really want to have the help strings of my program translated, and this seems to be the issue about it.
Though it seems clap doesn't support it, I gather from the comments there is a workaround (possibly with heavy work?), but I feel the solution is alluded to, not explained and I can't manage to assemble the pieces.
Is there and example or howto?

[epage]

@mleduque As my comment alludes to, I've never tried this out and don't have good use cases for trying it out and am looking for someone to pioneer this. I did update my comment with some more explicit steps.

[mleduque]

I've read your comment once again but I don't find where to start from.
I've got a #[derive(Parser)] struct where do I go from there?

[epage]

It sounds like some more basic information is needed and this issue is not a good place for us to have that conversation. I will also say that this isn't my focus area at this time and I'm only able to provide limited support.

The one thing I will point out is that all builder methods are exposed as attributes in the derive (see reference). So you can take Command::disable_help_flag and use it like #[command(disable_help_flag = false)]. Similar for each of the other steps (except 6 but I already described that).

[mleduque]

The use case it, well, the most classic there is :

Use the LANG/LC_ALL/LANGUAGE environment variable to chosose at runtime which help message you will show

Which is probably a minimum for any CLI program.
I'll try this and see if I can do something with the info above.

[str4d]

@epage wrote:

Would some users be willing to experiment with this within their applications and report on the results and what can be changed / improved?

I have experimented with this in str4d/rage#442, and successfully localized my usage and help text! See str4d/rage@b3de9b9 specifically for the localization changes (relative to what I was able to do previously before using clap, which was only localizing the after_help text).

The first issue I ran into was that passing localized Strings into the various fields that take Str didn't report very helpful error messages in derive mode. I had to manually look at the documentation for Str to figure out that I needed to enable the string feature flag.

1. Replace Commands: header with a localized value via Command::subcommand_help_heading

I didn't test this, as my app doesn't use subcommands.

2. Replace <COMMAND> value name with a localized value via Command::subcommand_value_name

I didn't test this, as my app doesn't use subcommands. However I did use Arg::value_name to localize the positional and flag value names.

3. Replace Arguments and Options: headers with localized values via Command::next_help_heading and/or Arg::help_heading

This was a little confusing to get working: just setting Command::next_help_heading will cause it to merge Arguments and Options together. To avoid this, I set Command::next_help_heading to the localized Options (as there were more of these), and then set Arg::help_heading on every positional argument.

4. Disable the built-in flags (Command::disable_help_flag(false) and Command::disable_version_flag(false)) and provide your own versions with localized Arg::help

This was somewhat confusing, as the derive syntax to get replacements working required them to use the argument type Option<bool> in order to get past the "required flag" checker.

5. If the help template has any hard coded strings, replace them with a localized version via Command::help_template

The help template hard-codes the "Usage" string inside the {usage-heading} tag:

clap/clap_builder/src/output/help_template.rs

Lines 199 to 206 in 2ab48b2

	"usage-heading" => {
	let _ = write!(
	self.writer,
	"{}Usage:{}",
	self.styles.get_usage().render(),
	self.styles.get_usage().render_reset()
	);
	}

I copied in the contents of DEFAULT_TEMPLATE so I could replace that tag with a manual localized implementation, but to emulate {usage-heading} correctly I needed to duplicate the command style inside the formatter (which meant I had to figure out what style my app was using - Styles::default() - and enable the unstable-styles feature flag).

6. Fork clap::error::RichFormatter, replacing any hard coded strings with localized values. Use Error::apply to swap the formatter (e.g. use Parser::try_parse to get the error, call apply, and then err.exit())

I have not yet tested this, but I was already doing something similar for my own app's errors (for invalid flag combinations etc) prior to using clap, so I might look into this at some point (either forking clap::error::RichFormatter to replace my logic, or integrating clap's errors into my logic).

---

The only other hard-coded string I noticed was the OPTIONS name used in the auto-generated usage string:

clap/clap_builder/src/output/usage.rs

Lines 173 to 180 in 2ab48b2

	if used.is_empty() && self.needs_options_tag() {
	let _ = write!(
	styled,
	"{}[OPTIONS]{} ",
	placeholder.render(),
	placeholder.render_reset()
	);
	}

This can be localized with Command::override_usage, but doing so disables the "context-aware" usage strings. I already needed to do this for one of the CLI apps (as I want to show two usage variants), but the other two would benefit from some way to alter this string.

[epage]

Thanks for the detailed report!

It seems the biggest issue is OPTIONS at this point. The thing I'm trying to weigh is the cost to everyone else for a localization change.

Options

• Add a setting in Command
• Use the in-work plugin system for a setting on Command so people using it aren't expected except in usage.rs code
• Have localization use custom usage generators some time after Design Document: First Step Towards Modularity #5236

[bin-ly]

Hello, could you please tell me what the current demand for localization is? Can clap support it?

[epage]

@bin-ly I've updated the issue post with the current localization instructions based on the work of str4d

[toadslop]

Hey all, I made a draft PR of an approach to enabling internationlization clap: #5853

Take a look and let me know what you think. It's just a minimal example to demonstrate the approach, but if it looks good I'd be happy to put in some time to polish it up.

[epage]

From #5852

While I recommend moving the conversation to the mentioned issue, I'll at least say I hesitate in making a bespoke localization framework like I was about a bespoke terminal styling API.

As shown in this issue, we are most of the way there to allowing people to override existing strings inside of clap.

[alerque]

Yes, lets please not move towards a besoke YAML key/value based localization system. It is easy to get off the ground that way but in will quickly become a problem for projects wanting to do a better job or with more complex needs. Good localization is hard even for relatively simple cases, and there is good tooling out there for it.

This should be refactored to use Fluent (or possibly one of the Fluent based systems that allow generating static localizations), or just provide an interface that people can bring their own localization framework). Any project localizing their CLI args probably also has other localization needs and probably already has Fluent assets or some other framework that can be leveraged for this purpose.