Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs: Improved introduction to i18n for Cog authors and Translators #6351

Open
wants to merge 3 commits into
base: V3/develop
Choose a base branch
from
+71 −10
Open

Docs: Improved introduction to i18n for Cog authors and Translators #6351

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b96442a
Docs: Improved introduction to i18n for Cog authors and Translators
jamesread Apr 11, 2024
c685f9e
Docs: Improved introduction to i18n for Cog authors and Translators (…
jamesread Apr 11, 2024
10bd410
Apply suggestions aikaterna's from code review
jamesread Apr 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 71 additions & 10 deletions docs/framework_i18n.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,54 @@
Internationalization Framework
==============================

-----------
Basic Usage
-----------
--------
Tutorial
--------

The Red framework supports translations into different languages, making it easy
for translators to contribute new language files to the Red project, and to Cogs
that use the framework. The framework is based on the popular gettext format,
Comment on lines +14 to +16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The Red framework supports translations into different languages, making it easy
for translators to contribute new language files to the Red project, and to Cogs
that use the framework. The framework is based on the popular gettext format,
Red supports an internationalization framework designed to allow users to
translate user-facing strings to a contextually defined language. This
framework is available for Cog Creators to use to define translations for the
strings in their cogs. The framework is based on the popular gettext format,

Just calling it "The red framework" in the first line is using an adjective without its noun. Splitting this sentence in two allows it to be explained in a bit more depth, while avoiding a run on.

(with some modifications), which is the standard for providing translations in
lots of different software projects.

* ``.pot`` files are typically generated by the original author of the cog, and
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section jumps into listing the structure of the files used for translations without any setup. I'm not sure if this needs to be moved, or if it just needs a sentence leading into the content about to be presented.

contain the basic messages in en_US (the language Red is written in), and is
used as a template for translators to build their translations on.
Comment on lines +20 to +22
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* ``.pot`` files are typically generated by the original author of the cog, and
contain the basic messages in en_US (the language Red is written in), and is
used as a template for translators to build their translations on.
* ``.pot`` files are typically generated by the original author of the cog, and
contain the basic messages in en_US (the language Red is written in). This
file is used as a template for translators to build their translations on.

Less long winded as two sentences.

* ``.po`` files (portable objects) are language specific, for example
``en-GB.po`` for British English, or ``en-ES.po`` for Spanish. Note that Red
uses a ``-`` character in the filename, not a ``_`` which is often seen as as
the standard.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhere, possibly here, you should link to https://translate.discord.red/. That site is where translators can contribute to Red's core translations, and lists all of the languages Red supports.

* ``locales/`` directory, where ``.po`` files are stored.

So, if your are the author of a Cog, or you are adding translations for the first
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence is too unsure to be here. At the highest level, translating is a 3 step process:

  1. Update the code to reference strings in the proper way that they will be translated, and can be detected as needing translations by redgettext
  2. Use redgettext to generate a messages.pot file (must be re-done if strings are changed/added)
  3. Clone the latest messages.pot file into a lang-code.po file, and fill out the msgstrs for each string with the appropriate translation for that language/string.

At this point in the guide, we only know vaguely what a .pot and .po file are. We should not already be telling the CC they need to generate a .pot file before they have actually updated the code to support translations. We also should not be implying to translators that they can translate a cog simply by generating a .pot file without touching its code.

Realistically there should probably be 3 sections for each of these 3 steps, rather than For cog creators and For translators sections.

time to python code, you would probably generate ``.pot`` files to make your code
easily translatable. See below for instructions.

If you are trying to translate part of Red, or a Cog, you would write a ``.po``
file.

A technical detail, is that Red actually uses a slightly modified version of
``pygettext``, called ``redgettext`` (https://github.com/Cog-Creators/redgettext).
This is mostly just an implementation detail though and does not affect Cog
authors or translators. If you are used to gettext, it is important to explain
Comment on lines +38 to +39
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does affect Cog authors in that our tool can extract command docstrings (which are part of cog's UI) while not extracting all the other docstrings (which generally aren't user-facing). This is what the redgettext usage example listed in this document does and is something that cannot be done with pygettext (which can only do no docstrings or all docstrings). The distinction probably isn't important for translators who just translate whatever .pot file is given to them but is definitely something that Cog authors would want to know.

More so, pygettext will sometime in the future supports more features than pygettext which is actually rather limited - it has no support for plurals or for context-dependent translations. Those features are already implemented in redgettext (see Cog-Creators/redgettext#9, Cog-Creators/redgettext#10, Cog-Creators/redgettext#11) and just haven't been released yet as I was hoping to have a working implementation of relevant runtime features in Red as well to make sure it aligns with what the support was implemented for.

While upstreaming those features to pygettext was mentioned as a possibility (https://discuss.python.org/t/modernize-and-add-missing-features-to-pygettext/26455), a year passed since so it's unclear that there's enough interest to get it there any time soon.

that Red does *not use `.mo` files* - it is not necessary to run ``msgfmt`` or
other tools, it reads the ``.po`` files directly.
Comment on lines +36 to +41
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this paragraph is needed, instead you can introduce redgettext when explaining how to generate .pot files as a helper provided to make generating them from a cog easier.


~~~~~~~~~~~~~~~
For Cog authors
~~~~~~~~~~~~~~~

For authors of Cogs, it's very easy to make your bot translatable using the
Translator class provided - code samples can be found below. In summary, the
steps are:

* Import the i18n Translator class and cog_i18n decorator.
* Decorate your class with @cog_i18n
* Translate literal strings (eg "This is a test command") to use the underscore
function like _("This a test command")
* Use redgettext to generate a message.pot file
* Write a test ``.po`` file, and put it in your ``locales/`` directory relative
to your ``.py`` code.
Comment on lines +51 to +57
Copy link
Member

@Flame442 Flame442 Apr 11, 2024 •
edited by Jackenmen
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Import the i18n Translator class and cog_i18n decorator.
* Decorate your class with @cog_i18n
* Translate literal strings (eg "This is a test command") to use the underscore
function like _("This a test command")
* Use redgettext to generate a message.pot file
* Write a test ``.po`` file, and put it in your ``locales/`` directory relative
to your ``.py`` code.
* Import the i18n `Translator` class and ``cog_i18n`` decorator.
* Instantiate an instance of `Translator` under the ``_`` variable.
* Decorate your class with `@cog_i18n(_) <cog_i18n>`.
* Wrap string literals with a function call using your `Translator` object. Example: ``_("This a test command")``.


.. code-block:: python

Expand All @@ -27,23 +72,39 @@ Basic Usage
"""command description"""
await ctx.send(_("This is a test command"))

--------
Tutorial
--------

Copy link
Member

@Flame442 Flame442 Apr 11, 2024 •
edited by Jackenmen
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.. note
If you need to interpolate values into parts of the string, you will need to use `str.format` **outside** of the function call.
Example:
.. code-block:: python
await ctx.send(_("Nice to meet you, {user}").format(user=ctx.author))
.. warning
You **cannot** use f-strings with translations. f-strings resolve before translations are able to run,
preventing the translation logic from matching the string with its translation.

After making your cog, generate a :code:`messages.pot` file

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point is when I think it needs a new section header for using redgettext.

After making your Cog, and using the _() function to make your strings
translatable, you will need to generate a :code:`messages.pot` file.
Comment on lines +77 to +78
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
After making your Cog, and using the _() function to make your strings
translatable, you will need to generate a :code:`messages.pot` file.
After updating your cog to make your strings translatable, you will need to
generate a :code:`messages.pot` file to be able to apply translations to your
strings.


We recommend using redgettext - a modified version of pygettext for Red.
You can install redgettext by running :code:`pip install redgettext` in a command prompt.
You can install redgettext by running :code:`pip install redgettext` in a
command prompt.

To generate the :code:`messages.pot` file, you will now need to run
:code:`python -m redgettext -c [path_to_cog]`
This file will contain all strings to be translated, including
docstrings.
(For advanced usage check :code:`python -m redgettext -h`)

You can now use a tool like `poedit
<https://poedit.net/>`_ to translate the strings in your messages.pot file.
~~~~~~~~~~~~~~~
For Translators
~~~~~~~~~~~~~~~
Comment on lines +90 to +92
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't love this header name, but I can't think of a better one. "Writing translations" doesn't seem specific enough, but anything else I think of is too wordy.


If your Cog is already made to be translatable by the original Cog author, you
simply need to write a ``.po`` file for your language.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than saying they need to create a .po file for their language, it should probably be explained that either an existing .po file should be edited or the .pot file can be cloned & renamed with the proper language code to be used as a base. If there is no .pot file, the translator will need to ask the CC to add translation support to their cog.


You can use a simple text editor to write a ``.po`` file, but you also use popular
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest adding a small code block example of what a .po file looks like, and how to edit it. Additionally, there should be a ..warning that anything within curly braces must be left as-is for the code to function correctly, as those are places where a value is inserted into the string.

#: redbot/core/_cog_manager.py:464
msgid "The following paths were removed: {paths}"
msgstr "Les chemins suivants ont été enlevés : {paths}"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest just linking to https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html but we actually don't support the majority of that spec yet (the missing features are implemented for redgettext but Red itself lacks support for those constructs) so maybe we will have to wait for that...

tools like `poedit
<https://poedit.net/>`_ to translate the strings in the messages.pot file. There
are also free online websites that can be used to get you started, like
https://crowdin.com/ which some developers link to their code repositories, or
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned before, Red core (and some third party cogs) do expect users to exclusively use crowdin, rather than PRing in changes directly to .po files. It might make sense to mention that here, or in the next paragraph.

https://www.potranslate.com/upload where you can just upload the ``.pot`` file.

Once you have written your ``.po`` file, you will need to send it back to the Cog
author. Most Cog authors would prefer pull requests on GitHub, but it's probably
best to open an issue with the original Cog author to discuss how they would
like to receive your ``.po`` file language contribution.
Comment on lines +106 to +107
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
best to open an issue with the original Cog author to discuss how they would
like to receive your ``.po`` file language contribution.
best to reach out to the Cog author to discuss how they would prefer to receive
your translation contribution.


-------------
API Reference
Expand Down
Loading