Skip to content

Internationalization: Coding

Jump to bottom Edit New page
Cyril Pauya edited this page Jan 23, 2015 · 19 revisions

Introduction

Internationalization is the process of designing a software application so that it can potentially be adapted to various languages and regions without engineering changes.

-- Wikipedia

KA Lite is a cross-cultural tool and internationalization of the app by translating strings is a key component to making this content accessible anywhere.

We use an automated, Django-based tool for finding and extracting translatable strings. This means that devs, when we are coding, are responsible to mark up the code in standard ways, in order to allow this program to find the translatable strings.

Below, we've linked to tutorials for how to use Django's built-in internationalization features when writing code for KA Lite.

How to wrap strings in Django:

  1. Django templates

  2. Python code

  3. Javascript. We import the JavaScript translation dictionary into our base template, so no need to worry about importing. In your js code, simply wrap strings in a gettext function like the following example: document.write(gettext('this is to be translated'));

  4. Handlebars templates must use the {{_ "hello world" }} helper function. Use literal strings because variables cannot be picked-up by the makemessages management command for i18n.

i18n on OSX

  1. locale.getlocale() problems on OSX

KA-Lite Internationalization Kitchen:

How to manually check translations.

References:

Requirements:

  • Central must be running at http://127.0.0.1:8000/ based on master branch

    • local_settings.py must have these entries:

       DEBUG = True
 CROWDIN_PROJECT_ID = "ka-lite"
 CROWDIN_PROJECT_KEY = "<project-key-here>"

    • an alternative is to get a copy of the kalite-en.zip and save it to /_crowdin_cache/kalite-en.zip

    • follow the doc on local_settings.py setup

    • the *CROWDIN_PROJECT_* keys are private; you'll have to get in touch with a (FLE) team member who has them.

  • Distributed must be running at http://127.0.0.1:8008/ based on develop branch and must be pointing to central server running locally

Everything you ever wanted to know about KA Lite internationalization

In the sections below, you will find information on how to create po files, which are translation files that allow swapping one language with another. These files can be uploaded to the web, translated by volunteers, and then incorporated back into the KA Lite codebase. We currently host our translation project here: http://crowdin.net/project/ka-lite.

Testing Language Pack Support

Language pack support requires an interaction between the central server (which creates and caches the language packs) and the distributed server (which downloads them from the central server and installs them).

In order to test language pack support, you will need to install both a test central server and test distributed server (see the TESTBED.md documentation for setting them up). After setting up each, the following steps are needed:

Central server:

  1. Run the update_language_packs command. Specify the -l option to update a single language (most efficient). Good test languages include:
    • ./manage.py update_language_packs -l zh-CN will create a Chinese language pack with many subtitles and some interface translations, and will test a fuller range of characters.
    • ./manage.py update_language_packs -l aa will create an Afar language pack with 8 subtitles and no interface translations.
    • ./manage.py update_language_packs -l ur-PK will create an Urdu language pack with no subtitles and some interface translations.

i18nize_templates management command

What does this management command do?

  1. It will i18nize handlebars and html in django app specified
  2. i18nize a specific file type.
  3. i18nize a specific file.

Sample Usage:

  1. python manage.py i18nize_templates coachreports
  2. python manage.py i18nize_templates coachreports --parse templates
  3. python manage.py i18nize_templates coachreports --parse hbtemplates
  4. python manage.py i18nize_templates coachreports --parse-file facility-select.handlebars

Distributed server

  1. Run the languagepackdownload command. Use the -l option to specify a language that you set up on your central server above.
  2. Try downloading through the update_languages interface, accessible via the "Update" link, then clicking the "Install more languages" option.

Wrapping Strings

Hot links: how to wrap strings in Django templates and views

How to wrap strings in Django:

  1. templates

  2. Python code (views, models, etc)

  3. Javascript. We import the JavaScript translation dictionary into our base template, so no need to worry about importing. In your js code, simple wrap strings in a gettext function like the following example: document.write(gettext('this is to be translated'));

Creating and Testing Interface Translations

Note: many questions about i18n with Django can be answered via the Django i18n Docs. Also, check out the i18n management commands for shortcuts on following the steps below.

  1. First, make sure you have the GNU gettext toolset installed. If you have any trouble, try Googling "How to install GNU gettext toolset for [insert-operating-system-here]"

  2. Next, navigate to the project root directory (e.g. [local-path-to-ka-lite]/ka-lite/) note: ka-lite is the default if you don't specify a directory when you clone

  3. Run the makemessages command to generate po files for the languages you want. Read the docs linked above for more info on how this process works. You'll want to ignore the bundled python packages, because they've already been translated and are big. It's also a good idea to add the --no-obsolete flag to remove outdated strings. Example command: python kalite/manage.py makemessages -l en --ignore=*/python-packages/* --no-obsolete to generate po files for English.

    • If you follow the example, you should see processing language en. This will create a django.po file in the locale/en/LC_MESSAGES directory.
    • get an error?: check out the common error messages and their solutions.

  4. Run the makemessages command again to generate the po file for javascript and handlebars. It's a good idea to ignore any admin, the project's python-packages, and i18n static files, as django or the python package has taken care of the i18n already. Example: python kalite/manage.py makemessages -d djangojs -l en --ignore="*/kalite/static/*" --ignore="*/python-packages/*" --ignore="*/js/i18n/*.js" --no-obsolete.

    • If you follow the example, you should see processing language en. This will create a djangojs.po file in the locale/en/LC_MESSAGES directory.
    • get an error?: check out the common error messages and their solutions.

  5. Inspect the two files you have generated. You should see a bunch of lines like:

    #: kalite/central/views.py:85

    msgid "Account administration"

    msgstr ""

    Explanation: each msgid string is a string in the KA Lite codebase. Each msgstr is where the translation for this language goes. Since this is an English po file and KA Lite is in English, no translation is necessary, but for testing, pick a string to translate into something else.

  6. Find msgid "Admin" and translate it to something fun: e.g. msgstr "What does the fox say?"

  7. Now that we have updated our translations, we need to compile the po files into a mo file so that it can be rendered by Django. To do that, we use the compilemessages command. Example: python kalite/manage.py compilemessages -l en.

    • You should see:
      • processing file django.po in /Users/dylan/Hacking/leq-fork/ka-lite/locale/en/LC_MESSAGES
      • processing file djangojs.po in /Users/dylan/Hacking/leq-fork/ka-lite/locale/en/LC_MESSAGES
    • This command compiles each po file in the en language directory and if you've been following along, should have created two new files: django.mo and djangojs.mo.

  8. Now, restart your local server (python manage.py runserver) and check out your translations! Note: Not seeing your translations? It could be a caching problem! Try opening in a different browser, clearing your cache, or turning caching off.

Common Error Messages

  • Error: This script should be run from the Django SVN tree or your project or app tree. If you did indeed run it from the SVN checkout or your project or application, maybe you are just missing the conf/locale (in the django tree) or locale (for project and application) directory? It is not created automatically, you have to create it by hand if you want to enable i18n for your project or application.

    • Solution: You need to create an empty locale directory in the project root path-to-kalite/ka-lite/locale/. After creating, try running makemessages again.

  • python: can't open file 'manage.py': [Errno 2] No such file or directory.

    • Solution: ensure that when you are running manage.py commands from the project root, you specify where to find manage.py, e.g. python kalite/manage.py [command]

Finding unwrapped strings and wrapping them

This should be done periodically. As the codebase changes, it is likely that user-facing strings will be added, and these will need to be wrapped in the appropriate translation tags so that they can appear in our po template files and be translated.

  1. The easiest way to identify unwrapped strings in the codebase is to translate the interface into something wacky like asterisks, and see what's left in plain english. Then you just find that spot in the code, wrap it in translation tags, and move on with your day! Therefore, the first step in this process is following steps 1-5 in the above section Creating and Testing Interface Translations

  2. Once you have fresh new po files, you basically just replace step 6 above with the following: translate every single string into something obviously non-English. For now you could just use a text editor to select all the msgstrs and fill them in with '*****'.

  3. Once some msgstrs have been filled in, follow steps 7-8 in the above section Creating and Testing Interface Translations to compile the translations and view them.

  4. Walk through all of the views in the interface. When you see something in English, you know that string needs to be wrapped. Follow the instructions in the Django i18n Docs to wrap the offending string in the correct translation tags. You can use examples in other parts of the codebase as well for help.

    Important: When running compilemessages you will probably run into the error: a format specification for argument 'variable_name' doesn't exist in 'msgstr'. Check out the line it specifies in the file. You'll probably see something like '%(variable_name)s' in msgid. This is how variables appear. You simply need to copy and paste this variable name (e.g. '%(variable_name)s') into the asterisk filled msgstr. There shouldn't be too many, so doing it manually isn't a killer. E.g. msgstr "***** '%(facility_name)s'". This is just how Django ensures that variables get carried over through the translation process.

  5. Once you think you've found everything, repeat the steps in this section. See if you find anything else. Rinse and repeat!

  6. Once you're done testing, be sure to delete the po and mo files you have created. Translations are not overwritten by creating running the makemessages command, so you will want clear the tests you created so that people using KA Lite in English won't just see a bunch of asterisks :).

How to use i18n Management Commands

Django management command are basically tools you can use from the command line to automate processes. We use them in our code for lots of things: handling video and subtitle downloads for instance. You've used management commands already if you've followed along in the previous sections. makemessages and compilemessages are built-in Django commands for i18n. You can also write custom management commands. For i18n we have two so far.

Command: update_pot
  • This command is useful for automating some of the processes described below, including generating po files and compiling translations. You can run python manage.py update_pot -h to see available options.
Command: generate_dubbed_video_mappings
  • This command creates a dictionary that maps english videos to dubbed video ids, using the manually curated online Google Docs spreadsheet provided by Khan Academy's i18n team.
Command: cache_translations
  • This command pulls down the latest translations from CrowdIn, obtains meta-data on them, and zips everything up into a language pack that can be downloaded by KA Lite installations around the world.

i18n resources

Here are a few commonly requested resources, for judging overall KA Lite support for different languages.

Add a custom footer
Add a custom sidebar
Clone this wiki locally