Pyramid i18n issues and documentation improvement suggestions

[raspi]

Hi,

I just started using Pyramid few weeks ago and it also took that time to set up i18n with gettext. My setup: Python 3.5, chameleon templates and gettext. Ubuntu and Arch as dev environment and PyCharm as IDE. I added translation to templates first and code later. Here's some struggles I had.

Issues with setup.py extract_messages

I added

message_extractors = { 'myapp': [
    ('**.py', 'python', None ),
    ('**.pt', 'chameleon', None ),
    ('static/**', 'ignore', None),
    ]}

To setup.py as was stated in manual. Running $VENV/bin/setup.py extract_messages couldn't find chameleon extractor. I tried xml, lingua_chameleon, lingua_xml and lingua.xml, lingua.chameleon, etc. All sorts of combinations. I couldn't find a way to list those extractors. I also tried setup.cfg approach.

So first suggestion: don't give extractor not found as error. Either list the possible extractors or give instructions how to list them.

Later I found that according to this message in SO the extraction method has been changed somewhere in or before 2015.

I abandoned setup.py extract_messages and moved to $VENV/bin/pot-create.

Issues with $VENV/bin/pot-create

I ran $VENV/bin/pot-create --output myapp/locale/default.pot myapp.
Error: Nothing found. Hmm. And the .pot file is completely empty. Next try:
strace -e trace=open $VENV/bin/pot-create --output myapp/locale/default.pot myapp
Ok, it is scanning .py and .pt files from right locations.

First suggestion: Please list scanned files while pot-create is running or add --verbose or other parameter to give this information. I got the impression that it was doing nothing.

I found that the issue was missing i18n:domain="myapp" attribute in the templates. After that strings started to appear to the default.pot.

I created my first translation and I could change to it through pyramid.default_locale_name in the .ini file. Finally some progress.

Issues with changing language from URL

Next thing I wanted to change language based on URL. For example /fi/home, /en/home, etc. And without adding {language} parameter to add_route(). This route pattern is very common in many web frameworks and there's example(s) how to implement it and it's usually documented in URL routing and/or i18n section. I think it should be added to i18n documentation. I couldn't figure out even where to start how to implement it so I went to IRC for help.

I got help from Mikko Ohtamaa and Michael Merickel and here's the current implementation:
Include file:

"""
Enable language in URL
"""

from pyramid.request import Request

import logging

log = logging.getLogger(__name__)


def add_localized_route(config, name, pattern, factory=None, pregenerator=None, **kw):
    """Create path language aware routing paths.
    Each route will have /{lang}/ prefix added to them.
    Optionally, if default language is set, we'll create redirect from an URL without language path component to the URL with the language path component.
    """
    orig_factory = factory

    def wrapper_factory(request: Request):
        lang = request.matchdict['lang']
        # determine if this is a supported lang and convert it to a locale,
        # likely defaulting to your default language if the requested one is
        # not supported by your app
        request.path_lang = lang
        request.locale_name = lang

        if orig_factory:
            return orig_factory(request)

    orig_pregenerator = pregenerator

    def wrapper_pregenerator(request: Request, elements, kw):
        if 'lang' not in kw:
            # not quite right but figure out how to convert request._LOCALE_ back into a language url
            kw['lang'] = request.locale_name
        if orig_pregenerator:
            return orig_pregenerator(elements, kw)
        return elements, kw

    if pattern.startswith('/'):
        new_pattern = pattern[1:]
    else:
        new_pattern = pattern

    new_pattern = '/{lang}/' + new_pattern

    # Language-aware URL routed
    config.add_route(name, new_pattern, factory=wrapper_factory, pregenerator=wrapper_pregenerator, **kw)


def includeme(config):
    """
    Load
    """
    config.add_directive('add_localized_route', add_localized_route)
    log.debug("included %s", __name__)

This wouldn't work with 'add_route' (changing 'add_localized_route' -> 'add_route') so existing routes had to be changed to:

    config.add_localized_route('home', '/')
    config.add_localized_route('login', '/login')
    config.add_localized_route('logout', '/logout')
    ...

Going to / would give error so I had to add some 404 logic:

@notfound_view_config(renderer='templates/404.pt')
def notfound(request: Request):
    if request.path == "/" or request.path.count('/') == 1:
        """
        Redirect to default language
        """
        import pyramid.httpexceptions as exc
        from pyramid.threadlocal import get_current_registry
        # You could also read the browser's accept-language headers here
        deflang = get_current_registry().settings['pyramid.default_locale_name']
        redirect = "/" + deflang + "/"
        return exc.HTTPFound(location=redirect)

    request.response.status = 404
    return {}

Custom layout would die because it lost the matchdict['lang'] var generated by add_localized_route() this fixed it in the __init__.py before main():

@subscriber(NewRequest)
def ReqLanguage(event: NewRequest):
    """
    Read language code from URL and set it to request object
    """
    request = event.request.path
    if request.count("/") >= 2 and len(request) >= 4:
        event.request.locale_name = request[1:].split("/", 1)[0]

Issues with gettext domains and paths

Now I had translations changing with URL correctly and templates with stuff like
<a class="navbar-brand" href="/" tal:attributes="href request.route_path('home')" i18n:translate="">Home</a> worked fine and language was added automatically without touching anything.

Now I started to implement login and there were the first translatable strings (error messages) in the code. I don't remember all the details but it was either pot-create didn't find the translations from the .py files or those translations weren't transferred to .mo file during compile. The problem was fixed with just changing the .po and .mo filename to messages.(po|mo), because I found from the internal code that messages was the default translation domain.

I couldn't find any .ini or config. parameters to change default domain. I think this should be configurable.

I also found that if .pt had just i18n:domain="" pot-create couldn't find the translations, but as long as the i18n:domain was just something the translation were found.

The <language>/LC_MESSAGES/ directory seems to be also hard-coded. What other frameworks I've used I had no problems just using myapp/locale/<language>.(po|mo).

I hope this helps to improve i18n docs and future code/modules and making i18n more dynamically configurable.

Some issues may not be directly Pyramid related because I don't yet know what's belonging to what :)

[ztane]

the extract_messages stuff comes from Babel. Lingua was started by @wichert to overcome problems in Babel. Now Babel is being maintained by a fellow Finn and I guess already has feature parity with Babel Lingua.

At first, lingua plugins, like the one you use for chameleon, worked as Babel extractors too, but Wichert changed that at some point, thus that's why the babel approach does not work for you.

FWIW I am actually using Babel only, I ditched pot-create and alike, because of the problems therein, and am using that extract_messages approach. But then, I am not using any of the newer lingua extractors either.

[wichert]

@ztane Out of curiosity, what are the "problems therein" that made you switch ?

[ztane]

@wichert I guess one was that I needed to use babel extractors anyway for JS back then; then there were some other problems, like the extractors didn't work quite like I wanted; for example pgettext() with variable names instead of strings cause the python extractor to end with error; some bugs I did report back then but been too busy to submit PRs as Babel did work for me back then again.

[ztane]

@raspi and you should report those bugs that are strictly lingua to lingua bug tracker