Apparently `lxml.html.document_fromstring` (and possibly other
`lxml.html` loaders) parses byte-strings as latin1 regardless of their
actual encoding, maybe because python2, maybe because there's a super
legacy html4 parser underlying it.

Either way that means ever since loading
`static/description/index.html` files was added 10 years
ago (4bf6a7e) `_get_desc` has been
loading these files in latin1 rather than the utf8 most people would
expect.

Add an explicit decoding phase to try and load html description files
in UTF8. Fall back to latin1 in case there are description files which
are genuinely in latin1, or even just some random-ass broken stuff
which very much isn't utf8 (the extended-ascii encodings -- of which
latin1 is one -- will happily accept and mangle any input as every
byte value is valid, utf8 is a lot more structured).

Closes #127846

closes odoo/odoo#133708

Signed-off-by: Xavier Morel (xmo) <[email protected]>

I missed a critical issue in #133708: various users had discovered
they could already fix description issues by adding an XML declaration
to their document which is very cool (though technically not really
valid).

What is a lot less cool is that lxml gets *extremely* unhappy when
asked to parse *strings* with an encoding declaration, raising a
ValueError, so the purported fix breaks on any module which does that,
which seems to include a lot of OCA modules.

Gate the encoding guessing by bailing if the document has an XML
declaration, in which case we just assume the author knows what
they're doing and we leave them alone. For extra safety, check the
encoding declaration in ascii and utf16. Could also have checked for
BOMs, but lxml seems to not care about them overly much (in fact it
seems to prefer them decoded which is odd).

closes odoo/odoo#133900

Reported-by: @rezak400
Signed-off-by: Xavier Morel (xmo) <[email protected]>

Upgrade (aka migration) scripts are a core part of Odoo, allowing
database manipulations for modules during version changes.

Any module, including custom ones can run upgrade scripts, even if the
`--upgrade-path` flag (and with it, the `odoo.upgrade` sub-module) is
not present. Currently only the "standard" modules benefit of easy
upgrade script testing. Any custom modules that want to run tests of
their upgrades have to import the tests in the usual `tests` folder,
which is not ideal.

Therefore, to allow TDD and programmatic testing of upgrade scripts in
custom modules, the test discovery is here modified to also parse the
module's `migrations` and `upgrades` sub-modules for tests.

closes odoo/odoo#122569

Signed-off-by: Christophe Simonis (chs) <[email protected]>

…3cc20b95e

Solved conflicts on:
- odoo/modules/module.py