Data corruption when sending UTF-8 encoded LDIF values.

[felixvf]

The current version of ldapply ignores the charset when reading the .ldif file but then tries to send the data to the LDAP server, it encodes the data to the encoding of the current locale as if the data was encoded in ISO-8859-1.

This means:

1. If there are just ASCII codepoints and no non-ASCII UTF-8 codepoints, everything is fine. Hence, "foo" stays "foo".
2. If the current locale is "C" (e.g. export LANG=), UTF-8 non ASCII codepoints get silently replaced by empty strings. Hence, "Iñtërnâtiônàlizætiøn" is mutilated into "Itrntinliztin".
3. If the current locale is "en_US.UTF-8" (e.g. export LANG=en_US.UTF-8), UTF-8 non ASCII codepoints get silently re-encoded into UTF-8. Hence, "Iñtërnâtiônàlizætiøn" is mutilated into "Iñtërnâtiônàlizætiøn".
4. If the current locale is "en_US.ISO-8859-1" (e.g. export LANG=en_US.ISO-8859-1), UTF-8 non ASCII codepoints get transparently re-encoded (as if they were not re-encoded at all). Hence, "Iñtërnâtiônàlizætiøn" stays "Iñtërnâtiônàlizætiøn".

Thus, oddly enough, only when the locale is a non UTF-8 locale, UTF-8 handling gets done correctly (albeit accidentally). It is expected that the action of ldapply be independent of the locale, of it is were locale dependent, then the locale would influence how the local .ldif file would be interpreted, not how to encode for the remote server.