Urgent: Error importing email

[ptav]

Hi, I started receiving a ValueError a few weeks ago after upgrading django-mailbox to 4.8.2 (django is 2.2.27). the DB is postgres which prevents string from been created from a NUL character. a empty string should be passed in this case.

ValueError: A string literal cannot contain NUL (0x00) characters.

Here is the full trace and I can also provide examples of the messages that are causing this error. It is a recurring problem that is causing considerable issues to our webapp so would appreciate if this could be looked at urgently - many thanks!

Traceback (most recent call last):
File "manage.py", line 16, in
execute_from_command_line(sys.argv)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/core/management/init.py", line 381, in execute_from_command_line
utility.execute()
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/core/management/init.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
output = self.handle(*args, **options)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django_mailbox/management/commands/getmail.py", line 25, in handle
for message in messages:
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django_mailbox/models.py", line 424, in get_new_mail
msg = self.process_incoming_message(message)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django_mailbox/models.py", line 240, in process_incoming_message
msg = self._process_message(message)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django_mailbox/models.py", line 378, in _process_message
msg.save()
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
force_update=force_update, update_fields=update_fields)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
force_update, using, update_fields,
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
using=using, raw=raw)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
return query.get_compiler(using=using).execute_sql(return_id)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
cursor.execute(sql, params)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/crr/.virtualenvs/forum/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
return self.cursor.execute(sql, params)
ValueError: A string literal cannot contain NUL (0x00) characters.

[Pietro395]

Can you provide an email example so we can investigate this issue?

[ptav]

Hi Pietro, will come back when I've come across the error again. Have been able to fix it by removing the offending email from the import stack.

Coincidentally I am now getting a similar situation with a different error message: LookupError: unknown encoding: iso-8859-8-i.

Both errors may be related as both happen only with meeting invite emails sent from countries with non-English native languages. May also be that it is the same error but the error message has been made more relevant recently.

Will come back when I've come across the error again, thank you!

[ptav]

Have now saved one of the emails that causes the error. As it contains personal email details would prefer to send it privately only - let me know how best to do that or, alternatively, what information you need

[Pietro395]

Have now saved one of the emails that causes the error. As it contains personal email details would prefer to send it privately only - let me know how best to do that or, alternatively, what information you need

Hi, can you remove personal information from the exported email? You can open it with a file editor

[ptav]

With 4.10.1 the error message is now more useful and probably leads to the correct source of the problem. Here is the trace. the error happens everytime we receive an email with this encoding standard (iso-8859-8-i)

Having done some research seems that simply mapping this encoding to iso-8859-8 is appropriate in most situations.

Thank you!

Traceback (most recent call last):
  File "/home/crr/sites/forum/manage.py", line 16, in <module>
    execute_from_command_line(sys.argv)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/management/commands/getmail.py", line 11, in handle
    Mailbox.get_new_mail_all_mailboxes(args)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/models.py", line 499, in get_new_mail_all_mailboxes
    for message in messages:
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/models.py", line 475, in get_new_mail
    msg = self.process_incoming_message(message)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/models.py", line 278, in process_incoming_message
    msg = self._process_message(message)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/models.py", line 424, in _process_message
    msg.to_header = utils.convert_header_to_unicode(message['to'])
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/utils.py", line 85, in convert_header_to_unicode
    [
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/utils.py", line 87, in <listcomp>
    _decode(bytestr, encoding)
  File "/home/crr/.virtualenvs/forum/lib/python3.9/site-packages/django_mailbox/utils.py", line 81, in _decode
    return value.decode(encoding, 'replace')
LookupError: unknown encoding: iso-8859-8-i

[ptav]

Further update. Having added a couple of lines to the offending code django-mailbox now processes the message but generates a warning further along: WARNING:django_mailbox.models:Unknown encoding iso-8859-8-i; interpreting as ASCII!

The new lines were inserted into django_mailbox/utils.py line 81:

    elif encoding == 'iso-8859-8-i':
        encoding = 'iso-8859-8'

Then inserted the following to django-mailbox/models.py line 377 in an attempt to better handle the condition:

    elif content_charset == 'iso-8859-8-i':
        content_charset = 'iso-8859-8'
        logger.warning(
            "Unhandled encoding %s; interpreting as iso-8859-8",
            content_charset
        )

However have not yet been able to fully test this as we only receive offending emails very occasionally and always delete those already processed.

Created a pull request (#298), awaiting testing and approval.