January 2017 release

We've switched to doing released in order to make sure folks are getting stable code. This release includes many changes (mostly bugfixes) from the past several months.

Here is an abbreviated commit log. If you are updating your local sync engine, note there are several migrations you may need to run.

4df11df [sync] TravisCI testing changes and version bump for OSS release
5619ef9 [developer] More logs display bugfixes
bcef868 [none] Update outdated comment
1b42c6c [admin] Add instructions for perf profiler and make easier to use
3e94a3f [balancer] Don't move Accounts that belong to debug sync instances
d08e69b [api, sync] Handle more IMAP auth errors. Provide better error messages.
aa9b82a [sync] Truncate long calendar names
de16267 [api] Fixes AttributeError where some messages categories are None
5cc00df [sync] Truncate traceback to prevent DataError cont.
dd7bad3 [sync] Truncate traceback to prevent DataError
bcfa1a5 [sync] Make sure the thread exists
4f4bc6f [sync] Don't process event during message reconciliation
0cbd15f [sync] Fixes T7297
c386528 [*] Fix event queue bug
825d9bc [api] Don't schedule draft deletion until message is sent
29a5ed4 [all] Don't include debug instances by default
6123e2a [sync] Handle JSON parsing errors from event queue data
5a38ac6 [sync] Use KillerGreenletTracer for SyncHTTPFrontend
124b57d [sync] Use PPAs instead of CPU history for process limits
9de67a4 [sync] Unpack the right number of values for event queue lpop
2f17ab1 [migrator] Make migrator send heartbeats
3005676 [sync, api, admin, migrator] Guard event listener against multiple commits
622a37f [sync] Use correct end time for all day events
b250782 [sync, api] Mark Accounts that already have post commit listeners
10243b5 [sync] Split shared and private event queue handlers
3b08cf4 [sync] Fix bugs in updated unschedule script
4b0b653 [balancer] Handle when account ID is missing
76f5b13 [sync] Update script to manually unload Accounts
4ed36b8 [balancer] Add stats logging for migration minimization
d58d651 [sync] Flush private queue when receiving an event
2a80cd4 [balancer] Use Hungarian algo to minimize migrations
4b0040f [balancer] Parameterize the timespan for load balancing
2191519 [none] Delete legacy scheduler code
0a9626d [sync] Release owned Accounts that no longer should be
2daba2d [sync] Turn noisy event queue timeout statement down to DEBUG
e850a9f [all] Make database transaction timing metrics optional
2a0719a [sync] Don't discard Account until it's marked as released in the DB
065eb63 [webhooks,sync,api,admin] Consolidate StatsD code, revise metrics names
b950d0a [sync] Notify when Account.sync_should_run changes
8dc90fa [api] [direct_fetching] Handle case where g_msgid is None.
498a9ff [sync] Comment out new statsd counters for now
2c69eb5 [sync] Don't send migrate_from if desired_sync_host is None
be41c73 [sync] Handle more folder roles edge cases
fe8f3db [sync, api] Ignore irrelevent Account updates when sending events
9f65c7b Fix another draft deletion edge-case
1dcf4f9 [api] Add custom provider rule for adaptiveinsights.com
0f323fb [sync] Use gauge for sync_hosts_count metric
82b20fb [sync] Fixup sync restart script
b8b1c27 [sync] Don't manually clear sync host
ed70ec8 [sync] Add function to get vcpu count when balancing fleet
205fc95 [sync] Fix some problems with finding Account loads
1bfe66b [sync] Reduce some code duplication
e67a9a5 [sync] Add script to dump per-Account loads in JSON
88d0184 [sync] Add scripts to dump and restore the assigned sync redis hash
531df0e [api, sync] Handle more errors during device provisioning.
ad6ac16 [sync] Add legacy-balance-fleet to setup.py
1e2aa19 [sync] Add legacy load balancer
8b1ae86 [sync][api] Don't double-save to our legacy S3 bucket anymore
aeae53a Log affected account id.
81e8407 [sync] Add script to restart sync for forgotten Accounts
5ed6047 [api] Direct fetching: handle non-existent UID replies.
d8bb3e4 [syncback] Delete messages in a safe way
36812e6 [migrator] Moar load balancer fixes
aea75f9 [migrator] More sync load balancer fixes
763fa83 [sync] Add script to balance sync fleet
226d322 [api] Try converting g_msgid to a number if we get a string.
3bd4764 [syncback] Merge adjacent SyncbackTasks for change_labels
2103eec [sync] Add service to schedule Account migrations across sync hosts
e7eba03 [sync] Deduplicate migration events
8e015e2 [sync] Add per-Account sync counter for alerts
b81bde9 [sync] Fixup string formatting in a couple scripts
ca24daa Display exception breakdown in direct fetching validation script.
d388df0 [sync] Add /build-metadata endpoint to SyncHTTPFrontend
df7d351 More fixes for account migration
c6c8a40 Fix Account migration stuff
7251aed [schema][sync, api] Add ability to migrate Accounts between sync hosts
b76ad14 Fix SyncHTTPFrontend after refactoring
ed580d6 Commit after undeleting a Thread
c7d1da6 [api] Bump nylas-production-python version
0926449 [sync] Only collect first message metric for inbox
c87a433 [sync] Include application_id in all requests
69df7aa [api] Add provider to api request logs
a4cc16f [api] Log more details on direct fetch failures
8d56dc6 [api] Raise the right exception when trying to fetch a deleted message.
89c78bd Skip null MIME parts.
7aa87f4 Add more graphite logs.
49576e0 [migration, sync] Update imapfolderinfo uidnext to be a bigint instead of int
15b4d17 [direct-fetching] [sync] [api] Phase 2: Stop reading from our permanent bucket
480c75e [sync] Fix truncation for TEXT columns
95cc875 Add the ability to interrupt blocked greenlets
3b0cdce Fix alembic migration script revisions
4ed8cfc More exception catching
0da4345 Log attempts on individual syncback actions
66152e2 Standardize logging actionlog.id as action_log_id
0b77782 Log account_id and num_connections as structured data in crispin pool creation
4d468ee Make the direct fetching code a bit more robust
e74efd2 Add profiler for SyncbackService
e220f21 Add test for correct Contact.uid collation
881548b Revert "Remove Block/Part ForeignKeys"
d7e2af3 Bump nylas-production-python version
ab7bb85 Bump nylas-production-python version
9e4100b Skip Messages that don't have a valid Thread
b8f8f81 [api, sync, schema] Make deleting Threads asynchronous
53fe179 [sync, api] Don't update remote events if the remote event creation failed
f5b5d82 [api] Update fastmail provider settings
25e7a6f [api] Prevent TypeError for recurring calendar events
723d84c Fix incorrect rebase of alembic migration script
9781733 [SCHEMA] Drop foreign keys to and from Contact table
e154aff Remove Block/Part ForeignKeys
fe86721 [Ask @khamidou before shipping] Fetch attachments and raw messages directly from the email provider
3a81f0f [api] Don't leak Exception messages when we don't know what they are!
7ff64a4 Don't send an event invite to the owner.
5b99c86 Set Part.content_id for inline attachments
1f55c83 [api] fix(logging): Add all auth scope logging to WSGI context immediately
727f9dd [api] fix(logging): Log namespace IDs before throwing 404s
e47e6e7 Add support for sending inline images
9a37aab [none] fix(logging): mysql “too many conn” is transient
ed63006 [none] Upgrade coverage dep for more features
9028025 [sync] fix(logging): Get default logger if none provided in callback
f308473 [none] fix(tests): Use a random port for SMTP server
e4aa649 fix(logging): Clip error messages saved to Account objects
2d78db4 fix(logging): Don’t log transient mysql errors
3461444 Standardize test database names
f388883 [sync] Bump IMAPClient to include folder_status fix
20f201f Provide detailed message to users when handling `OperationalError`'s
572b70b Roll back to older pytest
32e2b51 fix(tests): Unbreak local testing, oops
2f8b12e fix(tests): Skip test_block_on_depleted_pool()
7693683 Fix broken JSON in config-test.json
482203a Split test sync-engine shard config out into separate config files
c88471a Remove redwood DB config from sync engine test config
e3eb09b Disable webhooks tests during jenkins runs until they are fixed
af0881f Add iontrading as an exchange provider
3512e7d Prevent ICS organizer CN field from being set to blank
20e121d Bump nylas-production-python version
ab6d56a Don't allow DetachedInstanceError crashes in model repr()
c873997 Fix passing in of echo to engine()
b27868f Quiet noisy 'syncing events' log statement
c48cb33 Fix from address.
f5bf5a8 Fix typo which made code super slow.
5e13deb Turn down the production log volume more
d7828d5 Remove Transaction ForeignKeys
ea8d940 Give more insight about what deletion is doing 😅
485250a Fix dev secrets filename
113b95c Add ZONE in prod test config
4e9603e For prod too
236e968 Restore old SYNCBACK_ASSIGNMENTS values
c860120 Syncback assignments really do have id '0' even though it's not valid JSON
435f06a Add back some keys to the sync-engine test config
7efb16b globs aren't strings
19be6ef Deduplicate test configs
e9f80b7 Make 'syncing contacts' message DEBUG also
2f51947 Further reduce log volume
363f98d Remove extraneous conftest.py
87d8359 Install pytest.ini into package
49c4371 Pass 127.0.0.1 as the hostname to mysql when dropping/creating test tables
f9d9b62 sync-engine system tests depend on unspecified nylas API client library
318e52e s/localhost/127.0.0.1/ on DATABASE_USERS also
7661f81 Use 127.0.0.1 instead of localhost in test mysql configs
1c02fee Give all test_util.py a different name
e24c1d8 Fix SMTP SSL test
cb1d356 Fix more tests
ce226d9 Fix more tests in install
da74be1 Ship JSON files too
511fb55 Fix a bunch of tests
c6901b1 Fix up MANIFEST.in to ship test data
3abcc69 Fix Exchange tests to work in installed form
c22eb3a Combine inbox.test dirs in installation
127e617 Make tests work when package is installed too
34276c6 Install test configs for sync-engine too
5928e13 Add tox to sync-engine deps
01fa0ac Distribute tests with source code

v0.3.0 Release, Calendars, and new SDKs

Hi Inbox Developers,

Over the past month we've been hard at work at Inbox HQ as well as around the globe to improve the Inbox Sync Engine, API, and SDKs, and are proud to announce the release of version 0.3.0. You can find all the gory details on GitHub, but the high-level changes are noted below.

API changes

While we strive to maintain API stability, we've been working with folks building apps on top of Inbox and have found that the following API changes make some apps drastically simpler—enough to justify a backwards-incompatible change.

• We've changed a number of endpoints to refer to items by item_id or item_ids rather than just item for greater clarity. This includes:
  • Rename the namespace attribute of all objects to namespace_id.
  • Rename the thread attribute of messages and drafts to thread_id.
  • Rename the messages and drafts attributes of threads to message_ids and draft_ids, respectively.
  • Rename the thread query parameter for filtering messages and drafts to thread_id.
• The Draft update endpoint now uses the PUT HTTP method, consistent with other objects like threads and messages.
• Drafts now have a version attribute, and updating a draft now simply changes its version, instead of creating a new draft object. (Note that updating or sending a draft now requires the version value as a parameter, in order to prevent write conflicts.)

New Calendar and Events APIs

These new APIs allow access to calendar and event information for accounts from Google, iCloud, Hotmail/Outlook.com and Microsoft Exchange (contact [email protected] to request beta hosted access). Check out the docs to get started!

New SDKS

We've released Ruby and Python bindings, making it even faster to get started building email apps against the hosted Inbox API.

Here's a quick Python example:

inbox = APIClient(APP_ID, APP_SECRET, token)

namespace = inbox.namespaces.first()
for thread in namespace.threads.items():
    print thread.subject

Other feature additions

• Attachment tag added to threads which have attachments.
• Create an archive folder where necessary for various providers.
• Namespace endpoint now support filtering options such as limit, offset and email_address.
• Files endpoint supports delete for files that are not currently attachments.
• User-created tags can be deleted.

Bug Fixes

• Propagate unread status changes to threads in update_metadata.
• Better handling of HTML stripping for snippets.
• Remove unused UIDNEXT from IMAP mailsync, since some providers don't support it.
• More graceful handling of various IMAP errors.
• Improved coverage for unit tests, notably over the API calls.
• Refactor message / parts / blocks relationship to avoid creating new blocks when they are attached to messages.
• Remove duplicated code around SMTP mail sending.
• Truncate fields if they are too long on calendar/event parsing.
• Properly check to, cc, bcc fields in drafts for formatting.

With love from Team Inbox,
-- mg, spang, kavya, eben, ben, charles, karim, dwayne, and zoë

2014.08.12 Release

Hi Inbox Developers,

We are pleased to announce a new release of Inbox. We are again excited to see the enthusiasm from the community and we've been hard at work integrating change requests, adding new features and fixing bugs.

Here's what's changed since the last release:

New Features

• Support for generic providers. We've now made it easier to add support for more providers through the "providers.py" file, where you may specify the relevant information for new providers without modifying other parts of the code.
• Basic multicore support for mailsync.
• Automatic detection for providers which support RFC4551 / 7162 (CONDSTORE).
• An improved threading algorithm for providers which do not support automatic threading on the backend.
• Theinbox-auth command is now scriptable with the options --exit_after_link and --token.
• Better handling for invalid API query parameters.
• The API message representation now includes detailed attachment information such as content_type, size, and filename.
• Initial support for Outlook.com/Hotmail including sending, receiving, archiving and draft management
• The drafts API supports ?thread=[filter], which returns all the drafts of a thread.

Bug Fixes

• Improved handling for accounts which no longer have valid credentials or cannot connect to the underlying provider.
• Updated html2text version to avoid issues with malformed html.
• A significantly reduced number of database connections are now needed to sync accounts.
• Cleanly handle corrupted Gmail metadata cache files.
• SIGTERM is properly handled in inbox-start/
• The file attachment API call now returns a proper JSON response.
• Assorted drafts/sending bugs.
• Many performance improvements and other various bugfixes.
• Improved provider identification during authorization process.

Upgrading

If you're running Inbox on your own infrastructure, make sure you've disabled the sync, API and syncback processes and run the following to upgrade:

$ ./setup.sh

Thanks to everyone for their continued support and interest in Inbox! If you run into problems with the latest, we hope you'll file an issue[1] or hop onto ##inbox to get help.

The following folks contributed to this release (GitHub usernames by # of commits):
emfree, charlesg3, kav-ya, spang, khamidou, grinich, bengotow, mmattioli, rmasters

[1] https://github.com/inboxapp/inbox/issues