Failed subscribe requests spin forever

[jwhite66]

the mc.subscribe python utility will throw an exception on error. That will force the subscribe request to re-queue. That's nice, as sometimes Mailchimp is down, and we retry.

However, for hard failures (e.g. bogus email), it essentially means we go into a loop, trying to subscribe over, and over again.

We could modify the code to catch the set of exceptions that are permanent. Simpler might be to figure out a way to queue the subscribe with a max # of retries.

[jwhite66]

More details: in backend/env.py, we have code to subscribe users to our mailing list.

We add a job to _subscribe_to_mailchimp to our deferred queue. In that code, the call to mc.subscribe is not bracketed with a try/catch. The mc.subscribe code throws exceptions for known error conditions. The way the GAE code works, if a deferred job throws an exception, the job gets retried later. That leads to the current situation where we have hundred of subscribes trying, and failing, over, and over.

Note that just bracketing the mc.subscribe call would 'cure' this, but at least at one point in time, the mc service was temporarily down, and that fail safe came in awfully handy.

So, to properly 'fix' this, we'd want to selectively try/catch, and discard known failures (e.g. email invalid, or email is believed to be a spammer), but propagate any unknown exceptions. Alternately, just having the retry logic fire only a few times would also prune things.

[jwhite66]

Sample of a typical error:
2014-09-02 22:12:48.148
An email address must contain a single @
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in call
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in call
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in call
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(_args, *_kwargs)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 309, in post
self.run_from_request()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 304, in run_from_request
run(self.request.body)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 146, in run
return func(_args, *_kwds)
File "/base/data/home/apps/smayday-pac/1.378270988232832417/env.py", line 209, in _subscribe_to_mailchimp
send_welcome=False)
File "/base/data/home/apps/smayday-pac/1.378270988232832417/mailchimp/mailchimp.py", line 2213, in subscribe
return self.master.call('lists/subscribe', _params)
File "/base/data/home/apps/s~mayday-pac/1.378270988232832417/mailchimp/mailchimp.py", line 349, in call
raise self.cast_error(result)
ValidationError: An email address must contain a single @

[jwhite66]

And another one: (actual email replaced with dashes)

2014-09-02 22:01:29.397
"[email protected]" has been banned
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in call
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in call
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in call
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(_args, *_kwargs)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 309, in post
self.run_from_request()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 304, in run_from_request
run(self.request.body)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 146, in run
return func(_args, *_kwds)
File "/base/data/home/apps/smayday-pac/1.378270988232832417/env.py", line 209, in _subscribe_to_mailchimp
send_welcome=False)
File "/base/data/home/apps/smayday-pac/1.378270988232832417/mailchimp/mailchimp.py", line 2213, in subscribe
return self.master.call('lists/subscribe', _params)
File "/base/data/home/apps/s~mayday-pac/1.378270988232832417/mailchimp/mailchimp.py", line 349, in call
raise self.cast_error(result)
ListInvalidImportError: "[email protected]" has been banned

[davidxia]

@jwhite66 How do you run tests? I tried PYTHONPATH=.:build ./testrunner.py and got the error below. I haven't really dug into it though in case you know off the top of your head.

Traceback (most recent call last):
  File "/Users/david/src/pledgeservice/unittests/test_e2e.py", line 56, in setUp
    super(PledgeTest, self).setUp()
  File "/Users/david/src/pledgeservice/unittests/test_e2e.py", line 43, in setUp
    from main import HANDLERS  # main import must come after other init
  File "/Users/david/src/pledgeservice/build/main.py", line 222, in <module>
    config=dict(env=env.get_env()))
  File "/Users/david/src/pledgeservice/build/env.py", line 20, in get_env
    j = json.load(open('config.json'))
IOError: [Errno 2] No such file or directory: 'config.json'

[jwhite66]

@brad is the expert; my recollection is that you had to run from the build directory, and I have the feeling that you have to copy a file as well (perhaps config.json), although I don't recall which file or from where. Hmm. Arguably the most useless comment ever :-/.

[davidxia]

yea some instructions on how to run tests in the README would be great.

[aaronlifshin]

David, have you run a build command first?

A
On Sep 3, 2014 7:59 AM, "David Xia" [email protected] wrote:

yea some instructions on how to run tests in the README would be great.

—
Reply to this email directly or view it on GitHub
#114 (comment)
.

[davidxia]

@aaronlifshin I don't think so. What's the command?

[aaronlifshin]

Try

Grunt local

in the root of the repo, followed by python testrunner.py also in the root.

This project has a build step which creates a local, development or
production version in the build directory. Config.Json is one of the files
created.

A
On Sep 3, 2014 8:05 AM, "David Xia" [email protected] wrote:

@aaronlifshin https://github.com/aaronlifshin I don't think so. What's
the command?

—
Reply to this email directly or view it on GitHub
#114 (comment)
.

[davidxia]

@aaronlifshin I did grunt local which created the build directory with config.json inside. Then I ran python testrunner.py in the root directory, but failed because it couldn't find config.json in the root directory. Am I missing something?

[brad]

@davidxia you then must copy config.json to the backend folder

[davidxia]

@brad thanks that worked.

[davidxia]

I've updated the README to include this step. #130