tipuesearch_content.js

var tipuesearch = {"pages":[{"url":"posts/2019/06/connected-content-and-the-slashdot-effect-or-how-i-learned-to-scale-apis/","text":"One of the coolest ways our customers integrate with our product is through a feature we call Connected Content. By utilizing custom HTTP endpoints—either owned by the customer or via one of their partners—our customers can inject customized, on-demand content into messages right before they're sent. Customers use this for a huge variety of use cases, such as personalized recommendations, weather information, and automated language translations. One of the common hiccups, however, is that these endpoints must be able to absorb huge spikes in traffic in order for Braze to send messages quickly. In the early 2000s, the phenomenon of a web service being unavailable due to large traffic spikes was known as the Slashdot Effect. Sites linked by popular articles on Slashdot would see huge amounts of traffic very quickly and almost immediately become unusable—this was just how it was before the days of cheap, easy-to-deploy auto-scaling and load-balancing services. Even today, this problem persists, and with regards to Connected Content, we've come to see a similar \"Braze Effect,\" where endpoints originally designed for lower-volume, consistent traffic are hit with a large traffic spike to inject Connected Content into messages due to Braze's sending speed. Building something capable of absorbing massive but infrequent traffic spikes can be tricky. All kinds of strategies can be taken around application server tuning, tactical database choices, and judicious load testing in order to build something to fit that profile, even at a relatively low cost. Check out the rest of this blog post at Building Braze !","tags":"Software Development","loc":"posts/2019/06/connected-content-and-the-slashdot-effect-or-how-i-learned-to-scale-apis/","title":"Connected Content And The Slashdot Effect (Or How I Learned To Scale APIs)"},{"url":"posts/2018/12/achieving-resiliency-with-queues-building-a-system-that-never-skips-a-beat-in-a-billion/","text":"Braze processes billions and billions of events per day on behalf of its customers, resulting in billions of hyper-focused, personalized messages—but failing to send one of those messages has consequences. To make sure those key messages are always correct and always on time, Braze takes a strategic approach to how we leverage job queues. What's a Job Queue? A typical job queue is an architectural pattern where processes submit computation jobs to a queue and other processes actually execute the jobs. This is usually a good thing—when used properly, it gives you degrees of concurrency, scalability, and redundancy that you can't get with a traditional request–response paradigm. Many workers can be executing different jobs simultaneously in multiple processes, multiple machines, or even multiple data centers for peak concurrency. You can assign certain worker nodes to work on certain queues and send particular jobs to specific queues, allowing you to scale resources as needed. If a worker process crashes or a data center goes offline, other workers can execute the remaining jobs. While you can certainly apply these principles and run a job-queueing system easily at a small scale, the seams start to show (and even burst) when you're processing billions and billions of jobs. Let's take a look at a few problems Braze has faced as we've grown from processing thousands, to millions, and now billions of jobs per day. Check out the rest of this blog post at Building Braze !","tags":"Software Development","loc":"posts/2018/12/achieving-resiliency-with-queues-building-a-system-that-never-skips-a-beat-in-a-billion/","title":"Achieving Resiliency With Queues: Building A System That Never Skips A Beat In A Billion"},{"url":"posts/2018/06/code-vision/","text":"While I'm currently on an early flight from Indianapolis to Newark, I'd love to expound on some thoughts I've had recently about what I call \"code vision\". I define this as a written, iterated-on plan for what a codebase should look like in the future. This is different from what features are in the pipeline - think more of a big-picture, directional plan for the architecture. I believe that this holistic view is needed as a codebase accumulates age and a significant number of active developers, much like a past-startup-stage company needs vision and values. I believe the process of refactoring to be largely ineffective without first having this vision . As a former Boy Scout, I have heard the phrase \"Try and leave this world a little better than you found it\" by Robert Baden-Powell more times than I can count, and I try to remember it in many different contexts - code included. I think that this is easier to apply with nature and the outdoors - \"better\" is quite clear to us: reduce waste, don't pollute, and try not to disturb the wildlife. Software is a bit different, however, in that \"better\" is often a very subjective thing. Refactoring - the process of leaving the codebase a little better than you found it - can be quite aimless unless we have a common concept of \"better\". Imagine that you have a class that is ugly - it has one gigantic method. You break it up into a series of smaller functions that are easier to test, and call them procedurally from one \"master\" method or function. Your comments make it clearer what is going on, but ultimately the class performs the same function. If you have several classes with similar structure - one big method or several small, well-commented methods - all wired together without a class hierarchy or unifying framework, you've only put lipstick on a pig. Your system itself is still disorganized, so you find some commonalities between parts - say you are rendering HTML pages and you create a base Controller class to take care of authorization, or you are processing SQL-driven reports, so you create a SqlReport base class and rewrite several classes to use this new class hierarchy. Without a holistic vision, it's likely that the next person to work on rendering pages or generating reports will do their own thing or simply pattern match (i.e. copy, paste, and modify) without thinking about how their additions or changes fit into the greater picture. This leads to JSON API controllers using a base Controller class geared toward rendering HTML and large bits of transactional SQL statements using a SqlReport class geared toward asynchronous queries run on a read replica. I have worked at a number of companies, but most of them had relatively young codebases, with fewer than 20 or 30 developers and fewer than 5 years of history. I've had, however, the benefit of working on contracts or patent cases where I've been privy to much, much larger and older codebases as well, and have seen ones with clear and obvious vision, as well as ones without. Needless to say, the ones without were the least organized, the least understandable, and the hardest to understand. The ones with vision tended to share several common, positive values: clarity, modularization, and simplicity . Moreover, they tended to exhibit what I refer to as \" framework-oriented thinking \" - an architectural approach that favors frameworks written by architects and senior engineers with components and extensions built by everyone. These are inherently modular, and often were surprisingly simple and clear. One only had to read the docs for the major framework components - often base classes or abstract classes that made up a pipeline, a composite rendering framework, or a communication paradigm - in order to understand the big picture. From there, any individual components made sense. It is easy to understand the purpose and function of FanOutPipelineTask or class ColoredButton extends Button (extends UIComponent) , and one couldn't (without significant effort, anyway) try to shim a subclass of SqlReport to run on the write primary! New development aside, with this level of increased clarity and understanding, the process of refactoring changes from a cloudy, somewhat aimless task to one that has purpose and is rather easy to understand. With a code vision and \"framework-oriented thinking\", one can evaluate methods, classes, and components and ask themselves \"does this component fit into the paradigm of the framework as is?\" and if the answer is no, then ask \"should this component change, or should the framework change?\". Methods can still be divided into less complex sub-methods, but now classes and components can be changed with purpose - to better fit into the rest of the system. Code review changes as well - a reviewer can now ask \"do these additions or changes fit in with the framework, or change the framework itself to better suit its purpose?\" rather than simply \"does this code function properly and is it tested appropriately?\". I'd love to hear your thoughts on this concept. Shoot me an email at z@z11k.com or find me on LinkedIn - I'm a sucker for organization and structure, but I love hearing the other side as well - maybe you have a good reason that such vision and structure hinders feature progress or is overly prescriptive. Let me know and let's discuss to make the software world a better place!","tags":"Software Development","loc":"posts/2018/06/code-vision/","title":"Code Vision"},{"url":"posts/2018/03/leaving-nashville/","text":"8 years, 6 months, 1 days. I moved to Nashville 8 years, 6 months, and 1 day ago. I spent the first night with my parents at the Comfort Inn Downtown Nashville/Vanderbilt on Demonbreun Street. At that time I didn't know how to pronounce Demonbreun or barely where I was in relation to campus. We woke up early the next day on August 22, 2009 - a Saturday - to start move-in weekend at Vanderbilt. It was an incredible day and over the course of the next 4 years I gained an incredible education and a number of lifelong friends. After undergrad, I started grad school but fairly quickly realized it wasn't quite for me (I'm sure I'll try again later...) and started working full-time for Optio Labs, which I had worked with previously as an undergrad. There, I made more lifelong friends (and friends of those friends!) and learned a lot about cybersecurity, software engineering, and teamwork. I also splurged once and bought a used Porsche Boxster I found while taking a 10 minute break at work one day, which I regret selling to this day. Aside from that, I lived in a cool apartment in 12 South and dated someone that I originally met while studying abroad as an undergrad. She and I eventually broke up, and I may have complained about the job a few times, but ultimately I would not give up any single experience during those chapters of my life. Eventually I changed jobs and started working as a remote engineer for a SF-based company. I also started dating someone new who I met within my first week of using a dating app - didn't see that one coming! A short few months later I bought a house in this up-and-coming neighborhood called The Nations for a decent bit more than I originally planned to spend, which was a bit worrying, but the neighborhood had so much potential so I went for it anyway. Since then, I've switched jobs a few times and made plenty of new friends at those, both on the engineering teams and from working across from tons of other extremely interesting people. I've also made other various friends - neighbors, friends of friends, and occasionally random people I've met out at bars, parties, and events in this vibrant city! All good things have to come to an end at some point, and some doors have to close in order to open new ones. I decided to close my consulting company about 2 months ago for good, as the underlying focus was not sustainable. I had a very hard conversation (definitely the hardest I've ever had) with myself and others, and eventually broke up with an incredible person I had been dating for several years. I also decided to sell my house here in Nashville to move to New York City, where I've tried to move unsuccessfully for the last 5 years. It took the absolute biggest all-in jump I've ever taken in my life, and I've definitely underestimated the emotional capacity I have for change/stress (thanks to all of you, friends, who have helped me through all/any of that). All of that just said - I'm extremely excited for the new things to come. I have about 8 days left in Nashville (mostly March 3 to 9) and it really overwhelmed me today, so I felt compelled to write out some kind of note detailing all of the things I'm happy for.","tags":"Life","loc":"posts/2018/03/leaving-nashville/","title":"Leaving Nashville"},{"url":"posts/2018/01/proverbs-and-programming/","text":"I collected a variety of proverbs over the course of 2017 that I thought had some insight into the world of software engineering as well. This won't be a horribly technical post, but a sort of fun rehash of experiences I've had thus far working with engineers, teams, product managers, CEOs, etc. over the years. Don't cross a bridge before you come to it Many systems that I've helped develop or analyze have had their fair share of IAbstractWindowFactory or IInterfacePrototype -like classes and interfaces. I've learned that the most effective libraries and frameworks are not the ones that account for everything possible from the beginning, but that consider the immediate needs and the needs 3 to 6 months into the future. Beyond that, the unknowns are still so unknown — will the project even exist then? Will the goals still be the same or similar? I have found that keeping frameworks and libraries relatively simple, especially when beginning, make them more manageable and simple for others to use. Don't let the perfect become the enemy of the good In the same vein as the previous point — working on frameworks and libraries (or even apps or other projects) have significant potential to get out of hand quickly. Potential features are dreamed up, opportunities for plugins and expansions run wild, and chances for over-engineering rocket skyward. I think that even for personal projects, one should put on their product manager hat and really spend significant time away from their IDE and in Trello (or Excel, or on post-its, or whatever you do). It's incredibly easy to dream up a perfect project with thousands of features, integrations, and capabilities, then never reach the finish line because of it. The longest mile is the last mile home As a contractor who has helped a number of companies develop all kinds of applications, it's easy to give clients the false perspective that an application is almost finished or is really ahead of schedule. The first few important features look like a ton of accomplished work, but at the end of the day, it's the last few features and list of improvements and bug fixes that end up taking a disproportionately long time. Time is money I remember at a cyber security company that I once worked for — we had a meeting with everyone on the team — full time or contract — for well over two or three hours planning out a research project. We debated technologies to use, skills we'd need, subcontractors we'd need to hire. We left exhausted but glad that we'd cranked out all of that good work. A week later (or so… my memory of the exact story is a bit fuzzy) the project was shelved. I thought about all of that work we had done and what, approximately, each person at the table's time was worth that afternoon. At the end of that day, we'd spent well over $5000. If some more front-end product planning had been done, we could have realized the impracticality of the potential product and saved quite a bit of money! More of these to come — I think I have well over 50 proverbs saved!","tags":"Engineering Management","loc":"posts/2018/01/proverbs-and-programming/","title":"Proverbs and Programming"},{"url":"posts/2017/08/running-geodjango-on-heroku-august-2017/","text":"Heroku Setup There are currently two ways to get the GDAL, GEOS, and PROJ libraries needed by GeoDjango onto your Heroku slug Via a custom buildpack ( https://github.com/cyberdelia/heroku-geo-buildpack ) 1. Via setting the environment variable BUILD_WITH_GEO_LIBRARIES=1 while using the default Python buildpack This has two issues as of 8–16–2017: The current \"stack\" on Heroku is heroku-16. This stack does not properly copy over libjasper, which is a required linked library by libgdal. One solution is to set your stack to cedar-14 instead. Another solution is to use a slightly modified custom buildpack ( https://github.com/dschep/heroku-geo-buildpack ) or a slightly modified offical buildpack ([https://github.com/TrailblazingTech/heroku-buildpack-python#fix-gdal-jasper] (https://github.com/TrailblazingTech/heroku-buildpack-python#fix-gdal-jasper)). If you stick with the heroku-16 stack, you'll want to add the following to your settings.py: GDAL_LIBRARY_PATH = os.getenv(‘GDAL_LIBRARY_PATH') GEOS_LIBRARY_PATH = > os.getenv(‘GEOS_LIBRARY_PATH') Then, if using the official Python buildpack, you will want to set your environment variables like so: GDAL_LIBRARY_PATH=/app/.heroku/vendor/lib/libgdal.so > GEOS_LIBRARY_PATH=/app/.heroku/vendor/lib/libgeos_c.so This may be required as well on cedar-14 — I didn't take notes during that part of the struggle :-) Django Settings Changes Add ‘django.contrib.gis' to your INSTALLED_APPS 1. Change your DATABASES dict to use postgis, like so: DATABASES[‘default'] = dj_database_url.config(conn_max_age=600, > default='postgis://localhost:5432/{}'.format(APP_NAME)) # VERY IMPORTANT BECAUSE HEROKU WILL USE ‘postgres' AS THE SCHEME DATABASES[‘default'][‘ENGINE'] = ‘django.contrib.gis.db.backends.postgis' Other Django Things In urls.py, change from django.contrib import admin to from django.contrib.gis import admin for admin urls In whatever app makes sense, add the following migration: from django.contrib.postgres.operations import CreateExtension from > django.db import migrations class Migration(migrations.Migration): operations = [ > CreateExtension(‘postgis') ] Credits cyberdelia and dschep on GitHub for helping provide and work through the gdal dependency issues with a custom buildpack. GitHub Example https://github.com/TrailblazingTech/django-gis-heroku","tags":"Software Development","loc":"posts/2017/08/running-geodjango-on-heroku-august-2017/","title":"Running GeoDjango on Heroku (August 2017)"},{"url":"posts/2017/07/devops-and-horizontal-scaling-part-3/","text":"Time to finish what we started: the last 4 points/suggestions on DevOps and Horizontal Scaling. Manual and Automated Code Review Manual code reviews are easy — GitHub, Gerrit, and a host of other tools support this workflow. The gist is that everyone's pull requests need to be seen and reviewed by a senior developer/manager/etc. before being merged into the main development branch. The idea is that with many eyes on the code, problems can be spotted and solved early while in context rather than later when the source of the issue can be harder to find. This is the first stage of code reviews. The second stage (and final, in my eyes, as it's not horribly complicated) of code reviews is automated code reviews. This sounds (for now, until AI takes over) more complicated than it really is — ideally through a Jenkins job or some other task runner, run a series of tools like pylint, jslint, Codacy, SonarQube, or even valgrind over the PR. These tools produce all kinds of output, so they'll need to be set up to provide the best kind of feedback to the developer who made the PR. These types of tools do style analysis, static analysis, and even memory usage analysis, and can find harder-to-spot problems or hotspots for null pointers, memory leaks, and other types of issues. As your team grows and every developer is handling a different kind of complexity, manual and automated code reviews help to make sure that everything is being reviewed and understood before being deployed, both promoting high-quality code and ensuring that if something were to go wrong, at least 2 developers should understand any particular part of the codebase. Redundancy is your friend. Scalability through PaaS & Non-monolithic Architecture For a very large number of systems, it makes a lot of sense to be using a PaaS (platform-as-a-service) provider like Heroku, Google App Engine, or Engine Yard. You don't need to worry about knowing how to use Ubuntu/Debian or Windows Server and you don't need to understand how to use Docker or lxd or rkt or any number of new containerization tools. These cloud PaaS providers just work . (As an aside, You might look at Heroku's pricing and think \"but hosting a small server on AWS and managing it myself will save me $200 per month!\", but in the end, I'm sure you will spend far more in paying employees for hours spent setting up, debugging, and maintaining than the $200 you save each month.) In addition to the relieved knowledge pressure by using a PaaS instead of self-managed cloud servers (or on-premises machines), you remove a specialization-of-function role of the \"ops/IT/deployments person\", so if something isn't working right, more of your team will be effective in diagnosing and solving the problem. That's a good thing! Paired with this is the use of a non-monolithic architecture. This one actually adds complexity, but the payback from this investment can be significant for some engineering teams. I'll go into this further down this section. \"Level One\" of this concept is simply migrating your web services and backend resources over to a PaaS (I'll use Heroku as an example, due to my familiarity with it) like Heroku. All of a sudden, all of the components in your deployments are accessible from one place, and often come with monitoring dashboards or inspection tools baked right in. Settings are configured via environment variables (12factor.net) and logs are all piped into one nice display. Cool! Getting to \"Level Two\" involves integrating your environments across something like a \"pipeline\" in Heroku. This may exist for your PaaS provider or you may have to create this abstraction yourself, but the results are fantastic. With Heroku's Pipelines feature, you can integrate \"Review Apps\" (where GitHub Pull Requests are automatically built and deployed to allow for manual testing and usage before merging into the development branch), continuous integration (previously with an outside tool, but Heroku has released their own CI function), and staged deployment. With these tools integrated, it becomes easy for developers (and honestly any other stakeholder) to see and understand the flow of how new code and bug fixes make it down the pipe. This makes it much easier to add new engineers and hit the ground running quickly. \"Level Three\" starts to introduce microservices or another non-monolithic architecture. Let me start by saying microservices aren't for every organization — in fact, I highly recommend against them for small organizations or for prototypes/new projects. In a lot of cases like that, having a monolithic architecture is going to keep things simple and keep your feature deployment cycle times much leaner. Adapting a non-monolithic architecture encourages a number of good practices: decoupling of components, increased test-ability, and system isolation, to name a few. It also increases the complexity of deployments and where things can go wrong. For a highly functional software engineering team, the benefits can often outweigh the drawbacks — especially one with engineers devoted to the infrastructure itself (especially when cost and other factors move you away from PaaS offerings). With microservices used in a large project, it becomes easier for new engineers to become acquainted with complex functionality quicker, because they (hopefully) no longer need to understand far-reaching implications between systems or comprehend huge modules that span a wide swath of functional domains. It also becomes easier to roll out new functionality: as the tight coupling between components decreases, the risk of changes to one function breaking another function decreases as well. Logging, Instrumentation, and Monitoring Logging seems pretty simple, and it is. Unless you have specific logging requirements, like from HIPAA, you should pretty much spit out a lot of logs (but not too many… I know that's vague) and save them for at least long enough that it's unlikely you need them again (and if it is likely, compress them and save them somewhere cheap). Instrumentation and monitoring get a little more complex. Tools like New Relic or Dynatrace are instrumental (ha!) to doing this successfully — they provide tools and dashboards that work with all types of projects to analyze performance and detect problems that otherwise go undetected. The \"Level One\" of logging, instrumentation, and monitoring, is simply to have these tools hooked up. On Heroku, I often use the free tiers of LogEntries and New Relic to start. The benefits you get from simply ensuring that these are hooked up right are immense: you can immediately start seeing slow requests/queries or performance bottlenecks in infrastructure, and you can begin to trace anomalous events back to specific sections of logs to get better context when hunting down issues. \"Level Two\" (the final level, much like my view on code reviews) of this concept involves completing full-system coverage using these tools, and configuring these tools for automated alerts (what I would call \"true monitoring\"). Instead of going into the New Relic control panel when something is going wrong, it's far better to be warned by New Relic (via Slack, email, or a host of other options) that something is slowing down or about to go wrong. LogEntries and New Relic can both be set up to warn you about certain things: Apdex score warnings in New Relic can warn you of overall app sluggishness, while New Relic's Key Transactions feature can be set up to warn you of sluggishness or errors in particular processes, such as customer checkout or PBX/customer service initial response. LogEntries has a number of built-in pattern recognizers for Heroku, and StackOverflow and Google are full of many, many more for detecting anomalous behavior or otherwise unseen errors. The benefit of these concepts to horizontal scaling (and general operation) of a team are plentiful: democratized error handling and response — the whole team has the tools to analyze even the most technically complicated of issues; continuous measuring and monitoring of performance —sluggishness introduced by new features or new code is caught early and quickly; and performance quantification — product managers and executives can better understand the impact of technical debt and architectural refactorings on the quality of the product. Architecture Review Team The last item on my list of thing for building out a \"DevOps culture\" and a horizontally scalable team doesn't have levels: the Architecture Review Team. I think at large enough organizations, a team needs to exist, composed of senior engineers of sub-teams and/or software architects with engineering managers (i.e. people with budgets). Ideally, all participants have a strong understanding of the impact that spending engineering time has on the overall organization (i.e. it's expensive), since software architecture at large firms is a balancing act of figuring out the best architecture at the appropriate cost. During each meeting, which can happen as often as once a month and as infrequently as every six months (though I like the happy medium of quarterly), the team should get together and discuss the recent and upcoming major initiatives regarding new features, the state of the bug tracker, the state of the performance monitoring tools, and the infrastructure budget (money spent on servers, software, etc.). The goal of the meeting is to make sure everyone has an understanding of the macro state of the system, and to allow for discussions about issues like tech debt, refactoring, hiring, and infrastructure changes. It should also prompt discussions about future software initiatives — architects and team leads will have a good idea about the scope of new initiatives and whether or not it will have a deleterious effect on the existing system without additional planning or a round of refactoring first. How do you do it? As I mentioned in the last post: careful planning and working with the engineering team and outside product owners/users is critical to implementing these steps successfully. This is actually something that I love to consult with companies about — shoot me an e-mail at z@z11k.com to get a bit more information about that. I love traveling and will gladly come to you wherever you are to help your company succeed in implementing these tools.","tags":"Engineering Management","loc":"posts/2017/07/devops-and-horizontal-scaling-part-3/","title":"DevOps and Horizontal Scaling: Part 3"},{"url":"posts/2017/06/devops-and-horizontal-scaling-part-2/","text":"Lets talk about the first 3 of the 7 tools I described in the last post! Product Management involving feature flags, switches, and samples Product management has a simple sort of flow to it — tickets and feature requests come in, things are prioritized between managers of different teams, and the end result gets put into a pipeline for engineers to work on. Ideally the flow is smooth and reliable, with refactoring and testing getting their fair share of time as well. In practice, most organizations are way behind where they'd like to be, so these things get de-prioritized. In my opinion, based on my time either working at or working for a number of companies that develop software, the first box to check in product management is just having a workflow that can be tracked — do you know the mean time between a bug being reported and it being fixed? What about a simple feature request? Obviously the variables involved make it so that these numbers don't fit into a nice and tidy box like \"how many widgets did our old factory produce, and how many does our new one produce?\", but they're still important to keep track of. I deem this \"level one\" of product management. \"Level two\" of product management is not much harder — just make sure the rest of the organization is aware of bug fixes and finished features. Some organizations use JIRA, which can be set up in a way to notify requestors about the status of their request, or post updates to Slack when releases get pushed to production. Automating this process is \"level two\". You'd be surprised just how happy people get when they have some semblance of an idea of what's going on in the engineering department related to their needs! \"Level three\" of product management, however, involves adding a significant QA process to the mix. Putting aside the \"developer-centric\" parts of QA (code reviews, testing, CI), this should involve at least some process at the roll-out level. Maybe you have a QA person or QA team that evaluates features in your development and staging deployments — this is a great first step — but changing the culture to liberally use feature flags, feature switches, and sampled roll-outs will bring you to \"level three\". If you roll something out that has a deleterious effect, you ought to be able to quickly roll it back out. With feature flags and switches, this is easy to do without redeploying or merging hot-fixes into your production environment. If you aren't sure about a particular feature — especially revamps of UI/UX — you'll learn to love sampling your releases — releasing only to select sets of users at once. These capabilities give you real-time control over your product allowing your engineering team to continue with their normal flow instead of needing to respond to emergencies on a regular basis. This allows you to keep adding engineers to your team while remaining efficient. Consistent, Realistic Development (and Staging) Environments Consistent environments in your development and staging environment (hosted) and for your engineers locally is key to continued success as more people work on more and more complex features. \"Level one\" here is simply having a procedure for engineers to run a production-like environment on their own machine. Without this, deployments are dangerous and feature release can be troublesome. The technologies don't need to mirror each other — for instance you might actually be deploying load-balanced Docker containers to a Kubernetes cluster, but locally you can run manage.py runserver and achieve 99% the same result — but they should be similar enough that you rarely run into issues in production that you don't have locally. \"Level two\" is where architecture and the \"ops\" part of DevOps comes into play — ideally your system should operate as stateless-ly as possible. If you have your database running separate from your application server, your ElasticSearch cluster in its own world, and your cache running in its own universe, then you're well on your way to achieving \"level two\". That kind of separation allows you to test your services independently and substitute out equivalent (but less complex/costly) services locally or on development/staging environments. The last, and often neglected, level of Realistic Environments is having similar (but sanitized) data available in different environments and locally. It's tough to work on pagination algorithms, document search features, or speeding up high-throughput views with a cache without having sufficient or realistic data to test locally. This is something else that lends itself well to automation on the \"ops\" side of DevOps. A daily/weekly/etc. Jenkins job that takes a database or cache backup, sanitizes out sensitive information, and stores it in S3 can save you from countless production deployments solving problems that only exist on production. With these strategies in place, \"tribal knowledge\" of how systems work starts to go away and onboarding new engineers becomes a piece of cake, allowing you to scale your team more effectively. Extensive Automated Testing and Continuous Integration The last of the three tools I'm going to talk about in this post is testing. It may seem obvious, given the number of \"How to Test using JUnit\" talks at local meetups, but testing should be your best friend. Countless bad deployments are saved due to effective unit and integration tests every day. The first hurdle is simply test coverage, and reporting on test coverage. A simple and effective KPI (though remember, engineering can be fuzzy and full of variables, so you can't worship the KPI) is test coverage. Several derivatives exist, such as \"test coverage of critical paths\", but having a few of these reported on weekly can be a great tool to keep the team abreast of how well-tested your code is. This might involve unit tests, integration tests, load tests, or more, but the first step is simply having the tests available and running/reporting on them on occasion. The second hurdle is automating those tests — this could be as simple as a Jenkins job that runs ./manage.py test on every PR and before deploying to development/staging environments. It may involve a suite of tests, or involve more complex testing frameworks like Cucumber using Behavior-driven Development. Knowing when tests fail and responding to them quickly and effectively is \"level two\" in this realm. I'm a big fan of using Cucumber or Behave — two Python frameworks for BDD — because of the ability to create cross-products of tests easily. This might be something like end-to-end tests of eCommerce, where each product has a suite of 50 tests run on it, and simply adding 3–4 additional product codes actually runs an additional 150–200 tests. The final stage of effective automated testing is actually less about testing and more about culture/product management. It takes more time to implement, being a culture thing, but part of the bug reporting/fixing process should involve unit and integration tests to prevent regressions from occurring more than once. Users are frustrated when things break, but they're even more frustrate when the same thing breaks a second time. This one is a little more loose, as there isn't a framework you can add or a tool you can use to solve it instantly, but is a great goal to try to achieve when it comes to automated testing. With this kind of testing in place, engineers don't need to know every last detail and caveat of system functionality — they can just start coding features and fixing bugs in their first couple of weeks. How do you do it? Careful planning and working with the engineering team and outside product owners/users is critical to implementing these steps successfully. This is actually something that I love to consult with companies about — shoot me an e-mail at z@z11k.com to get a bit more information about that. I love traveling and will gladly come to you wherever you are to help your company succeed in implementing these tools. What next? In my next post, I'll go over the last 4 tools to implementing a DevOps culture to enable Horizontal Scaling within an engineering organization. Stay tuned!","tags":"Engineering Management","loc":"posts/2017/06/devops-and-horizontal-scaling-part-2/","title":"DevOps and Horizontal Scaling: Part 2"},{"url":"posts/2017/06/devops-and-horizontal-scaling-part-1/","text":"How do engineering departments scale beyond teams of 10–15 people and stay nimble? Let's imagine scaling an engineering department that starts with 3 employees and an \"efficiency factor\" of 100%(bear with me). You hire a couple more experienced engineers, start having daily stand-ups and a weekly planning meeting and compared to your previous \"efficiency rating\", you're now outputting the work of 5 engineers at 90% efficiency — not bad! The business starts to need more things, and managing tickets and support requests start to become a task of their own, so your efficiency drops to around 75%. You hire a product manager and a support engineer, bringing you up to 7 employees and back to an efficiency rating around 95%(your core engineers are no longer managing tickets or support requests). Time continues and business is good, but as the features keep piling on — internal CRM features, ERP features for driving workflow, eCommerce features for increasing sales — things start to break down. Deployments aren't as smooth or fast as they used to be, minor feature additions are breaking existing features, and bug fixes are slow due to the manual testing involved. You're smart and have faced these types of multifaceted problems before, so you focus on an easy win: infrastructure. You hire an infrastructure engineer to sort out deployments and scaling. While she's getting integrated with the team, you start a feature freeze to work on better testing. Things seem to be looking up — engineering KPIs are stabilizing and people feel good about it. Time moves on, you have occasional cycles of writing lots of tests, and you start to accept the occasional deployment hiccup as a fact of life. You aren't too concerned with that engineering efficiency rating anymore (which is hovering around 50%) because the product has just gotten larger and that's how software works — isn't the phrase \"oh look at that big and slow-moving company\" just normal? It does not have to be! This is where complacency stops engineering teams from achieving the very possible horizontal scaling that many hugely successful companies achieve. With the right investment in DevOps , horizontal scaling is absolutely possible beyond 10, 20, 50, or even 100 engineers. DevOps is a phrase that gets used a lot for a lot of different reasons, so let's start with a couple definitions: DevOps — a portmanteau of \"development\" and \"operations\". A combination of the engineers who write new features and the IT staff that keep the servers running. DevOps is an umbrella term for a culture of collaboration between these participants in the organizational software lifecycle. Horizontal Scaling — the ability to scale by adding more resources to your pool of resources. Note that this isn't vertical scaling — adding more powerful resources (i.e. a superstar engineer), but simply adding more resources. I like to think of implementing \"the DevOps culture\" as something like an organization adopting Six Sigma or Kaizen — it's a set of tools with a cohesive purpose that you can pick and choose from to address your needs. What are those tools, then? Product Management involving feature flags, switches, and samples Consistent, Realistic Development (and Staging) Environments Extensive Automated Testing and Continuous Integration Manual and Automated Code Review Scalability through PaaS & Non-monolithic Architecture Logging, Instrumentation, and Monitoring Architecture Review Team In my next post, I'll go over each of these tools, tying them back to the ultimate goal of keeping \"engineering efficiency\" high while allowing an organization to scale horizontally in engineering team size.","tags":"Engineering Management","loc":"posts/2017/06/devops-and-horizontal-scaling-part-1/","title":"DevOps and Horizontal Scaling: Part 1"},{"url":"posts/2017/02/tags-for-sentry-for-android/","text":"I could not figure out how to add tags or anything to my captured exceptions for Android, which led them to be rather bland/unhelpful when trying to pin down what was causing a bug on the Android camera (notorious for being inconsistent across manufacturers, devices, Android versions, etc.) I ended up doing the following, which hopefully helps others! import com.getsentry.raven.android.Raven ; import com.getsentry.raven.event.EventBuilder ; import com.getsentry.raven.event.helper.EventBuilderHelper ... Raven . init ( this . getApplicationContext ()); try { Field f = Raven . class . getDeclaredField ( \"raven\" ); f . setAccessible ( true ); com . getsentry . raven . Raven raven = ( com . getsentry . raven . Raven ) f . get ( Raven . class ); final ConnectivityManager connManager = ( ConnectivityManager ) getSystemService ( CONNECTIVITY_SERVICE ); final NetworkInfo mWifi = connManager . getNetworkInfo ( ConnectivityManager . TYPE_WIFI ); final PackageInfo pInfo = getPackageManager () . getPackageInfo ( getPackageName (), 0 );; raven . addBuilderHelper ( new EventBuilderHelper () { @Override public void helpBuildingEvent ( EventBuilder eventBuilder ) { eventBuilder . withTag ( \"wifi\" , String . valueOf ( mWifi . isConnected ())); eventBuilder . withTag ( \"app_version\" , pInfo . versionName ); eventBuilder . withTag ( \"build_version\" , Build . VERSION . RELEASE ); eventBuilder . withTag ( \"build_manufacturer\" , Build . MANUFACTURER ); eventBuilder . withTag ( \"build_brand\" , Build . BRAND ); eventBuilder . withTag ( \"build_device\" , Build . DEVICE ); eventBuilder . withTag ( \"build_product\" , Build . PRODUCT ); eventBuilder . withTag ( \"debug_app\" , BuildConfig . DEBUG ? \"true\" : \"false\" ); eventBuilder . withTag ( \"build_brand\" , Build . BRAND ); } }); } catch ( Exception e ) { Raven . capture ( e ); } Obviously you can see the pattern here to add tags using the eventBuilder. I have no idea why this was omitted in the Android version of Raven, but reflection saves the day per usual.","tags":"Software Development","loc":"posts/2017/02/tags-for-sentry-for-android/","title":"Tags for Sentry for Android"},{"url":"posts/2016/04/github-fork-and-pr-model-unknown-repository/","text":"Have you ever had a PR that you can't quite merge because it needs changes, but the user has gone away (either deleted their account/branch/fork or are no longer a collaborator on a project)? Do you see \"unknown repository\" when you pull up the PR on GitHub? Fear no more — you can salvage that code! Add the following line to the origin block (or whatever remote you are using — be sure to change to refs/remotes/REMOTE/pr/*) in /.git/config. fetch = +refs/pull/ /head:refs/remotes/origin/pr/ Then you can do git checkout pr/### to grab their code, then git checkout -b my-new-branch to create a local branch out of the once-lost PR! Pretty neat, eh?","tags":"Software Development","loc":"posts/2016/04/github-fork-and-pr-model-unknown-repository/","title":"GitHub Fork and PR Model — \"Unknown Repository\""},{"url":"posts/2015/10/docker-ubuntu-1404-and-apt-show-versions/","text":"If you're using docker and the ubuntu:14.04 image (and possibly other images as well), then you'll want to follow the following steps to get apt-show-versions to work, which is an essential tool if you want perfectly reproducible builds/deploys (so that you can figure out exactly what versions to provide to apt-get!) If you first install apt-show-versions and get something like: Errors were encountered while processing: apt-show-versions Then if you try to run it and get: root@docker-host:/# apt-show-versions python3 Error: No information about packages! (Maybe no deb entries?) It's likely that the ubuntu:14.04 docker image has taken some space-saving shortcuts that won't allow apt-show-versions to read the dpkg index. Let's fix that! root@docker-host:/# rm /etc/apt/apt.conf.d/docker-gzip-indexes root@docker-host:/# apt-get update That should fix it! Next time you run apt-show-versions it should work just fine!","tags":"Software Development","loc":"posts/2015/10/docker-ubuntu-1404-and-apt-show-versions/","title":"Docker, Ubuntu 14.04, and apt-show-versions"},{"url":"posts/2015/07/benchmarking-django-model-saving/","text":"When using Django's ORM as a basis for data harvesting (probably not the best idea in the first place, but sometimes it's easier to go with what you know), I learned that Django isn't exactly crazy fast when saving models to the database. Here are some stats on doing different things with Django to try to speed it up. First, I wrote two methods and used cProfile to profile them. MyModel is a model with a single max_length=100 CharField. def benchmark(): for i in range(1, 10000): x = MyModel(data=\"Some Random Data\") x.save() def benchmark_bulk(): items = [] for i in range(1, 10000): x = MyModel(data=\"Some Random Data\") items.append(x) MyModel.objects.bulk_create(items) I used cProfile to profile these methods, using SQLite as the backing database. benchmark() took 14.87 seconds, and benchmark_bulk() took 0.400 seconds. Obvious improvement there by using bulk_create, but you can't use FKs to link subrecords to the records, as the pk property of those objects will not get set. When switching to MySQL: 8.09 seconds on benchmark() and 0.379 seconds on benchmark_bulk(). A little bit better — definitely better on benchmark, but not that much better when bulk inserting. When switching to Postgres: 6.71 seconds on benchmark() and 0.385 seconds on benchmark_bulk(). Even faster on benchmark, but not any faster for bulk insertion. Bulk insertion may be effectively capped out on speed. The bulk_create method on query.py seems to take about .2 seconds no matter the backend. Switching to Cassandra led to bulk_create not working, and benchmark() took 17.88 seconds! That wasn't what I expected here! Looking at the profiler stats, it seems it spent 11.63 seconds in time.sleep(), so maybe I'm doing something wrong — subtracting these two gives 6.25 seconds, which is closer to what I'd expect. Obviously the use cases here are very simple, but for simple model insertion speed, Postgres seems to win if you want a relational database and migrating to new software isn't hard. If you don't want to use Postgres (in our case, we like MySQL), or you need intense speedups, then modify your code to use bulk_create()! I'll make another post later this week or next week, with a more complicated, real-world data structure, and show how (likely) Cassandra would beat out both MySQL and Postgres for saving more complex models quickly, since we can eliminate FK relationships and save objects in a more NoSQL fashion. This comes at a querying and filtering cost, so I'll examine those too!","tags":"Software Development","loc":"posts/2015/07/benchmarking-django-model-saving/","title":"Benchmarking Django Model Saving"},{"url":"posts/2015/03/wrestling-with-python-on-os-x-yosemite/","text":"So I have been using my Macbook Pro for about 3 years for various projects, mostly in Java/Android, some Node here and there, I've installed Ruby and RVM for whatever reason once or twice, and I probably have 8 version of Python installed. I started to work on an open source Python project that I'd been working on (started on a different computer), and noticed some funky things about my Python install on my MBP. I was constantly getting errors like the following when I tried to install virtualenv and virtualenvwrapper: IOError: [Errno 13] Permission denied: ‘/usr/local/bin/virtualenv' Obviously at some point in time, over a few version of OSX and lots of Homebrewing, I'd screwed up my permissions. Let's fix it! First I went ahead and did a ‘brew update && brew upgrade'. I then did a ‘brew uninstall python3' and ‘brew uninstall python' just to clean everything up — that way I'd only have the OS X Yosemite default system Python installed. Next I reinstalled Python (2.7.9 when I did it) with ‘brew install python — framework'. Homebrewed Python comes with pip, so I did ‘pip list'. Hmmm…. that's funny! I have a ton of crap listed as installed, but I definitely just did a clean install. It turns out that pip will look in a number of different folders, ‘/Library/Python/2.7/site-packages/' being one of them. All of the packages I had installed previously using the system Python, such as awscli and goobook, were all sitting here. The Homebrewed version of pip is listing them though. This definitely isn't what I want… I did an ‘ls' in the ‘/Library/Python/2.7/site-packages/' folder and did a ‘sudo pip uninstall xyz' for every folder such as ‘xyz-1.2.3' in that directory, excluding pip and setuptools. I then did a ‘sudo rm -rf' for the pip and setuptools folders in this directory (just to make sure they don't cause problems). This left me with only easy_install.py and pkg_resources.py in that directory. Now let's install virtualenv. ‘pip install virtualenv' and ‘pip install virtualenvwrapper' now both work perfectly without needing to use ‘sudo' in front, because pip is attempting to install everything and link against existing packages in ‘/usr/local/lib/python2.7/site-packages' instead of in ‘/Library/Python/2.7/site-packages'. Yay! That's exactly what we want. I reinstalled awscli and other Python tools that I had gotten used to using, and pip installed them all correctly into the Homebrewed Python's site-packages folder and linked all executables to ‘/usr/local/bin'. Exactly what we want to make it modular and manageable, and especially not to require ‘sudo'. Great success!","tags":"Software Development","loc":"posts/2015/03/wrestling-with-python-on-os-x-yosemite/","title":"Wrestling with Python on OS X (Yosemite)"},{"url":"posts/2014/02/self-extracting-bash-patcher/","text":"Hey all, I haven't posted anything cool in a while, so here is a cool bash shell script I wrote a while back to generate a self-extracting patch (specifically for source code repos). It uses diff to generate output that can be fed into patch, then uses grep and sed to do some formatting as well as find all of the binary files that differ. The file list is piped to tar and gzipped, then the whole thing is jammed into a single, self-extracting bash script with the ability to do an initial dry run, as well as to revert all of the changes. I've found that sending entire source trees (20+ GB across 300+ git repos) can be inefficient if the recipient has the original code (he/she just needs your changes) — sending a 100 MB patch file is much easier! Here it is below — let me know if you can think of anything to improve it! You can also check it out or fork it on GitHub! https://github.com/zachmccormick/PatchGenerator Here is a link to the original article that I read a while back when I was first interested in making self-extracting bash scripts. Bash Self-Extracting Script | Linux Journal In this post I'll show you how to create a self extracting bash script to automate the installation of files on your… www.linuxjournal.com","tags":"Software Development","loc":"posts/2014/02/self-extracting-bash-patcher/","title":"Self-Extracting Bash Patcher"},{"url":"posts/2013/10/where-do-android-components-run/","text":"If you ever wondered where the different Android components run by default: Activity — always runs in UI/main thread Local-process Service — runs on invoking thread Local-process ContentProvider — runs on invoking thread Remote-process ContentProvider — runs on a thread in a thread pool on remote application process Remote-process (AIDL) Service — runs on a thread in a thread pool on remote application process Local-process BroadcastReceiver — always runs in UI/main thread Remote-process BroadcastReceiver — always runs in UI/main thread of remote process Link to GitHub repo with test project used for data: https://github.com/zachmccormick/AndroidComponentThreads","tags":"Software Development","loc":"posts/2013/10/where-do-android-components-run/","title":"Where do Android components run?"},{"url":"posts/2013/10/anonymous-shared-memory-library/","text":"I created an Android Library project that supplies you with an AshmemBuffer object, that you can pass between processes/applications via a service. Unfortunately, even though it is parcelable, you can't pass it via an Intent because Android doesn't let you do that (it uses file descriptors under the hood). It has IO functions such as \"writeByte\", \"writeBytes\", \"readByte\", and \"readBytes\", with functions for setting the pointer and getting the capacity. I plan to add more things later if anyone finds it useful. I just wanted to make something cool so that people could use ashmem at the Java layer without having to mess around with system calls/C/JNI/etc. Find it here: https://github.com/zachmccormick/AshmemLibrary","tags":"Software Development","loc":"posts/2013/10/anonymous-shared-memory-library/","title":"Anonymous Shared Memory Library"},{"url":"posts/2013/06/wwoofing-in-romania/","text":"From June 1 to June 14, I went to the Transylvania region of Romania to volunteer on a farm that does almost everything the old way — no tractor, no fertilizer, no pesticides. I headed out via train from Budapest to Huedin, which is a town of about 9000 about an hour from Cluj. From there, I went to Rachitele, a small village about 30km south of Huedin. That's where I start my two-week adventure… Rather than write it out chronologically, I wrote down titles for certain stories that I'll tell in bits and pieces. The trip was incredibly fun — I got to help do some hard labor (moving lots of dirt), cook food for everyone, and plan and execute a project to bring electric power to the old barn, which is being converted into a workshop and living area. I learned a lot about reducing, reusing, and recycling, which are essential skills in this fairly remote region, as well as some things about cooking and other various things. I highly suggest doing something like this! Hitchhiking In this area (and Eastern Europe in general), hitchhiking is an accepted practice. That said, to get to Rachitele from Huedin, I hitchhiked with an old man who did not speak a word of English and listened to folk accordion music on the way, with the old man pointing out all of the mountains, valleys, and rivers excitedly with a huge grin, trying to tell me in Romanian about how beautiful it was. I tried responding in Hungarian but he just looked at me blankly, so I just smiled and responded \"Da! Da!\". I saw him a few more times throughout my trip around Huedin, so I assume he lived there and was headed out that direction that day anyway. Hiking up the mountain When I arrived in Rachitele, I was told to ask for the \"casa Mexicano\" — the Mexican's house — and tried to do so at the town store/bar. They explained to me how to get there in a smattering of hand gestures and slow Romanian. After 2 or 3 minutes of explanation, I gave up and just pointed in a direction and said \"Da?\" They fixed my orientation a little bit and I headed down the road uphill. After 10 or 15 minutes of walking mostly uphill, I saw a farmer out in a field and asked the same question. He made a grand gesture that said to me \"keep going straight up\", so I did. After another 30 minutes of walking up the mountain, I saw a couple tending to their garden, and asked the same question. This time, they said \"no, no, no\" and had me go back downhill about 20 minutes and told me to take a different path. This pattern kept going for a couple of hours until, exhausted, I arrived at the farm. The paths were half footpaths and half dirt roads, and I figured that in the worst case, I could just sleep somewhere off the path, hope it didn't rain, and try again in the morning. Luckily I found it around 5pm or so. West Germany One of the days, my job was to cut out a hole in the side of the barn to put a window that had been recycled from somewhere in Germany and brought to Romania. Once I cut the hole, I was to build a windowframe to put the glass in, with a slot routed on the edges for the piece of glass to fit snugly inside. How do you cut a hole in the side of a barn? With a chainsaw of course! What kind of chainsaw might you have in Romania? One from West Germany! Not as in, a city in the western part of Germany, but it was made before the end of the Cold War, and said in bold letters \"Made in West Germany\". I thought it was pretty cool. The gear for the pullstring to start it stripped after using it for a while, but otherwise it worked fine, despite the lack of any safety features. After a couple of days, I had the windowframe built and the hole cut, and it fit securely inside. Project well done! Digging and Pálinka Part of the first week's work was to dig 6 cubic meters of red dirt from one place and bring it to another to eventually make plaster with. The owner of the farm, another two volunteers, and myself dug 4 cubic meters in the morning and loaded it onto the neighbor's tractor, and he moved it next to the barn after each load. After lunch, one of the volunteers, myself, and the owner all came back to dig more, but 4 men were standing around a car in the neighbors yard all holding shotglasses, with one of them holding a 2.5L bottle of Coca-Cola with a clear liquid in it. We came up to them and the owner asked in Romanian about more digging, but instead they said \"no, no, no\" and handed us glasses and filled them with pálinka, or ţuică in Romanian, which is a strong fruit brandy made by people in the countryside and in Eastern Europe. I, speaking absolutely no Romanian, stood in a circle holding my shovel with one hand and my shotglass of pálinka with the other, listening intently to the joking conversation between the 6 other men in Romanian. One of them was fake fighting and slapping around the owner of the farm and I think the old man was arguing with the other volunteer. I mostly just laughed when everyone else laughed or when I could understand rare words that were similar to Spanish. The best and most Eastern European part was when it would get mostly silent — everyone would be looking off in some direction, and someone would just sigh heavily and say \"Bine…\" (pronounced BEE-nay), and everyone would slowly nod and respond with \"da\". Needless to say after about an hour of conversing in Romanian drinking some probably-flammable fruit brandy, the rest of the day's digging went a little slower than the morning. We had to return another day to get the last 2 cubic meters of dirt, but we finished making a fence around the dirt to keep it from eroding in the rain. All in all we actually got a lot done despite the long midday break. I think we all slept well that night! Visit to Town This is a combination of two stories, but occasionally I would walk 30 minutes downhill to Rachitele either to go to Huedin or to get something from the bar or store, and several things happened there. On the last week, a German volunteer and I went down with large backpacks to buy several bottles of beer for all of the volunteers, as the next night was my going-away campfire. We walked into the town bar around 3pm, when it looked like most of the town must have been sitting down having a midday drink, and as soon as we walked in, it was something like a movie. Everyone went silent and started staring at us, saying nothing and examining us. I guess we looked that out-of-place, but the lady who ran the store and bar eventually did come help us and eventually everyone started acting normally again, so it worked out. On the way back, the German girl went to put a postcard in the mailbox, so I sat on the edge of a bridge with all of our things. Almost as soon as I sat down, I was targeted by, who the owner of the farm called her, the town drunk. She was a thin, weathered, elderly lady at least in her 60s, and she came up to me yelling something in Romanian and waving her hands back and forth trying to get her point across. Unfortunately by this time, I had not learned much of any Romanian, so I had no clue what she was saying. I just shrugged my shoulders and ignored her, but she came up to me even closer and said the same thing, just much louder, and made a pushing motion about a half meter from my face as if she was going to push me off of the bridge. I got up and said loudly, shrugging my shoulders with my hands in the air \"I DON'T KNOW\", and she finally backed off and said some angry-sounding remark in Romanian. I escaped that time, but I was convinced that I was going to end up pushed off of a bridge by an old Romanian woman and have to explain the bruises to everyone later. Disaster averted! Woman seeing Internet for the first time The last story I'll tell is that of a woman seeing the Internet for the first time. One day when I came inside after working on some different projects on the farm, the owner had someone sitting down next to him working on his laptop. She turned around and smiled at me and motioned to take some of the food that she brought, so I happily obliged and exchanged pleasantries in my (broken) newly learned Romanian. After cleaning up a bit, the owner explained that she had never seen the Internet before, and that he was helping her learn how to use it to get the Romanian equivalent of a high school diploma. He set her up with an e-mail address and showed her how to do several different things, and it was quite amazing to see her face light up and the excitement in her voice upon hearing all of the things she could do. This was so cool to me, as someone who grew up with this knowledge. I can't imagine how exciting that was for her, and I hope that it goes really well! Maybe I'll post more stories later, but these were some of the more comical and interesting highlights of the last two weeks. I highly encourage you to try volunteering on a farm somewhere, especially if you've never done such work or always lived in a city or in the suburbs. The pace of life and the quiet beauty of it are unparalleled, and I think it really gives you some time to think about whatever you need to think about in life. Not only that, with different organizations like WWOOF, you can visit a place that you may otherwise never get to go. Maybe I'll do it again someday somewhere even crazier, like Siberia, or somewhere in Africa!","tags":"Life","loc":"posts/2013/06/wwoofing-in-romania/","title":"WWOOFing in Romania"},{"url":"posts/2013/04/quant-interview-questions-part-3/","text":"I should be studying for a math test right now, but my brain needs a break from groups, subgroups, factor groups, rings, and fields (for my test on Wednesday) and from almost linear systems of differential equations and matrix exponentiation (for my test tomorrow morning), so here are the details of another one of my interviews! This one was definitely more computer science-y than the rest. Question Implement a heap data structure with the following methods that run in the following amounts of time, prove their runtime, and meet the extra specifications: Create heap in constant time Add a new element in log(n) time Retrieve the minimum element in constant time Remove the minimum element in log(n) time Data structure must be entirely immutable Data structure must be a balanced binary heap Answer I started out with no immediate idea how to do this in an immutable way, so I just went ahead and started out with a static create() method (I did this in Java) that returned a HeapNode object. It was obviously constant time. I knew that I needed to make a min-heap, so I went ahead and started on the add() method. I either set the left-child pointer to a new HeapNode containing the Object to be added if it didn't have a left child, or I called add() recursively on the left child. Likewise for the right — for now I chose randomly. I explained to the interviewer that I knew it needed to be balanced, but I'd go back to that after I had the basic methods done first. Retrieve in constant time was easy — obviously you are just returning the Object held by the top HeapNode. Removing the minimum element in log(n) time was a bit harder. I remembered my algorithms/discrete structures classes though, and knew you just needed to \"bubble down\" much like you do with add(), so I wrote the code to move the furthest-down node to the top, then keep swapping it with lower elements until the heap property was fulfilled. Again, I knew that it needed to be immutable and balanced, but I wanted a working heap first. Now that I had the 3 main methods of my heap finished, I worked on the balanced part. I decided that each node should keep track of the number of its left and right children. This works perfectly instead of my random approach from before for add(), so I went ahead and changed this. As for remove(), I had to go back and change this to bubble down to keep it balanced (remove the top node and replace it with the top node of the left or right subtree — whichever has more elements, then swap recursively to maintain heap structure). Now, about 45 minutes into the interview, I had the correct methods commented with very terse proofs of runtime, but without the immutability property. After pondering aloud for a couple of minutes about it, the interviewer gave me some sort of advice (I don't remember exactly since the interview was over a month ago), and, due to lack of time, I explained how I would implement the immutability: \"The Cool Part**\" ** denotes my personal opinion! Think about a min heap with elements (from top to bottom and left to right) 12, 7, 9. To insert 8, you will create a heap that looks like 12, 8, 9, 7 (where 12 is on the 0th row, 8 and 9 are on the 1st row, and 7 on the 2nd row). You might immediately think, \"Oh, I need to swap the left-child pointer on 12 to 8, then point 8's left-child pointer to 7.\" This is correct for a mutable heap, but I need to preserve the original heap. To preserve both heaps, instead create a new HeapNode with 12 as the element. Next, decide if you are going to add to the left or right subtree (you know the sizes). If you are going to add to the left, then set the right-child pointer of the new HeapNode to the original HeapNode containing 9. Next, at 7, you decide if 8 will replace 7 or descend — it is greater, so it replaces 7, so go ahead and create a new HeapNode with 8 inside, and point your new HeapNode with 12's left-child pointer to this new HeapNode with 8 inside. Finally, create a new HeapNode with 7 inside, and point this new HeapNode with 8's left-child pointer to this node. Return the new HeapNode with 12 as the element, and you now have a pointer to the old, still-the-same heap, and the new, changed heap too. Notice that we have changed NONE of the pointers in the original heap, thus we have achieved immutability. We only created 3 new nodes (since for the right subtree, we just pointed the new root to the old right subtree), which is still log(n) time for our heap. You can continue this process of creating new branches in log(n) time, and while you may generate an ugly looking diagram of pointers if you draw it out, you achieve immutability even with the runtime requirements. I'll leave the remove() method in an immutable way as an exercise to the reader! Sad News I couldn't quite meet all of these requirements in the 50 minutes or so I was allotted, so I unfortunately did not get the job. I admit fully that I came to this interview less prepared than the other ones (the other 3 interviews, I had studied at least 30 hours the week prior to each one; and for the whole set of interviews, I had probably studied probability, combinatorics, data structures, and algorithms for about 200 hours from November through February). I am still very interested in computational finance, and may apply to some of these firms again in the future post-PhD, as I think it would be thrilling to work on \"the street\" with some of the greatest mathematicians alive, but for now, I'll focus more on my research!","tags":"Mathematics","loc":"posts/2013/04/quant-interview-questions-part-3/","title":"Quant Interview Questions (part 3)"},{"url":"posts/2013/03/quant-interview-questions-part-2/","text":"Here is another interview question from a separate interview with the same firm: \"Given the results of a round-robin tournament, can you always create a list of teams ordered such that, when looking at a single team, the team to its left won against them, and the team to its right was beaten by them? \" Clarification A round-robin tournament, for those who are not familiar with the term, is a tournament where every team plays every other team once. For example, if you had 16 teams, then each team would play 15 games — one game against every other team. A concrete example of this question can be stated: 4 teams play a round robin tournament, team 1 beats all of the teams, team 2 beats teams 3 and 4, and team 3 beats team 4. A correct ordering would be [1,2,3,4]. Proof Strategy Looking at this problem, my strategy was first to prove it was not impossible with a small number of teams (4 for instance as above), then come up with a strategy from there. It immediately became apparent after giving one concrete case that I could do this quickly and succinctly through a proof by induction. Induction Base Case The base cases could be n=0. A tournament of 0 teams has a sequence of no elements and is properly ordered. Next, we can try n=1, which has a sequence of one element and is properly ordered. Next, we can look at n=2 and we start to see what our inductive step is going to look like. With n=2, either team 1 beat team 2, or team 2 beat team 1, giving us two orderings: [1,2] or [2,1] respectively. Inductive Hypothesis The inductive hypothesis is that we have a tournament of n teams with a properly ordered sequence. Essentially, as in any inductive proof, we assume that this situation exists (we have shown it for n=0, 1, and 2, so it must exist for at least 3 values of n). Inductive Step The inductive step is to look at the situation of n+1 teams. Consider our original n teams, with a correctly ordered sequence such as [1, 2, 3, …, n-1, n]. When we add team n+1, let us first prepend it to the front of the sequence, so we have [n+1, 1, 2, 3, …, n-1, n]. If team n+1 beat team 1, then we have a satisfactory ordering, because the subsequence [1, 2, 3, …, n-1, n] must still be correct, and the introduced relationship of team n+1 beating team 1 is correct. If team n+1 lost to team 1, then try moving it one position to the right, to get [1, n+1, 2, 3, …, n-1, n]. If team n+1 beat team 2, then we have a satisfactory ordering, because it lost to team 1, beat team 2, and the subsequence [2, 3, …, n-1, n] must still be correct. If team n+2 lost to team 2, then continue moving it to the right. Continue this process until a correct sequence is found, or until you reach the end of the list. If team n+1 reaches the end of the list, then it must have lost to team n, thus the subsequence [1, 2, 3, …, n-1, n] must be correct and the introduced relationship of team n+1 losing to team n is correct, thus the ordering must be correct. Conclusion Thus, we have proven that it is ALWAYS possible to order a round-robin tournament as such. Generating such an ordering using this method would take O(n²) time. On further consideration of generating this ordering given this data, it became apparent that one could adapt merge-sort to reduce this sequence generation to O(n log n) time. Due to this being a comparison sort, this is absolutely the best we can do (maybe I'll prove this another time!).","tags":"Mathematics","loc":"posts/2013/03/quant-interview-questions-part-2/","title":"Quant Interview Questions (part 2)"},{"url":"posts/2013/03/quant-interview-questions-part-1/","text":"I finally have a free moment (I should be doing work, but I don't have any crazy deadlines to meet today), so I thought I would start posting the questions and answers to one of my recent job interviews, as I found them incredibly interesting and thought provoking. In fact, I would say that their interview process rekindled a love of mathematics that I once had, and lost in high school due to the sheer amount of busywork I was made to do in the \"accelerated track\" at my school. Background I interviewed for a developer position at one of the top quantitative/proprietary trading firms out there, and thus they are looking for the cream of the crop in terms of quantitative reasoning and programming ability. I had four interviews over the phone, of which the first three went well, but I struggled a bit with the fourth. I applied to a handful of other similar firms and have not heard back from any of them. Difficulty of quant programming positions For reference, they all required practically every math-related statistic about me they could before the interviews: my GPA is about 3.55 out of 4 at a top 20 school, I scored an 800 on the math part of the SAT, an 800 on the Math II SAT Subject Test, a 36 on the math part of the ACT, and a 168 out of 170 on the quantitative part of the GRE. All of this was required to submit an application to most firms. I applied to about 20 of these firms and was only approached by 1, so you have been warned! :-) Question The first question was \" given an input stream, how would you collect a k-sized evenly-distributed sample of the input? \"' Constraints and clarification First, to answer these types of questions, you need to fully understand the question and all of the constraints. First, I asked about data types — are these numbers? are they objects? — and the interviewer told me it didn't matter, but we could pretend they are integers for now and move to the general case later. Next, I asked about total size — my intuition first says that I need to know the size of the sample space in order to sample it. He said \"input stream\" is meant to be like an iterator — you just keep calling next() until it stops returning results — so you don't know the total size of the set. I next asked \"okay, so if we start at 0, we always know how many we have at each step , right?\" to which he answered, \"yes.\" At this point, I could restate the problem in a more explicit form: \"Given an input stream of objects, how would you collect a k-sized evenly-distributed sample of the input, so that at any point in time, you have a k-sized evenly distributed sample of the input so far. \" How to start Next, I started typing out the code (we used a Google Docs-esque screen sharing thing to share coe). I knew that, given a constant K, I would always need to iterate through the first k items and add them to the sample. Based on the intuitive concept: If you have k items, then a k-sized sample includes all of the items. Inductive step Next, the hard step: how do I determine whether or not the k+1'th item goes into the sample set or does not? If it does, which item does it replace? It turns out, as long as we keep a running count of how many items we've seen so far, let's say y, then we just need to roll a y-sided die to determine whether or not it goes in the sample. If you roll a 1, 2, …, or a k, then it goes in the sample. If you roll a k, k+1, …, y, it does not. If it goes in the sample, then you roll a k-sided die to determine the index of the element it replaces. This can then be proven to work inductively and thus answers the theoretical question. Programmatically, I implemented this with the Random class of Java. Concrete Example Consider a 6-sized sample of natural numbers. We fill it at first with 1, 2, 3, 4, 5, and 6. Next we get a 7. We roll a 7-sided die (or in the program, we generate a random number from 1 to 7). We get a 5. Since 1 <= 5 <= 6, we put 7 in the sample. We roll a 6-sided die (generate a random number from 1 to 6). We get 4. So now we replace 4 in the set, so our sample looks like {1, 2, 3, 7, 5, 6}. 1, 2, 3, 4, 5, and 6 all had a 100% chance to be in the sample. With the 7th item, they all need to have a 6/7 chance of being in the sample for it to be evenly distributed. The roll for 7 makes sense, since it had a 6/7 chance of being in the sample just based on its roll. Anything currently in the sample, in our case 4, now has a 1/6 chance of not being in the sample. Thus, we get a (6/7)*(1/6)=1/7 chance of not being in the sample REGARDLESS of the result of the roll for 7, thus giving it a 6/7 chance of being in the sample, just like 7 has based on its roll. Cool math, eh?","tags":"Mathematics","loc":"posts/2013/03/quant-interview-questions-part-1/","title":"Quant Interview Questions (part 1)"},{"url":"posts/2012/12/pan-balancing-problem/","text":"I haven't written anything interesting on here in a while, and I've recently been sharpening my algorithm design, algorithm analysis, and general math skills, so here is a short post on one of my favorite kinds of problems! Pan Balancing A pan balance is a balance with two pans suspended from either end of a rod with a pivot in the center. It can be used as a ternary operator to determine whether or not an object is heavier, lighter, or the same weight as another object. A fun mathematical brainteaser is determining the minimum number of weighings to find a certain element in a set, such as the heavy marble in a set of otherwise identical marbles. Optimality Pan balancing can be viewed as a decision tree problem using ternary decision trees. For instance, the classical pan balance problem is: You have a pan balance and 8 marbles. They are all identical, except that one is heavier than the other 7. How can you identify this marble in the fewest number of weighings? Think about the decision tree for this problem, rather than the algorithm. There are 8 possible results that can occur: the first marble is the heavy one, the second marble is the heavy one, …, the eighth marble is the heavy one. This means our decision tree must have at least 8 leaves. We are using a pan balance, which means our decision tree is ternary, thus we have: This equation must be solved for d, which is the number of decisions that must be made to get to a result (i.e. the fewest number of weighings). Thus, we know that there must exist an optimal algorithm that can always solve this problem using only two decisions. Can you come up with that algorithm? Algorithm Sometimes deciding the algorithm can be tricky, but a possible way to start is to draw the tree and put each answer at a leaf, and work your way up. I drew the decision tree with forward knowledge of the answer so that our solution makes more sense, but this could be done with a full ternary tree of depth 2 and produce a valid result as well. Our blank decision tree in Figure 1 has 8 leaves, and we fill in the possible end results in Figure 2. To make this make more sense, let's add some meaning to the branching order. If the decision branches left, then the left pan was heavier. If the decision branches right, then the right pan was heavier. If the decision branches to the middle, then the pans were equal. Our decision tree in Figure 3 shows the next level filled in with statements. Intuitively, this makes sense for results 1, 3, 4, 5, 6, and 8, but why does 1=3 imply 2 and 6=8 imply 7? This is the next step, as we could have switched 2 and 7's positions in this tree and our deduction so far is still correct. What do we know about the first decision from this? We know that based on the first decision, if the weighing branches to the left, the result must be 1, 2, or 3. Similarly, if it branches to the right, it must be 6, 7, or 8. If the two pans are equal, then the answer must be 4 or 5. What comparison can we do to first to guarantee all of these cases for the second level? Figure 4 shows the correct decision for the first level, and thus completes our decision tree. Try picking any of the marbles to be the heavier marble and walking through the tree. You will always get the correct result in two weighings! Adding Complexity Now let's imagine we have 8 coins, and at most one of them is bad, meaning it is either lighter or heavier than the other ones. How many weighings does it take to determine if one of the coins is bad? How many weighings does it take to determine which coin it is, and if it is heavier or lighter? The first question should be simple. There are two possibilities, either one of the coins is bad, or they are all good! Putting 4 on one side and 4 on the other will answer this question! If they are equal, then there is no bad coin, but if they are not, then there must be a bad coin! The second question is harder. What if we use our decision tree from the last problem. We weigh {1,2,3} against {4,5,6} and it branches left. We then weigh 1 against 3 and it branches to the middle. Does this mean 2 is our bad coin? Not necessarily! What if 7 was lighter than the rest? Then {1,2,3} would branch left against {6,7,8} and 1 would branch to the middle against 3. This gives us an answer of 2 but that is clearly not the case! How many different results are there in this case? Well, 1 could be lighter. 1 could also be heavier! The same for 2 through 8. Also, they could all be the same! This gives us 8 results for a light coin, 8 for a heavy coin, and 1 for the same! That is 17 possible results. Our 2-level decision tree can't determine the answer for a question with 17 distinct possibilities! A 3-level ternary decision tree can though (why is this?)! Figure 5 shows a much less intuitive filled-in decision tree for this problem. 1 means 1 is heavier, and 8' means 8 is lighter, while \"x\" means there is no bad coin. Since it has only 3 levels, we know it is an optimal algorithm. Can you derive a different optimal algorithm for this? How many coins at most could we test using only 3 decisions? (i.e. could we do this with 9 coins? 10 coins? More?) Bonus questions: How many decisions does an optimal algorithm take to identify at most 2 bad coins out of 36 and their relative weights? How many decisions does an optimal algorithm take to identify 3 bad coins out of 36 ignoring their weights? Are there situations where it takes n decisions to identify a single bad coin out of m coins ignoring their weights, as well as n decisions to identify a single bad coin out of m coins and determining its relative weight? (i.e. it takes just as many steps to determine the weight too as it does to ignore it)","tags":"Mathematics","loc":"posts/2012/12/pan-balancing-problem/","title":"Pan Balancing Problem"}]};