From 799c5d1e3bbcfb4277aedfb29556283cde855e45 Mon Sep 17 00:00:00 2001 From: Gabby Losch Date: Fri, 6 May 2016 17:09:05 -0400 Subject: [PATCH 1/5] pgconf draft --- .../2016-05-05-pgconf-2016-learnings.markdown | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 _drafts/2016-05-05-pgconf-2016-learnings.markdown diff --git a/_drafts/2016-05-05-pgconf-2016-learnings.markdown b/_drafts/2016-05-05-pgconf-2016-learnings.markdown new file mode 100644 index 0000000..e3dc59f --- /dev/null +++ b/_drafts/2016-05-05-pgconf-2016-learnings.markdown @@ -0,0 +1,90 @@ +--- +layout: post +title: PGConf 2016 Takeaways +author: gabby_losch +summary: +image: http://res.cloudinary.com/wework/image/upload/v1462568584/pgconf.jpg +categories: conferences, industry, databases +--- + +##All hail PostgreSQL! + +Here at WeWork, we rely heavily on PostgreSQL for our data needs. From our member-facing and public technologies to our internal tools, PostgreSQL has us covered. That's why a few of us on the engineering team jumped at the chance to nerd out for a few days at PGConf 2016. How better to spend the first nice days of spring than in a windowless hotel conference room in Downtown Brooklyn?!?! + +Over three days, we delved into some lesser-known features, learned about the latest additions coming to version 9.6 and 9.5 patches, and generally soaked up the shared knowledge of the PG community. + +###PostGIS + +Here at WeWork, we love maps. At the time of this writing, WeWork has just shy of 100 locations in 9 countries, with more popping up seemingly every day. With such a huge footprint, having high-quality location and mapping data is a must. PostGIS provides some massively powerful features for our uses. Here are some highlights that were covered in a great demonstration by Leo Hsu and Regina Obe: + +####Results Within a Specified Distance + +Using [ST_Distance](http://postgis.net/docs/ST_Distance.html) and some basic API calls to Seamless, Yelp, or any other local aggregator/search engine, we can return a list of locations based on distance from a fixed point. If that fixed point is your WeWork office, this query could do everything from finding your next lunch spot, to the nearest ice cream place, to a great neighborhood bakery....I think I might just be hungry. + +``` + SELECT name, other_tags­>'amenity' As type, + ST_Distance(pois.geog,ref.geog) As dist_m + FROM brooklyn_pois AS pois, + (SELECT ST_Point(­73.988697, 40.69384)::geography) As ref(geog) + WHERE other_tags @> 'cuisine=>indian'::hstore + AND ST_DWithin(pois.geog, ref.geog, 1000) + ORDER BY dist_m; +``` + +name | type | dist_m + +‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐ + +Asya | restaurant | 704.31393886 +(1 row) + +####Render 3D content + +When paired with [X3DOM](http://www.x3dom.org), PostGIS can render 3D shapes directly into HTML. Imagine 3D renderings of our buildings on the [Locations Page](http://wework.com/locations) that you can move around. + +####DateTime Ranges + +PostGIS has a whole set of built-in functions to handle datetime calculations and manipulations, including collapsing contiguous ranges into a single range, and consolidating discontinuous or overlapping ranges. This can be particularly useful in ensuring that we maintain clean, understandable usage data around our conference room bookings and building tour schedules. + +``` + SELECT id, + to_daterange( + (ST_Dump( + ST_Simplify(ST_LineMerge(ST_Union(to_linestring(period))),0)) + ).geom) + FROM ( + VALUES + (1,daterange('1970­11­5'::date,'1975­1­1','[)')), + (1,daterange('1980­1­5'::date,'infinity','[)')), + (1,daterange('1975­11­5'::date,'1995­1­1','[)')) + ) x (id, period) + GROUP BY id; +``` + +id | to_daterange + +‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ + +1 | [1970‐11‐05,1975‐01‐01) +1 | [1975‐11‐05,infinity) +(2 rows) + +This is obviously a very simplified example with minimal data. But imagine starting with thousands and thousands of rows -- from such simplified output, we can extrapolate meaningful data such as when our conference rooms are most heavily in use and which ones are most-often booked. This helps us determine how many conference rooms new buildings should have, and what sizes they should be. + +###Don't lock your tables + +As your data grows and becomes more complex, your needs and the way you interact with it will likely also change. Avoid locking your precious production tables by following some general rules (care of [Lukas Fittl](http://twitter.com/LukasFittl) of Product Hunt): + +- Don't remove columns on large tables +- Don't rename columns +- Always index concurrently +- Carefully change the column type +- Carefully add columns with a DEFAULT +- Carefully add NOT NULL columns + (note the theme of using care) + +###PostgreSQL Version 9.6 + +It's gonna be faster. Like, out of the box. You won't have to do anything. ![Wooo!](http://res.cloudinary.com/wework/image/upload/v1462566101/engineering/colbert_celebration.gif) + +Thanks to the team behind PGConf for putting on a great event! From 9c841a47c1fc3220b7399582551b302be2fe3fbe Mon Sep 17 00:00:00 2001 From: Gabby Losch Date: Wed, 11 May 2016 16:26:14 -0700 Subject: [PATCH 2/5] some edits, update gems array in config --- _config.yml | 2 +- .../2016-05-05-pgconf-2016-learnings.markdown | 60 ++++++++++--------- 2 files changed, 34 insertions(+), 28 deletions(-) diff --git a/_config.yml b/_config.yml index 4ef4362..ea404e9 100644 --- a/_config.yml +++ b/_config.yml @@ -6,7 +6,7 @@ description: "" baseurl: "" url: "http://engineering.wework.com" -gems: [jekyll-paginate, jekyll-watch, redcarpet, octopress-autoprefixer] +gems: [jekyll-paginate, jekyll-watch, kramdown, octopress-autoprefixer] # Permalinks permalink: pretty diff --git a/_drafts/2016-05-05-pgconf-2016-learnings.markdown b/_drafts/2016-05-05-pgconf-2016-learnings.markdown index e3dc59f..035216a 100644 --- a/_drafts/2016-05-05-pgconf-2016-learnings.markdown +++ b/_drafts/2016-05-05-pgconf-2016-learnings.markdown @@ -4,7 +4,7 @@ title: PGConf 2016 Takeaways author: gabby_losch summary: image: http://res.cloudinary.com/wework/image/upload/v1462568584/pgconf.jpg -categories: conferences, industry, databases +categories: databases --- ##All hail PostgreSQL! @@ -15,13 +15,13 @@ Over three days, we delved into some lesser-known features, learned about the la ###PostGIS -Here at WeWork, we love maps. At the time of this writing, WeWork has just shy of 100 locations in 9 countries, with more popping up seemingly every day. With such a huge footprint, having high-quality location and mapping data is a must. PostGIS provides some massively powerful features for our uses. Here are some highlights that were covered in a great demonstration by Leo Hsu and Regina Obe: +At the time of this writing, WeWork has just shy of 100 locations in 9 countries, with more popping up seemingly every day. With such a huge footprint, having high-quality location and mapping data is a must. Also, we just really love maps. [PostGIS](http://postgis.net/) provides some massively powerful features for our uses. Here are some highlights that were covered in a great demonstration by Leo Hsu and Regina Obe: ####Results Within a Specified Distance Using [ST_Distance](http://postgis.net/docs/ST_Distance.html) and some basic API calls to Seamless, Yelp, or any other local aggregator/search engine, we can return a list of locations based on distance from a fixed point. If that fixed point is your WeWork office, this query could do everything from finding your next lunch spot, to the nearest ice cream place, to a great neighborhood bakery....I think I might just be hungry. -``` +~~~ sql SELECT name, other_tags­>'amenity' As type, ST_Distance(pois.geog,ref.geog) As dist_m FROM brooklyn_pois AS pois, @@ -29,60 +29,66 @@ Using [ST_Distance](http://postgis.net/docs/ST_Distance.html) and some basic API WHERE other_tags @> 'cuisine=>indian'::hstore AND ST_DWithin(pois.geog, ref.geog, 1000) ORDER BY dist_m; -``` +~~~ -name | type | dist_m +~~~ sql + name | type | dist_m + ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐ -‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐ - -Asya | restaurant | 704.31393886 -(1 row) + Asya | restaurant | 704.31393886 + (1 row) +~~~ ####Render 3D content -When paired with [X3DOM](http://www.x3dom.org), PostGIS can render 3D shapes directly into HTML. Imagine 3D renderings of our buildings on the [Locations Page](http://wework.com/locations) that you can move around. +When paired with [X3DOM](http://www.x3dom.org), PostGIS can render 3D shapes directly into HTML. The results are a bit boxy looking, but if your needs don't require high levels of detail, this is a great solution that doesn't require tons of overhead. ####DateTime Ranges -PostGIS has a whole set of built-in functions to handle datetime calculations and manipulations, including collapsing contiguous ranges into a single range, and consolidating discontinuous or overlapping ranges. This can be particularly useful in ensuring that we maintain clean, understandable usage data around our conference room bookings and building tour schedules. +PostGIS has a whole slew of built-in functions to handle datetime calculations and manipulations, including collapsing contiguous ranges into a single range, and consolidating discontinuous or overlapping ranges. This can be particularly useful in ensuring that we maintain clean, understandable usage data around our conference room bookings and tour schedules for buildings. -``` +~~~ sql SELECT id, to_daterange( - (ST_Dump( - ST_Simplify(ST_LineMerge(ST_Union(to_linestring(period))),0)) - ).geom) + (ST_Dump( + ST_Simplify(ST_LineMerge(ST_Union(to_linestring(period))),0)) + ).geom) FROM ( VALUES - (1,daterange('1970­11­5'::date,'1975­1­1','[)')), - (1,daterange('1980­1­5'::date,'infinity','[)')), - (1,daterange('1975­11­5'::date,'1995­1­1','[)')) - ) x (id, period) + (1,daterange('1970­11­5'::date,'1975­1­1','[)')), + (1,daterange('1980­1­5'::date,'infinity','[)')), + (1,daterange('1975­11­5'::date,'1995­1­1','[)')) + ) x (id, period) GROUP BY id; -``` +~~~ + +~~~ sql -id | to_daterange + id | to_daterange -‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ + ‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ -1 | [1970‐11‐05,1975‐01‐01) -1 | [1975‐11‐05,infinity) -(2 rows) + 1 | [1970‐11‐05,1975‐01‐01) + 1 | [1975‐11‐05,infinity) + (2 rows) +~~~ This is obviously a very simplified example with minimal data. But imagine starting with thousands and thousands of rows -- from such simplified output, we can extrapolate meaningful data such as when our conference rooms are most heavily in use and which ones are most-often booked. This helps us determine how many conference rooms new buildings should have, and what sizes they should be. ###Don't lock your tables -As your data grows and becomes more complex, your needs and the way you interact with it will likely also change. Avoid locking your precious production tables by following some general rules (care of [Lukas Fittl](http://twitter.com/LukasFittl) of Product Hunt): +As your data grows and becomes more complex, your needs and the way you interact with it will likely also change. This is perfectly normal, but typically requires modifying your schema. Avoid locking your precious production tables by following some general rules (care of [Lukas Fittl](http://twitter.com/LukasFittl) of Product Hunt): - Don't remove columns on large tables -- Don't rename columns +- Don't rename columns* - Always index concurrently - Carefully change the column type - Carefully add columns with a DEFAULT - Carefully add NOT NULL columns (note the theme of using care) +*You may be thinking that this suggestion prevents you from making a necessary change to your tables. The key (inadvertent database pun, I swear) is to duplicate any data you'd like to change, make changes to the duplicate, then point your application to the new data. Once that's done, you can delete the old data. This process ensures that the production data isn't locked by the changes. + ###PostgreSQL Version 9.6 It's gonna be faster. Like, out of the box. You won't have to do anything. ![Wooo!](http://res.cloudinary.com/wework/image/upload/v1462566101/engineering/colbert_celebration.gif) From 0df0eae4f45e4688b4acc1f49357d291bac8cab4 Mon Sep 17 00:00:00 2001 From: Gabby Losch Date: Wed, 11 May 2016 16:53:32 -0700 Subject: [PATCH 3/5] move post to posts folder, center gif --- .../2016-05-12-pgconf-2016-learnings.markdown | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) rename _drafts/2016-05-05-pgconf-2016-learnings.markdown => _posts/2016-05-12-pgconf-2016-learnings.markdown (94%) diff --git a/_drafts/2016-05-05-pgconf-2016-learnings.markdown b/_posts/2016-05-12-pgconf-2016-learnings.markdown similarity index 94% rename from _drafts/2016-05-05-pgconf-2016-learnings.markdown rename to _posts/2016-05-12-pgconf-2016-learnings.markdown index 035216a..d73c3e9 100644 --- a/_drafts/2016-05-05-pgconf-2016-learnings.markdown +++ b/_posts/2016-05-12-pgconf-2016-learnings.markdown @@ -33,7 +33,7 @@ Using [ST_Distance](http://postgis.net/docs/ST_Distance.html) and some basic API ~~~ sql name | type | dist_m - ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐+‐‐‐‐‐ + ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ Asya | restaurant | 704.31393886 (1 row) @@ -66,7 +66,7 @@ PostGIS has a whole slew of built-in functions to handle datetime calculations a id | to_daterange - ‐‐‐‐+‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ + ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ 1 | [1970‐11‐05,1975‐01‐01) 1 | [1975‐11‐05,infinity) @@ -91,6 +91,8 @@ As your data grows and becomes more complex, your needs and the way you interact ###PostgreSQL Version 9.6 -It's gonna be faster. Like, out of the box. You won't have to do anything. ![Wooo!](http://res.cloudinary.com/wework/image/upload/v1462566101/engineering/colbert_celebration.gif) +It's gonna be faster. Like, out of the box. You won't have to do anything. + +![Wooo!](http://res.cloudinary.com/wework/image/upload/v1462566101/engineering/colbert_celebration.gif){: style="margin: 0 auto; display: block"} Thanks to the team behind PGConf for putting on a great event! From b241d697495dcf03da6369f4700f5f5e6137c490 Mon Sep 17 00:00:00 2001 From: Gabby Losch Date: Thu, 12 May 2016 12:07:24 -0700 Subject: [PATCH 4/5] better title --- _posts/2016-05-12-pgconf-2016-learnings.markdown | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_posts/2016-05-12-pgconf-2016-learnings.markdown b/_posts/2016-05-12-pgconf-2016-learnings.markdown index d73c3e9..f3d6b4c 100644 --- a/_posts/2016-05-12-pgconf-2016-learnings.markdown +++ b/_posts/2016-05-12-pgconf-2016-learnings.markdown @@ -1,6 +1,6 @@ --- layout: post -title: PGConf 2016 Takeaways +title: The State of PostgreSQL 9.6: PGConf 2016 Takeaways author: gabby_losch summary: image: http://res.cloudinary.com/wework/image/upload/v1462568584/pgconf.jpg From 080613e84955efac2c0aea9c5924128086ef8350 Mon Sep 17 00:00:00 2001 From: dvmlls Date: Fri, 13 May 2016 12:57:29 -0400 Subject: [PATCH 5/5] Update 2016-05-12-pgconf-2016-learnings.markdown --- .../2016-05-12-pgconf-2016-learnings.markdown | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/_posts/2016-05-12-pgconf-2016-learnings.markdown b/_posts/2016-05-12-pgconf-2016-learnings.markdown index f3d6b4c..3d11064 100644 --- a/_posts/2016-05-12-pgconf-2016-learnings.markdown +++ b/_posts/2016-05-12-pgconf-2016-learnings.markdown @@ -9,23 +9,24 @@ categories: databases ##All hail PostgreSQL! -Here at WeWork, we rely heavily on PostgreSQL for our data needs. From our member-facing and public technologies to our internal tools, PostgreSQL has us covered. That's why a few of us on the engineering team jumped at the chance to nerd out for a few days at PGConf 2016. How better to spend the first nice days of spring than in a windowless hotel conference room in Downtown Brooklyn?!?! +Here at WeWork, we rely on PostgreSQL for our data needs. From our member-facing and public technologies to our internal tools, PostgreSQL has us covered. That's why a few of us on the engineering team jumped at the chance to nerd out for a few days at PGConf 2016. How better to spend the first nice days of spring than in a windowless hotel conference room in Downtown Brooklyn? -Over three days, we delved into some lesser-known features, learned about the latest additions coming to version 9.6 and 9.5 patches, and generally soaked up the shared knowledge of the PG community. +Over three days, we delved into lesser-known features, learned about additions coming to version 9.6, and soaked up the shared knowledge of the community. ###PostGIS -At the time of this writing, WeWork has just shy of 100 locations in 9 countries, with more popping up seemingly every day. With such a huge footprint, having high-quality location and mapping data is a must. Also, we just really love maps. [PostGIS](http://postgis.net/) provides some massively powerful features for our uses. Here are some highlights that were covered in a great demonstration by Leo Hsu and Regina Obe: +At the time of writing, WeWork has just shy of 100 locations in 9 countries, with more on the way. Given the huge footprint, having high-quality location and mapping data is a must. Also, we really love maps. Enter [PostGIS](http://postgis.net/). + +Leo Hsu and Regina Obe, authors of PostGIS in Action, gave a great presentation. Here are some of the highlights. ####Results Within a Specified Distance -Using [ST_Distance](http://postgis.net/docs/ST_Distance.html) and some basic API calls to Seamless, Yelp, or any other local aggregator/search engine, we can return a list of locations based on distance from a fixed point. If that fixed point is your WeWork office, this query could do everything from finding your next lunch spot, to the nearest ice cream place, to a great neighborhood bakery....I think I might just be hungry. +Using [ST_Distance](http://postgis.net/docs/ST_Distance.html), we can return a list of locations based on distance from a fixed point. If that fixed point is your WeWork office, and the locations come from API calls to Yelp, then we've just found all possible lunch spots. Or ice cream places around the park. Or bakeries in your neighborhood. I think I might just be hungry. ~~~ sql SELECT name, other_tags­>'amenity' As type, - ST_Distance(pois.geog,ref.geog) As dist_m - FROM brooklyn_pois AS pois, - (SELECT ST_Point(­73.988697, 40.69384)::geography) As ref(geog) + ST_Distance(pois.geog,ref.geog) As dist_m + FROM brooklyn_pois AS pois, (SELECT ST_Point(­73.988697, 40.69384)::geography) As ref(geog) WHERE other_tags @> 'cuisine=>indian'::hstore AND ST_DWithin(pois.geog, ref.geog, 1000) ORDER BY dist_m; @@ -41,7 +42,7 @@ Using [ST_Distance](http://postgis.net/docs/ST_Distance.html) and some basic API ####Render 3D content -When paired with [X3DOM](http://www.x3dom.org), PostGIS can render 3D shapes directly into HTML. The results are a bit boxy looking, but if your needs don't require high levels of detail, this is a great solution that doesn't require tons of overhead. +When paired with [X3DOM](http://www.x3dom.org), PostGIS can render 3D shapes directly into HTML. The results are a bit boxy, but if you don't need high levels of detail, then this is a great solution with little overhead. ####DateTime Ranges @@ -49,10 +50,10 @@ PostGIS has a whole slew of built-in functions to handle datetime calculations a ~~~ sql SELECT id, - to_daterange( - (ST_Dump( - ST_Simplify(ST_LineMerge(ST_Union(to_linestring(period))),0)) - ).geom) + to_daterange( + (ST_Dump( + ST_Simplify(ST_LineMerge(ST_Union(to_linestring(period))),0)) + ).geom) FROM ( VALUES (1,daterange('1970­11­5'::date,'1975­1­1','[)')),