Skip to content

Commit

Permalink
adding pandas extra and requirements
Browse files Browse the repository at this point in the history
  • Loading branch information
@mcburton
mcburton committed May 21, 2019
1 parent c38deba commit df0ea3b
Show file tree
Hide file tree
Showing 12 changed files with 7,794 additions and 0 deletions.
391 changes: 391 additions & 0 deletions day-two/5-apis-and-json.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,391 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Application Programming Interfaces (APIs)\n",
"\n",
"![Diagram of an API, Image from Software AG](https://www.softwareag.com/pl/images/SAG_API_Portal_KD_1_Large_tcm403-160297.png)\n",
"\n",
"* API or application programming interface is not a new term, but it has taken on a new significance with the Web\n",
"* Now we talk about \"Web APIs\" or \"Web Services\" which are APIs that use web technologies and standards ;)\n",
"* If UIs are interaces for the user to access a system, APIs are software interfaces for systems to access other systems (with the help of a Developer).\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What might an API look like?\n",
"\n",
"* Imagine having an API that takes a time zone string and returns the current time for that time zone\n",
"* This API would take a string like \"America/Los_Angeles\" and return \"2019-02-28T20:09:45-07:00\"\n",
"* One design of the API might look like this:\n",
" * http://api.example.com/timezone?tz=America+Los_Angeles\n",
"\n",
"```json\n",
"{\n",
" \"time\": \"2019-02-28T20:09:45-07:00\",\n",
" \"zone\": \"America/Los_Angeles\"\n",
"}\n",
"```\n",
"\n",
"* This API has an *endpoint*, `/timezone` that expects a *query paramater, `tz={Timezone location}`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## APIs Support Multiple Interfaces\n",
"\n",
"* With an API you don't need to build multiple interfaces to applications and platforms\n",
"* You build a generic interface and the platforms conform to the API specification and data formats\n",
"\n",
"![A web page that accesses an API. Image from Express In Action](https://cdn.glitch.com/47219279-662d-49cf-9388-c11e70fac7be%2FScreenshot%202019-02-28%2013.06.03.png?1551377204363)\n",
"\n",
"* By separating the content from the representation we can support multiple modes of access\n",
"\n",
"![A command line application accessing an API. Image from Express In Action](https://cdn.glitch.com/47219279-662d-49cf-9388-c11e70fac7be%2FScreenshot%202019-02-28%2013.06.31.png?1551377204295)\n",
"\n",
"* APIs can use different data formats like XML or proprietary formats\n",
"* These days many web-based APIs, including the LC API, use a data format called JSON"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## What is JSON\n",
"\n",
"* JSON stands for [*JavaScript Object Notation*](https://www.json.org/)\n",
"* Is a text-based format that is valid JavaScript code\n",
"* It is \"self-describing\" and easy to read and understand\n",
"* A lightweight format for storing and transferring data\n",
"* While it uses JavaScript syntax, it is supported by nearly every programming language."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## JSON Data Types\n",
"\n",
"* Strings - Strings must be double quotes. `{\"name\":\"Bob\"}`\n",
"* Numbers - Numbers must be integer or floating point numbers. `5, 5.6`\n",
"* Objects - Values can be JSON objects. `{\"employee\":{ \"name\":\"John\", \"age\":30, \"city\":\"New York\" }}` (look like anything you know?)\n",
"* Arrays - Arrays must be an order list of any values. `{\"employees\":[ \"John\", \"Anna\", \"Peter\" ]}`\n",
"* Boolean - Must be true or false value. `{\"sale\":true}`\n",
"* Null - Values can also be null. `{\"middlename\":null}`\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Working with JSON in Python\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# import the JSON module so we can load and save JSON data\n",
"import json\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reading JSON\n",
"\n",
"* In Python you parse JSON text into Python data structures using the `json.loads()` and `json.load()` functions.\n",
"* "
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'\\n[\\n{\"first\":\"Matt\",\"last\":\"Burton\",\"city\":\"Pittsburgh\"},\\n{\"name\":\"John\", \"age\":30, \"city\":\"New York\"},\\n{\"first\": \"Jessica\", \"city\":\"Chicago\"}\\n]\\n'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# create a string that represents some JSON data\n",
"json_string = \"\"\"\n",
"[\n",
"{\"first\":\"Matt\",\"last\":\"Burton\",\"city\":\"Pittsburgh\"},\n",
"{\"name\":\"John\", \"age\":30, \"city\":\"New York\"},\n",
"{\"first\": \"Jessica\", \"city\":\"Chicago\"}\n",
"]\n",
"\"\"\"\n",
"json_string"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* This is a Python string, but it can be parsed into a list of dictionaries using `json.loads()`"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'city': 'Pittsburgh', 'first': 'Matt', 'last': 'Burton'},\n",
" {'age': 30, 'city': 'New York', 'name': 'John'},\n",
" {'city': 'Chicago', 'first': 'Jessica'}]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"json_data = json.loads(json_string)\n",
"json_data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Now the data had been loaded into Python and we can access it "
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'city': 'Pittsburgh', 'first': 'Matt', 'last': 'Burton'}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# get the first item \n",
"json_data[0]"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'New York'"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# get the city field of the second item\n",
"json_data[1]['city']"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* Parsing JSON can be very picky and cryptic\n",
"* For example, why doesn't this work?"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"ename": "JSONDecodeError",
"evalue": "Expecting property name enclosed in double quotes: line 5 column 22 (char 124)",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mJSONDecodeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-7-9c7bb4442ce1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 7\u001b[0m ]\n\u001b[1;32m 8\u001b[0m \"\"\"\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0mjson\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbad_json_string\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
"\u001b[0;32m~/miniconda3/lib/python3.7/json/__init__.py\u001b[0m in \u001b[0;36mloads\u001b[0;34m(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[1;32m 346\u001b[0m \u001b[0mparse_int\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mparse_float\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32mand\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 347\u001b[0m parse_constant is None and object_pairs_hook is None and not kw):\n\u001b[0;32m--> 348\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_default_decoder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 349\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mcls\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 350\u001b[0m \u001b[0mcls\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mJSONDecoder\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/miniconda3/lib/python3.7/json/decoder.py\u001b[0m in \u001b[0;36mdecode\u001b[0;34m(self, s, _w)\u001b[0m\n\u001b[1;32m 335\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 336\u001b[0m \"\"\"\n\u001b[0;32m--> 337\u001b[0;31m \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0m_w\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 338\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_w\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 339\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m~/miniconda3/lib/python3.7/json/decoder.py\u001b[0m in \u001b[0;36mraw_decode\u001b[0;34m(self, s, idx)\u001b[0m\n\u001b[1;32m 351\u001b[0m \"\"\"\n\u001b[1;32m 352\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 353\u001b[0;31m \u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscan_once\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 354\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 355\u001b[0m \u001b[0;32mraise\u001b[0m \u001b[0mJSONDecodeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Expecting value\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0merr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mJSONDecodeError\u001b[0m: Expecting property name enclosed in double quotes: line 5 column 22 (char 124)"
]
}
],
"source": [
"# create a string that represents some JSON data\n",
"bad_json_string = \"\"\"\n",
"[\n",
"{\"first\":\"Matt\",\"last\":\"Burton\",\"city\":\"Pittsburgh\"},\n",
"{\"name\":\"John\", \"age\":30, \"city\":\"New York\"},\n",
"{\"first\": \"Jessica\", city:\"Chicago\"}\n",
"]\n",
"\"\"\"\n",
"json.loads(bad_json_string)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Reading JSON from files\n",
"\n",
"* If you have a JSON file you can read it from disk using `json.load()`\n",
" * Example file comes from [wikipedia](https://en.wikipedia.org/wiki/JSON)\n",
"* "
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'address': {'city': 'New York',\n",
" 'postalCode': '10021-3100',\n",
" 'state': 'NY',\n",
" 'streetAddress': '21 2nd Street'},\n",
" 'age': 27,\n",
" 'children': [],\n",
" 'firstName': 'John',\n",
" 'isAlive': True,\n",
" 'lastName': 'Smith',\n",
" 'phoneNumbers': [{'number': '212 555-1234', 'type': 'home'},\n",
" {'number': '646 555-4567', 'type': 'office'},\n",
" {'number': '123 456-7890', 'type': 'mobile'}],\n",
" 'spouse': None}"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# open a file handler in read-only mode\n",
"with open(\"test.json\", \"r\") as f:\n",
" # pass the file handler to the json parser\n",
" json_from_file = json.load(f)\n",
" \n",
"json_from_file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* This has been parsed into a JSON dictionary\n",
"* The JSON keys are now Python Dictionary keys"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'John'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"json_from_file['firstName']"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'John Smith'"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"full_name = json_from_file[\"firstName\"] + \" \" + json_from_file[\"lastName\"]\n",
"full_name"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading

0 comments on commit df0ea3b

Please sign in to comment.