1-TheBigPicture.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Module 1: The Big Picture (2 hours)\n",
    "\n",
    "\"[Trouble for Science](https://xkcd.com/1574/)\" (left) and \"[Machine Learning](https://xkcd.com/1838/)\" (right) from xkcd by Randall Munroe\n",
    "\n",
    "<table>\n",
    "<tr>\n",
    "<td><img alt=\"Trouble for Science\" src=\"https://imgs.xkcd.com/comics/trouble_for_science.png\" height=\"400px\" /></td>\n",
    "<td><img alt=\"Machine Learning\" src=\"https://imgs.xkcd.com/comics/machine_learning.png\" height=\"400px\" /></td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Research Life Cycles\n",
    "\n",
    "<table>\n",
    "<tr> \n",
    "<th>Grants</th>\n",
    "<th>Data</th>\n",
    "</tr>\n",
    "<tr>\n",
    "<td><img src=\"http://www.weber.edu/wsuimages/OSP/NewSite/33.jpg\" alt=\"grantcycle\" style=\"height: 450px;\" /><br />From [Office of Sponsored Projects](http://osp.utah.edu/grant-life-cycle/)</td>\n",
    "<td><img alt=\"data_lifecycle\" src=\"https://library.sydney.edu.au/research/data-management/images/xResearch_data_lifecycle_final_20150827.png.pagespeed.ic.VG8oRep6aH.png\" style=\"height: 450px;\" /><br />From [University of Sydney Library](https://library.sydney.edu.au/research/data-management/research-data-management.html)</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "From [Reproducibility2020: Progress and priorities](http://dx.doi.org/10.12688/f1000research.11334.1)\n",
    "<center><img src=\"https://f1000researchdata.s3.amazonaws.com/manuscripts/12234/79954f9f-f381-48bc-a639-183aaa87c94b_figure1.gif\" alt=\"Many opportunities exist to improve reproducibility across the research life cycle.\" /></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is reproducibility? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This section draws from the Wikipedia article on [Reproducibility](https://en.wikipedia.org/wiki/Reproducibility).\n",
    "\n",
    ">**Reproducibility** is the ability of an entire analysis of an experiment or study to be duplicated, either by the same researcher or by someone else working independently, whereas reproducing an experiment is called replicating it. Reproducibility and replicability together are among the main principles of the scientific method. \n",
    "\n",
    ">The term **reproducible research** refers to the idea that the ultimate product of academic research is the paper along with the laboratory notebooks and full computational environment used to produce the results in the paper such as the code, data, etc. that can be used to reproduce the results and create new work based on the research."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Breaking down \"reproducible research\"\n",
    "This section draws from Victoria Stodden's [Resolving Irreproducibility in Empirical and Computational Research](http://bulletin.imstat.org/2013/11/resolving-irreproducibility-in-empirical-and-computational-research/)\n",
    "\n",
    "* **Empirical reproducibility**\n",
    "* **Computational reproducibility**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Notes\n",
    "* **Empirical reproducibility**: traditional scientific notion of experimental researchers capturing descriptive information about (non-computational) aspects of their research protocols and methods \n",
    "* **Computational reproducibility**: the computational details and other information necessary for others to replicate the findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Another way to define it\n",
    "This section draws from Goodman, et al. [What does research reproducibility mean?](http://stm.sciencemag.org.ezproxy.lib.utah.edu/content/8/341/341ps12.full)\n",
    "\n",
    "* **Methods reproducibility**\n",
    "* **Results reproducibility** (aka replicability)\n",
    "  - **Robustness**\n",
    "  - **Generalizability** (aka transportability)\n",
    "* **Inferential reproducibility**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Notes\n",
    "* **Methods reproducibility**: the ability to implement, as exactly as possible, the experimental and computational procedures, with the same data and tools, to obtain the same results\n",
    "* **Results reproducibility** (aka replicability): the production of corroborating results in a new study, having followed the same experimental methods\n",
    "  - **Robustness**: the stability of experimental conclusions to variations in either baseline assumptions or experimental procedures\n",
    "  - **Generalizability** (aka transportability): the persistence of an effect in settings different from and outside of an experimental framework\n",
    "* **Inferential reproducibility**: the making of knowledge claims of similar strength from a study replication or reanalysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Maybe we need more nuanced terms based on who's doing the experimenting... \n",
    "This section draws from ACM's [Artifact Review and Badging](https://www.acm.org/publications/policies/artifact-review-badging).\n",
    "* **Repeatability**: Same team, same experimental setup\n",
    "* **Replicability**: Different team, same experimental setup\n",
    "* **Reproducibility**: Different team, different experimental setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Is there a crisis?? \n",
    "Let's take a poll."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAkJCQkJCQkICQkJCQkICQgICAgICQgICAkJCQgI\nCAgIChALCAgaCQgIGCEYGhERHx8fCAsgICASIBAeHxIBBQUFBwYHDwgIDxoVEhMaGhsaHx8dGxgd\nHxodFx0XHx8dHRoaGB0dHx0YFhcaFRcYGBUdFx8YGB0VFRUXFRcYFf/AABEIAWgB4AMBIgACEQED\nEQH/xAAdAAADAAIDAQEAAAAAAAAAAAAAAQIDCAUHCQYE/8QAWRAAAQMDAAQJBwUJDgMHBQAAAQAC\nAwQFEQYSITEHCBhBUVSU1fATFCIyYXGRI0KBsdEJFUNSU5Oh0uEWJDM0YnJzdIKSs8HU8TZEwmWD\norK0tcMlY2R1o//EABwBAQACAwEBAQAAAAAAAAAAAAAFBgMEBwECCP/EADcRAQABAwICBAsIAwEA\nAAAAAAABAgMEBRESMQYhQeEVIlFSU2GRobHB0QcTFhcygfDxFGJxJP/aAAwDAQACEQMRAD8A0yQh\nCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQh\nCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQh\nCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQhCAQh\nCAQhCAQhCAQtixxPdIuu2LtNf/oExxPNIuu2LtNw7vQa5oWxvI70i67Yu03Du9PkdaRddsXabh3e\ng1xQtjTxO9Iuu2LtNw7vRyOtIuu2LtNw7vQa5IWx3I60i67Ye1XDu9Lkd6RdcsXabh3eg1yQtjeR\n3pF1yxdpuHd6OR1pF12xdpuHd6DXJC2O5HWkXXbF2q4d3pcjrSLrti7VcO70GuSFsbyOtIuu2LtN\nw7vQeJ3pF12xdpuHd6DXJC2N5HWkXXLF2q4d3o5HWkXXbF2q4d3oNckLY7kdaRddsXarh3elyO9I\nuu2LtVw7vQa5IWxvI60i67Yu03Du9HI60i67Yu03Du9BrkhbG8jrSLrti7TcO70+RzpF12xdquHd\n6DXFC2O5HOkXXbF2m4d3o5HWkXXbF2m4d3oNcULY3kd6RdcsXarh3egcTrSLrli7VcO70GuSFsdy\nOtIuu2LtNw7vQOJzpF12xdquHd6DXFC2O5HWkXXbF2m4d3o5HWkXXbF2m4d3oNcULY7kdaRddsXa\nrh3ejkc6RddsXabh3eg1xQtjuR1pF12xdpuHd6OR1pF12xdpuHd6DXFC2N5HWkXXbF2q4d3o5HWk\nXXbF2q4d3oNckLY3kdaRddsXarh3ejkd6RddsXabh3eg1yQtjeR3pF12xdpuHd6OR3pF12xdpuHd\n6DXJC2N5HekXXbF2q4d3o5HWkXXbF2m4d3oNckLY3kd6RddsXabh3ejkd6RddsXarh3eg1yQtjeR\n1pF12xdquHd6OR1pF12xdquHd6DXJC2N5HekXXbF2m4d3oHE70i67Yu03Du9BrkhbG8jvSLrti7T\ncO70cjvSLrti7VcO70GuSFsbyO9Iuu2LtVw7vRyO9Iuu2LtNw/0CDXJC2N5HekXXbF2m4d3o5Hek\nXXbF2m4d3oNckLY3kd6RddsXabh3elyO9IuuWLtNw7vQa5oWxnI80i67Yu1XDu9HI90i67Yu03Du\n9BrmhbGcj3SLrti7TcP9AlyPdIuuWLtNw/0CDepoVBIJhAymEAIQCYCYCeECASAVIQLCA1UEwgjC\nRCpyQH2oJwmWoIT8fYgnCoBVhGEE6qMKwghBGqjVV6qYCCNVGFRQAggBDgrwjHjCDCranqBUGoIw\nmqIQUE6qWFaMIISIWTH2JOKDGDtVYVAJ4QYwEyFYCMIMQScFlLVJCDEgBWcBUG+AggBUAnhUAgjV\nSIWQhIhBCRVICCEJ6ioBBgcTlZGqizalzoDCMKwnhBjwn4+hVhLCBKSFRCRCBKSrLVJQThThWgoH\nhUAoaVYQUgBCGhBQTUgqwUCJSAVaqWUBhUhoQ44QCSYOxS/2IFhPCpUgkJ4QkHZQMBGEYTwgWEYT\nC/HebrTUkZmq6mmpYRvmqpoqePPRryuDc/Sg/XhNcLo5pdbLgSKC42+tLQS5tHWU9S5o6SyJ5IC5\nooBGE0FAiEsJlJA0kykAgWEBAVIEVJaFaC1BICeE3DwEEIEUkYUkoGVKYKRCCSAqCnUXEaQ6W2y3\n4FfcaCiLtrW1lZT07nDpa2V4Lgg5oBVhfhs11pqyMTUlTTVUR/C0s8U8fu8pE4tz9K5AIEFJCyJF\nBhITCohIIFhNBCMIFhMIxsVAIJwmmhBKkhW4JAIJKQVFTlAipcFZUlBHj7EBCECYFkaoCoIKCpqh\nU1BTQrASATygaQCaAECHj3powoGUGQI1UNVBAsIITATQLCnVTJ2pEoGE8JBMNQcPpppBDbLfWXCf\nJhoqeWqe1uA5/kmFwiYTs1i4AD2uC8vuE7T24X6ulrrhM+RznP8AIw6x8jRwuPoU9NHujiDQ32nG\nTrEkn0b4xVhmuGjV5pKdrnTSUZkjYz1pHUskdV5JoG9xEBGP5S8ukH6LbWzU8rJoJZYJonB8c0Mj\nopY3jc+ORhDmO9oK9GuKfwoSaRWbylWQbhQyCkq3gNaKjLA6Cr1G7GFzNYHd6UUhAAIA83Vuz9zp\n0fmioLvcHhzYayppqaEOBAf5iyZ0sjM+s3Wq2jPTG4cyDaklCA1JAFJGEBAOKQKZSygYVLErB2IK\nQElQCBYQnhMBBjKSt4UgIJwjHSrKnVQdRcavhOk0ds3laUgXCtl8zo3kNcKc6hfNVlr9jy1gGBgj\nWljyCMrzmuVdNUSvnnllnmlcXyTTyPllked7pJHkue73lbpfdEbDNLbbVXMD3RUlVPTzaoJDPPY4\nzFI/A9FmtSluemVo+ctJEH1HBpp1cLFWxV1vnfE9jmmWLWd5CriB9KnqogcSxFpd7RkEapAI9QdB\n9IYrrb6G4wBzYq2miqWsdguj8o3L4Xluwva/WB/mleSy9QOLVYJrdoxZqScOEzKV08jHjD4zWzzV\ngicDta5oqAMfyEHYRSa3pVowgjCRYsmEYQQWox4wrwjHjx42oIIRhUkUCCEmFUUEFCopOQQ4KNVZ\nCkUEFSVahyCSkgpBAwrWNg2qyUFhJAKMILa5WQsbQVlHj/NA0wkE0ASjCAEHx7UB7lQU4VBAJ4ST\nygFgc09OxZ0EIMYaraNyE0DC1m4auKZSXSqlrrTVsts07nST0ksTpaR8zzl0kJjIfS5cSSMPG3YG\n7lzPHS4SLtYKS1y2qpFM+pqamOYmnp59dkcUbmDFRG4N2uduxvX5+JRwm3fSCK8Pu1UKk0slC2DF\nPTU/kxO2rMuynjbr5MMe/O7myg6+0N4lk3lmOut2h83a7L4bbFI6WVv4oqKlrWwHONvk37j9G3Gj\nlkprfTQUVHCyClpoxDDAzOqxjfaSS5xJJJJJJJJySvmuHm/VNt0eutdRyCGqpaUywy6kcmo/ykYz\n5OVrmO2OO8Faw8WLh30kvGktut9wr2TUlQK0yxCiooi/yNvqp4/lIYWvbiWKM7D832oN0ihwTHj9\nqkoE5QrI8fX/AJqUCI+1GE8JBAlbQkFaBhvjYmAhMBBKYSeFTR48e5AnBSQsiRCCQEsePG5WUkHH\n6RWamrqaajq4WT01TG6GaF+dV8bvaDlpzgggggtBGCFqRpnxLZTM91pu0Igc7LILlFIJImn5pqKZ\nrhNz7fJs+3DxpeHPSKzaR1lBb65sNLFFRvjidR0UxaZqSGWT5SaFzzl73HfzrZXgA0gqrlo7aq+t\nkE1VU07pJpQyOPXcJpWA+Tia1jfRY3cBuQdOcDHFKpLZVR112q2XKWBzJIaOKIxUjZmHLXzukJfV\nAOAIbqsHo7dYHC2bWvnHT4Srvo/T2qS1VIpnVM9WyYmnp59dsUcDoxiojcG4L3bsb0cSvhJu2kFN\ndZLrUipdTT0jISKenp9Rssc7pBinjbrZLG787kGwiEFJAIKYKD9KBZSymUsIJITcmUkCCEyjCDHk\neNiaNTagoEsb1WUFBjylhV/ukUGN6gFZHLEQgtu9ZAsUR3rMEA1W0LGCmCgyhPKlpVILCRQqwghu\nUz0K8IQGEBPmQgEYCbU0ElSVkUlqCcpuT1fATAQaofdGzmgsn9cq/wDBhX5vubX8BpB/S2z/AMle\nsn3SCUCksUfO6or349jI6UH/ABFk+5vQYor5JzPqqFn5uGocf8UIO6ONF/wnff6kf8aJaTcSn/jK\n0fzbl/7VXLd7jMRa2it+H/Z8rv7hY/8A6Vo/xLXAaZWfP/aI+l1qrgB8SPig9IQPH+ySvCTggkqS\nqKEEowmUZQMBMJBUAgoBATwkEEkKkOVAIFhJPCRQIpowgoPOfjxf8X3D+gt//oadblcVT/hKx/1R\n/wD6mdaXcdmXW0xuo/EZbmDH/wCso3H3bXH4Ldfivwlmiliaeeha/wDOSSyD9DkHSn3SAfvOxf1m\nv/wqVH3N7+J33+s0P+FVLP8AdHos26yv/Frapn9+CM//ABlfi+5uTjze/wAefSE1tfj2OZWtzjoy\n1BtyQkU0YQS1MqWj6UnuQUjHjxvUxgrIgnCRVqUCwhNHjx0oJKkhWVDignVSKeUigkqCrKxkoIco\ncsjlBKBsCyBSFQQPHj7ENamFTR4+tA2hWFIVZQUmFGU2oMgCYUhPKCkBJMf5IAJoQUCcjIQSgoAh\nATUSODQS4hrWguc5xAa1oGSXE7hjPwQaT/dHLux9xs9EDl1PRVFU72CtnbG0Z6f3ifiOldlfc9bW\n6LR2rqHDHnV0mLD+NFBT08Wfznlh9BWpPGG03F+v9fcIyTTukFPRg5/idMBFA4AgFpcGl+OmYr0P\n4BdETZtH7Vb3t1ZYaZslQ0nOrV1LjUVLSefE0zx/YCDkuFa0urbJd6RvrVFsroGDGflH00ojwOc6\n+qvN/i3XgUWlFjndsb98IYHH8VtZmlc453ACc/Ar1GHj3Lyu4adFX2K/3GhaHRtpqt0lI5pcD5rI\nRPRPa47SfIvj+lp6EHqeUnHx9i+Q4GNNY77ZqC5xkF88LW1LB+CrYR5OriIO0DyrXEdIew86+w8b\n0E48fYhVhCCMI1VeNiQb4KAaFTQkUwfHjmQNCM8yYKBA+1MKHqggCgp4SQGE0ivkOGPTSKxWauuc\nhGtTwuFOw5+VrJPk6SHA24MzmZ34AeeZB52cZi7trdKb5Ow5aK6SnaekUYbSZHs+QXoxwQWs0dis\n9K7AfBbKGOQYI+UbTR+U2c3p6y81eCDReS/X6goXa0nnlW19U8ucD5s1xmrZHPbtDvIsl+kjpXqn\njo2dAGMY6B0BBrl90HtjptHKedoz5pdIHvOPVjmgqYTt6PKOh+IXV33OS7tZc7vRH1qmghqW9B8y\nn8m4Z/GxW5/su6FtVw56Jm82C6W5rdaWele6nGcZq6ctqKUZG7M8MY/tFedXAFpt94L/AEFwfrCC\nOXyNYBrZ8zqAYak6rQS5zWv1gMbTE1B6lKXIje1wa5rg5rgHNe0gtc1wy1zXDYW4I+ITKBBM4SVI\nJTQkCgCknlL2oEfBQnlCBKXKnFJBjDfd4/YkVZUlBJWMrI5Y3BBDisblTkkFNVBQHLI1BQVhQFYQ\nMIcEgqBQJvjx8VecIaFSAiOVeEmhMoApoCA1ABARhCCkihIlALVHjtcNzIIJtHLbKHVM7TFdaiM5\nFNTuHpUDHDYZ3A+l0NJbvedQ40fGZZS+WtGj8rZKrDoqu7RuDo6U+q6CgcNktTvy/c35us7bHrbw\nIcFNx0przFBrMp2u1665Stc+Ona85OSSPLVTtuG5ydpOACQH23Et4Kn3m7suNRGfvbaZWTyFwOrU\n1zfTpaVp2B2Hhj3b9jGgj5UL0MK4PQLRSjs9BT26gi8lTU7dVoOC+R52yTTOAGvM520n6gBjnECw\ntYOPlwWOr6Nl+pI9apt0RjrmNHpS20FzxMABtdG9zyf5Mjz+DW0Cl7AcggEEEEEZBadhBB3jCDzp\n4pfDR+5uudTVhc60V72Cpxlxopx6DK6Ng9ZurgPA2loaRrGMA+iVJPHNGyWJ7JIpGNkjljcHskje\nA5j2PacOaWkbfaFonxteL1JaZJbvaIXPtUjjJUU0bcutb3HJIaNpoSTsPzc4OzBPDcV/jC1Gjz2W\n+4GSpssjtjRl81te85M1KCfSp8kl0ftLm4ORIHoU0IA8e5fisV3pq2niq6OaKppp2B8U8Lw+ORvS\nCNxyCCNhBBBwQv2koJKSsBSQgR3oCeE2DxlAAp5SAQghysFSSgOygsIKQX475daeip5Kqrmip6aB\nhklnmeGRxsHO5x58kbN5JAGSUH6aqdkTHyyPbHHG10kkkjmsZGxg1nve9xw1gaDt9i88ONxw1fuj\nrW0lG4i0UEj/ADcnLTW1GNR9c9h9VmrrBgO0BzicGQhv6+NBxiKjSB77fbjJTWVjsHOWT3NzDkS1\nIB9CmyAWx+wF2TgR8nxTeL5Jd5IrvdonR2mNwkgp5Glrrq9p2DB2ihBG0/Oxgc5aHaXEN4K3UNI+\n/wBXGW1Fwi8jQMdkOjtxc1z6jVO50j2Mx/JjaRskW0QSjYGgNa0Na0AANAa1rQMBoA2AYVYQGF58\ncdbgqdZ7s6508Z+9t2lfKC0HVprg7L6mncdwDna727tjngfwa9CMrhNOdFqO70NRbq+IS01QzVe3\nYHMcNscsTsehM12CD7OdBrNxJeHBk0MOjlzlDJ4QI7VUvOBUQgejb3k7BM0D0OkYbsLRr7ZkLzB4\nc+CW4aLVwjm15KWRxfQ3GNpZHO1pyASD8jVDZludmwjIIJ2M4rfGabUiG0aQzNZUYbFSXaVwDKn5\nrYK97tjKjdiQ7HfOw4ZeG2I/ahNJAIKeEkElBTIQAgQKMePHuTQCgkpFUUighSVTlOsglygqyVJK\nDE4JNCpxU5QS1qyZUMKyOQEZ2LI1Q0qgfHjegbghgTCrKCmhUFCoFBSoJBAQMDx7U8qEwgpqaQX5\nLxcoaSCWpqZY4KenjdNNPK7VZFGwZc5x6MfWgy1tVHDHJNNIyKKJjpZZpXNjjijYNZ8kj3ENawNB\n2+xaOcaDjLy3PylqsUksFuyWVNc3WinuAGwxxbn09EfoLtx1Wktd8txnuH6o0jldRUZkp7LFICyI\n5bLXvjOW1FZt9TWGWs3DYTkgavP8V3i2yXgRXW8NkgtRw+npcmOe5jeHZHpQUR/G2F3zcA6yD47i\n78AtfpNK2d+tR2iOTE9c5vpzap9OCgY4YlmyMF3qt251iNV3oZoVotRWijioLfTsp6aEYaxm1z3H\n1pZXn0pZid5O34LkLbRRU8UcFPFHDBCxsUUMLGxxxRsGGxxxtAa1gA3BfoQMJ4SRnwEDCRTykUCk\njDmua5rXNcC1zXAOa5pGHNc12xzcZ+K0v4znFedCZrto7EXwelLU2eMF0kHO+W3t3yQ7z5PaR83W\nGxm6LSn4yg8yOADhpuGjFSfJE1NumeHVdte8iOQ4DTPTnb5vVaoG0DbqtDgcDHohwbadW+/UMdfb\nZvKwu9GRjsNmpZgAXU9TFkmKYZHOQQQQXAgno/jP8WyG7Ca6WWNkF09KWejGpHBcztL3NzhsFcen\nYHHfgkuOo3BdwgXXRa5Gel14ZI3mCtoKlr2RztieWyU1XCcOZIHa23Y5pzjG3Iep5KML4vgd4SaD\nSO3srqF2CCI6qleR5ajqMZMUuPWad4duI6CCB9mgE0t3SmAgChNAQQQk1UvjeF7hGoNHKB9fXOzv\njpqVhHlqyoxlsMQOwDnLtwH0Ah+7hG03t9ioZK+5TiGFnosaMOmqZsEtp6aLIMsx1Ts2YwSS0Ake\neHGA4arhpPU/KF1NbYXE0tuY/LG8wnqSP4xVY5zsGSG4ydbheFThCumlNybPVa8j3vEFDb6dr3x0\n7ZXgMp6WEZc+Uu1MnBc446ABtrxYOLTFaxDdL3HHPc9ktPRO1ZILc7ILHyYy2etBxt2hp3ZIBAfC\ncWPivvnMN10iiMdP6MtNaHgtlqOdkte3fFBuOpvd87VGx+6MMbWtaxjWsawNaxrQGta1ow1rWtwG\ntAAVhBQBSJQpcUFZTUgqgg4bTTRiiu1HLQXCnZUU04w+N+wtcPVljePSilB3OBBC89uMTwB12jMj\nqiMvrLRJJiGuDfTp9Y+hBXtaMRy7QA4ei7ZjVJ1R6REL8tyo4qiKSCeOOaGZjopYJWNkiljeNV7J\nI3DVewgnYg0c4r/GWltfkrXfJJZ7bsZT1p1pai3DOxkm99RRDo2lo3ZADRvLb6yKeKOeCSOaCVjZ\nYZontkjlieA5kkcjSWvYWkYIJ5lotxouLdJaBLdrMySa1jMlTSelJNbW7zI0nLpqIdO0tHrZALh8\n3xY+Hyp0clbR1ZkqbLLIS+Ha6Shc8+lU0YJ9XJy5m47SME5IeimUifGF+SyXOCsp4aqlljnp542y\nwzRHWZLG8Za5p8Ywc4wv1oGgpZRlAkIKECSKCkUCKnCopFBjeEiFRUFBLlBVOWMoKYFTjuUNKfOg\nyBqtoUMWRqBgKgEBNAAJ4TATQCRVYRhBAyqaVWEg1BS0K46nDQ661j7LQSu+9lDK5lS9hw24V0Ts\nOJI9emY4ODeYnWd6WGkbK8bbhEdYbBM6B+pXV7vMKNwPpRmRpNRUjbkFsAdg8zpIlpRxZ+DI6SXq\nGlkDhQ0487uEjdZv72jc0CBr2+rK95a0bQQC8jOog7U4n/F/bcfJ3y8wh1vadahoJG7Lg9rsec1D\nTvoQ4EBvzyDn0W4k3gZsGMYAGABsAA3YA3BY6SlZExkUTGxxRMbFHGxoayOONoaxjGgYa0NAH0BX\nuQXlLKQKWUF5TChVlBQQ5GUiUDz48fSgFIePYkT48fQgtdB8azgFh0gp5Lhb42R3uCMkauqxt0jj\nbsppycAVGqAGvPsaTjBZ3zlCDy44GuEOu0WuzaqJsga1/m9woJA6PziBr8TU8rHbY5wQSDjLXNG8\nZB9NtFb9TXGjpq+jkEtLVRNnhkAwSxw9V7fmSB2QQdxa4cy1A4+/BS2CRmkdHHqsqJG090Y0HDak\ntxT1mqBhocG6rjsGs2I7TIVl+58cIrmy1OjtRISyUPr7drO9WWMZraZms7cYwHgAfgag/OQbmePA\nVBSPGUwgYCChGUHGaUXumt9JU11XIIaWkhfPNIRnVYwZw1o2vkJwABtJc0DevMrhq4Ra7Sm7OqpB\nJ5Mv83t1AzL/ADeBzsRQxsb/AAlQ52CTvLndAAHf/wB0I4RXF9Lo7TyYa1rK+46p9Z7s+ZUz8HcG\n6zyCPwlOfmr8PEI4Km1E0mkVZHrR0sjqe2MeHar6sN+XrNUjD2sa8Nado1nSbjEEHa/FU4BIrBBH\ncbjEyW9zszh2q9trjkbtghO0edapIc8dJa3Zkv2ACEsIBcZpDpBR0DGyVlRDTse4RsdK8N1nb9Vo\n3nZnPQASdy+S4d9LKi129r6UtbPUzimbKQHGFpjlkdIxp9EvxHgZz63Phav3K71M7WtnqamcMLnN\nFRPLNqOdscW+UcdU7lF5upRjVcERvK2aD0XuanR99VVtRvt6270MrXta5jg5rgHNc0hzXNO0Oa4b\nCMH9KCtfeLlp95F7bTVP+SlcfMpHH+CldtNMTzMcSSPbkfOGNhMLbxcmnIt8cIbVtLu6bkzZr/af\nLCQsVZVRwxvlle2OONpe+R5DWsY0Zc5zjsAws0jg0EkgNAJJJwAAMkknYBhax8N3CSblIaOkcRQR\nO9J7dnnsjTsfs304O4c+/ox5mZdONRxTz7GXRdHu6pf+7o5RznyR/OTk9MeHOpfVxm3Nayjhfktm\nYNeubgtIkBGaeLBOANuQ0n8Vd56NXuGup6eoic0GenhqTDrtdJG2ZjXgPaNo9bG7mWlGAu9OLBop\nURvmuj/kqeWJ1PDHjBqcyNc+c9EYczA6dZ+4D0ojTs69dvTTV17+5cukvR/BxMGLlueGaer11O9+\nkb87OnI58jnC0i44PF+bb/KXyywhtCTrV9BCzAoHuP8AGqdo2CiJIBaB6BwR6J+T3d1VjqaZkrHx\nyNbJHI10ckb2hzHseNV7HtOxzS0kY9qsLmrQ3iX8NDrTWMs1fK77110rWwOectt9dK7Va8H5lM9x\naHcwOq70fSzvsRheZnGb4MTo3epaaJrvMKked297i52Kd7iHU7nu9aRkgc3eTjyZPrrdHiicIrr7\nYIfOJNeutzvMKtx9aUMbmlqXZJJLoNUE87oZTsQdxp4SBTCBEJJlLxtQLCkqkEIIKkqypQQSocqc\npKDG5QVkcsRKDIxJoyqCTQgvCyMQxPH7OdBSsJAKgEAAmR0J4VBBijaedZNVUnjx7UEpK8JY3INA\neP5pWazSFlA12YrTSxxFuNgqqsNqZ3A8/wAk6kH/AHS7+4imhYt+jra57NWpu8xqnEga3mkDnQ0b\nM87cCZ4/rK0n4aro6t0gvNQSXGW51mpvPybZ3xwtHujawfQvUHQyzsobfQUTBhlHR0tK0bc4p4GR\nZOdufR/SUHJvSVAJoIIRhXhLHjx42IADYqwgJgoFhIhXvQQgx48e33pYWQDx48bUavP0/s8fQgxY\nQCshagAIPn9P9Gobtba62z/wdbTyU5dgHyb3D5GZuQRrtlEbhs3xheYWg15qLDe6Sqc18c9sr2+X\ni9HWxBKY6unPNtjErT/OK9XMLzP429lFFpbeGNZqMmnjrW7MBxrYIqiVwH9PJL8Cg9LKeZr2tew6\nzHtD2OHzmOAc1w9mCPisrV8HxermavRmxTlxc7720sLnEkuc+mYKZ7nE7S7WhK+9a1ABKWRrWlzj\nqtaC5zjua1oy4n2YBVgL4bjAXQ0mjV8na4tc22VcbHgkObJUROp2OaRucHyj9CDza4Qb5Pf75WVj\nWvkmuVc7zePYHFssgio4BnZkReRb/ZC9PeDrRaK0WyhtkGPJ0VPHBrAAeUlA1p5jgAazpnSOP9IV\n508U2ytrdLLNG9usyKodWO2ZANDDLVRk+zysMfxC9MwgpCSEHTPGshlNHRPaxxiZUvMjxujc6Mti\n1+gHWf8AADnGdeCPHvW8V7tkNXBLTTsEkMzDG9p6DuIPzXAgEHmIC0+090YmtVbLSS5Ib6UMuMCe\nBxPk5BzB2wgjmLSq1rGNVFf3scpdR6Eapbqsf4c9VVO8x647nANOMYyDsII2EY3EHeDlbTcB2nwu\nlKYahw8+pWgS5wDURbAypaOnmPQegOC1XcF+mirJYXF8MkkTyx8ZfG9zHGORurIzLT6pB+roWjhZ\nlWNXv2SsWu6Lb1WxwT1VRyl27w9cJnnJktlC/wDezSY6udh/jLgcOp43D8ADnJ+du3A63TQTDdmO\nb2cwXO6CaLT3WrZSU+wH05piMtp4QRrSu6TtGBzkjdtI+Lt25l3d+2eTLh4mLouHtHVFPXM+Vz3A\n9oC+71OtKHNoadwNQ8bPKuxrNpYz+MRjJ5geYuC2rggZGxkcbGsZG0MYxoDWsY0ANa0DYAAAvx6M\n2OCgpoqSmZqRRDA/Ge47XyPd86QuySfauRVpwcOnGt7dva5Hr2tXNUyOKeqmP0wRCMJ+N6eVuoN0\nDx5tChcNHX1jGa1TaJRVscANbzWUthrWZO0M1TE8/wBVG9a98QjSs0ekTqFzsQ3Wlkg1PmmqpQam\nneT+N5NlS0f0/uW92l1oZXUNbQyDLKykqaR427W1ML4jgg5B9P6l5ccD9zdQ3+z1GSwwXOjL+Y6n\nnDGzNPsMZePpKD1ZATwmRgpYQLCWFSCEEJePsVFBQYyFJVkJFBiIUELKWqD4+xBiIWNwWZygoGFQ\nChpWRpQWFbVDVQQWFYUAqggsKgpymCgYTCSYQMqWO2j3j60EqY/WHvH1oPJPRk+cXOkMxHy9dAZX\nOIA+VqGmRxJ2AekV6rP0nt+T+/qLef8Am4P115QeQdS1hjf69PUGN386GXVd+lpWwrwOYBR2fnTi\n7bRvutHR3o/Rq8V8VfDw7dm/NuydJ7f16h7VB+ug6UW/r1F2qD9fctJdnQPglgdA+CjfDlXm+9Zf\nwBb9LPs727f7qLf16i7VB+umNJ7f16h7VB+utJmtHQPgEwB0D7E8OVeb7z8AW/Sz7O9uz+6e39eo\nu1Qfro/dPb+vUXaoP11pQQOgJgDoHw/QvfDlXm+95+Abfpp9ne3YGk9v69Rdqg/XR+6e39eou1Qf\nrrSjA9nw8e1PA9nj6E8OVeb7z8A2/Sz7O9ut+6e39eou1Qfrp/unt/XqLtUH660paB0D4BAA6PqT\nw5V5vvPwDb9LPs726p0nt/XqLtUH66X7p7f16i7VB+utKy0eAk1o6PqTw5V5nvPwDb9LPs726p0n\nt/XqHtUH660J4980MmkwlglilbJbqQufFIyRuu108ZGswkZ1WM+IX0JA6Aul+GSQGvwMehDG3Z7c\nu+py3MHUqsm5wTTshNe6L0aXjRei5v17ctvm3w4lchdodaM5Jaa9u3oFxq8D3YI+C7lHj/NdPcTK\nndHodZw4YLhWy7edslwqnMPu1dVdwlSynBdPcc+ZzNDbwWnBIoWE/wAl9yo2uHu1Sfiu4guoOORS\nul0PvLW72so5T/NhuFJI/wD8LSg1R4hTWfupa97mtEdurXguIaNZ3ko9558SFegQr4PysP5xn2+x\neb/FEqAzSBjTj5Wkqox7SGtlx/8AyW5TSqN0l6W3NHyosU24qiYiee3b3JbB02Mm3xzVs7W8/g/K\nxfnGfajz+H8tF+cZ9q6rDvHuQq7+ZN70Me3ubngOnznavn8H5aH84z7V8NwzaMU92oiI5YBWU4dJ\nSPMjBrOIBfA452McGj3EMPMuDBXyfCHpY2hZ5KIh1VI3LRsIhafwrx09A9nQNuWz06yc+r7imxE7\n+vub2naNdoyaaseueKJ6v58XTskbmuLXAtc0lr2nYWuBwWkcxz9SSbpS9znOLnOJJc5xJLnHaXE8\n5yfrSIUrG+3W67G+3XzDRkgZAyQMuOGjpLjzBba8FujlFaaNsUc8Ek0obLUVIez5aQjZq7dkIBwB\n0bdpJzqT8fHsXZXBVpLkChmO0D97OPO0DLoM9I3j2ZHMF83dTuabbm9atxVMfBWelGn3s3HiKKtq\nY5x5f6bMC4wflovzjPtS++MP5aL84z7V1YQlhQ/5kXvQx7e5zvwHT5ztQXCH8tF+cZ9qf3wg/LRf\nnWfauq8Iwn5kXvQx7e554Dp852q2vgyPlod/5Vn2ryc03b5G614iOBFcKoRluNgjqZNQj4Beho5v\nevO6+Zq7jUeSGXVNbL5NvS6ed2oPi4K29F+klzWvvOKjh4du3fn/AEj8/BjF22nfd64uOTn25+P+\n6STgMnoGfgMp5VuRoSyhLKAIScEJEoJJ8fBIplSUCKxlWVCCXFQU3KCdiBtWQeP2rG1VhBlBVKGq\ngUFgqs+PsUKgUF5VAqGpoKymCoQSgZKWVBcpBQeXvGDspoNJr1TncLjPOzZj5Krd51Ds/op2Lsyx\n1nlqaCUYOvDG4/ztUB4/vA/Bc390K0PdDcqK8Mb8jWwCjncB6tXSZLHPdne6mewDZ/yr/o6y4ILt\nrwPpifShOuwHnjfvx7n5/OBRGsWZrs8Udi59Cc2LGbNqrlXHvj+S+9CYUhUT4CqzrppgqE8oLTUe\nOZUgop5UBGsgyNKonx45tyxgqgUeHlGUvZ9YQjwOXQOn9YJrhUvG7yhYPaIgIwfgz9K7o0uuwo6W\nafPpAasftldsZs9+3+yV1rwCaHm+aQW2gLdeGSoE1Vlpc0UdNmep1yPVBjjLRnnlYOdWHRLM+Ncn\n/jnHTzNifu8WOfOfhHzej/AvZHUFgs1G8askFtpBK0fNndC1849vyr5F9dlST43JEKfc4Mu8BfKc\nMNmdX2K8UbAHSVFtrI4Qdxn8g90GcAn+FbGvqR45kIPK3gPu4or7bJ3bGipbC87BqsqgaZ7jnmDZ\nifoW+y0V4wehxsukFyoQ3VhFQ6opfRLWmjqvl6cMJ2OAY/VJHPE/o2be8EGljbvaaSs1gZSwQ1QH\nzauEBs2Rk4ycOA6JWrmH2jafNVFvLpjl4s/GPmn9EvRE1W5/6+uQpyjPg/sXKIjedlhhwGm+k7KC\nLI1XzyZEURP0GR+NojH6d3u6TrKp8sjpZHF8jyXPe7eSef2D7Av0X6rnlqJnVB1phI5j8H0WlhLd\nRg5mAjYuPz7V0/SNMt4Nrq66p5yvmmYFGLb35zPaya/vX66uiljZE+WNzGTtL4nEYD2g4yP0fQ5p\n519DwdaKmsk8tMP3rE7aD+HkG6Pb+D6foHOcdmaVWOOtpnQOw1w9KJ+P4KRo9E7Pm8x9hKwZuu2s\nXIps8/L6mHL1e3Yvxajr8rorKbHlrg5pLXNIc1wOC1wOWuHQcj9ARV0z4pHxSDVkjc5r2neHD6x+\nxSMnYMknYABkkncABtJypzemqnfsS+9NVO/Y7v0D0jFdT5dgTx4bMwbNvzZGjmYcH4Ec236DC+Q4\nM9HJKOKSSbZLUamY+eJjNYta487/AEzn3BfXlcp1SmzTlVxY/Tu55m02qb9UWuQ8fYmUJKParhdO\nrsKK3V9Wf+XpJ5WgnGtI2N3km53bZCwfStKeAKyOr9JLLTNGQ6400rwRn5GmkFTPs/oYZF3xxxdL\nRBQQ2uN3yta8TTAYyykp3ZaHbcjWnDcf1eRfg+56aHOnulbeJG/I0FOaWBxbvrKz1nMcdmW0zJQf\n6yz6e3dANPqxtOm9VHXcnf8AaOqPmq2sXorvcMdjeMoQUlekQCkU0igklGUyEggRUkpkpFBJUKnK\nECJWN5VOKiXcfZjx+lBTVQCxtKyhBYKYUhUCgsJqGq0DBVAqMKgUDJSIQUHx450EOUq3BYig+L4c\ntA49IbNV21xDZntE1JKcfI1sHpQPJIOqwnLT/JmevNKgnqLXXOEjHRz0s0kE8D9jmvjcY54XjmOQ\n4fQvWJaqcdXgTdVB+kNsiLqiNn/1Oljbl08UYwK+MDa6VrAA4c4a07NU63zVTFUcMslq7VZriuid\npjrh1va66OoiZNEdZj26wPRzFp6HA5+C/VtXSeg2lT6F+q7L6d59NnO07vKMzs1sfH6u5LfWRzxt\nlhe2RjtzmnZ7iPmu9iqGdg1Y1X+rtega9a1SzETO1cc4+cP0DxuVHx+xQUyVoLCoFPKxtKoBBQQk\nAllBlb48fSqChqpAFyZKROz6NpPQN5zzLrHhF04a4OpaR2QctlnbuI3GOI87d+Tz83t2sXFryK+G\nlFatq1jTLM3Lk9fZHbMuH4UdJRVzCGJ2YIMgEHZJIdjn+1vMPp6VtvxDODI0FvlvlTHq1VzaI6QO\nGHRWxpDvKbRlpklaD/NghPzl0DxVeBWTSSu85qmPZZ6ORpqpNrfO5W6rm0EL9+SCC4j1Wu5i9q9F\noo2sa1jGtaxjQ1jGgNaxrQA1jWjY1oAGz2BXGzaps0RRTyhw/OzLmbfqv3OcnlLWSJSWVqKymoCe\nUGtHHx4NDcLfHe6WPWqrWxzKoNA1pbY4l7n7AS4xyku3j0ZpzzBa4cWjhFbaK401S/VoK4tZI5xO\nrTVA2Q1HQ1m3Vd7C0/MXpHNG1zS1zWua4FrmuaHNc1ww5rmnY5uCRj2rzt41PAtJo7WmqpGPfZ6y\nRxgfgu8ymcS40MrugDOqTvaOcsK1c7DtZ2PVj3Y3iqNmWzdqtVxXT2NuQfA8bUZWsfF74a2QMitd\n2k1YmasdJXPOyFo2Mp6lx3RAYw7mxg7BluzDXAgEEEEAhwOQQRkEEbCNy/P+taHkaTkTbuR4vZPZ\nMfznC44uXRkU8VLpDTKx1EFRUvdFKYTM97Z9RxjLZHlzS54Gq31gObavz6H2N9dUNiGWsHpzSY9S\nMHmz88nYPf7F3nXUzJo3xSDWZIwxuHS1wxsPMVx+jlggoWPZAHem7Xe95Dnuxsa0uAHogZ5uc9Kl\nKOkv/lmmY8flH1W2nXpjHmjbxuUORoKaOGNkUTQyOMarGjmA6ek5+srOoBVhVKuqap4quau1TNU7\ny+W0w0Mjr5optfyLh6E7mtBMsY9XGdgkB2Z6Dz4CzUOhVFDURVETXtMQ2Rlxcxz8YbKdbaH7/wBB\n5l9ESkCt2NTyotxbiudojZs/51+KIt8U7cmXKalqCo9qqBXG6TXunt9LNWVTxHTwMMj3Hedwaxg+\ndIXEADnLgnfr1TUNPJVVc0cEETcvlkOB7GtA2veeYAEnZjK014d+FeW+ziKIPht0DiYICfTlftHn\nNRjYZME4G3VBO/JJtHRro3e1a/FVUbW45z8o9fwaGdm049HV+rsfNaa36rv92fPqPfPWTsgpaZhL\ny1rnCKlpYgd52tHNkucedekHANoBHo7ZaS3DVdOGmetlbjE1dPh07sgDWYMNYD+LAxdDcSTgRdTe\nT0jucZbNIw/eqlkbtijkbg3CVpGRIWEhnse523Wbq7ZLvdq1Taoi3RG0RG0KhVVNU7yefsTyoygH\n/dfb5UUJIKAKRKRKRKAckU1JQS4qXBUVLigxkKX7sexUVCBtWQLGwrIEFNCoJNCoIGFQKkKgPegp\nAQ1MIEUFUkSgxucpCCgH7EDwgBAVAINS+MvxYDO6W66OxNEriZKq0NwxshO181vz6LX52mPYDk6u\nNjTqZabvV26Z7W68T2PMc1PKwjD43Fr45ongFrwQRzEbdy9aQF13wucC9l0jbrV1N5OrDQ1lxpC2\nGrYB6oe/VLJ4/Y5rt5xhfNVNNcbVR1Mlm9cs1xXbnaYaQ6P8IVLNhs4NPIcDJy6In2OG1v0j6V9j\nSzskaHxvY9v4zHB4+LSsXCPxRr7Ql0lsfBdqfe1rHNpaxo2+tTzO8m/Zj1ZHE7dgXR95s1ztUupV\nU1dQTZIAqIZ6Zxxv1S8DWb7sqIv6Nbrne3Oy64HTnKsxw5FMV+vlLvjCprV0bSacXGPH74Lh0SNZ\nJn3ueCf0rko+EyuA2iB3tMZH/lcFoVaLejlMLFb6dYFUeNTVHs+ruEe7wEEePHjauoDwm134tOP+\n7d+uvxVXCDcn7BM1g/kRRj9JbkLyNFvz2w9r6c6fTHVFU/tH1d3DYCTsHOTsAHSSvnb7prQ02wyi\nZ4/BwYec+1/qD4ldOvrK6ukbFrVNVI8+hEzykznO6GRtySfcF2jwf8WjSi6FjnUX3up3YJqbo7zb\nAyN1Lg1LnYJ/BgbN4yt2zolETvcndB5vTu7XHDjUbeuev3f2+D0s04qa3LB8jAfwTCcu/pH73/oH\nsXaHF04uNdf3xVteJaGz+i8TOGrUXBp26lExw2RkD+EIx6Q1dfbjZPgf4rVks7mVFZm71rNVzX1U\nbW0cLxn0oaLJDnbd73Seq0gNK78A+r9A3KYt2qLVPDRG0KRlZl/LuTcvVTMuL0YsNJbaWGhoYI6a\nlp2eThgjGGtbkkkkkl8hcSSSSSXOJJJXI4VE7EFZGsxkKS1ZCEYQYwEyFSCEGJy47SSx0twppqOs\ngZUUtQwxzQyjLXt3g5G1rg4AgggggEYIXJlGEHnvxieLnXWJ0tbb2y11n9KQyNGtUW9o26lYxoy+\nIA/wgGPROtqbNb5Dgt4ZrlZg2HIrKIf8pO4jyY//ABptroNvNhzdp2ZOV6bYXRPC/wAV+yXhz6ik\nzaK1+XOkpWNdSSvPzpqLIDXZ52Oj9Yk6xWvlYljLtzav0xVTPlZLd2u1VxUTtL5LQjhpslyDW+ci\ninOAaeuLYfSJxhk5PkZP7wO7YF2MzDgHNILSMhwOsHA7iCNhC1B0/wCLRpPbC9zKL75U7ckVFsd5\nwSMnfSkCpDsAfMI27yuubdfrtapDHDUV9BIx2XwiSenId/8AcgJAJ94VBz/s7x7kzVi3Jp9U9cfX\n4pizrVcRtcjd6CYVALSm28PWkcOw1rJhswJ6Wmdu/lNja4/ErmG8ZO+/k7afaaaX/KfCr1f2ealE\n+LXRP7z9G7Gs2J5xLbzx/shrVp/NxkL8RgC3sP4zaVxI/vyuH6FwF24btI58g3GSJp+bTRQU5Huf\nFGH/APiX1b+zvUKp2rroiP3n5PKtasRyiW7VyroaaMy1E0UETdplnkZCwe98hAHxXT/CDxh7VRh0\ndAHXGoGQHNzFSsI2ZdM4a0u/5rSDj1gtYaCgu96nxDFcrpUbG5YyprZACdms4axa3bzkc67r4OOK\nNfq5zZLm+G0U+8tkc2qrHDYfQp4HajNmfWkaRjcVZdO+z7Cx5ivJqmufJyj6++Ghf1m5XG1EbOoN\nNtNbpfqlhqZHzOc8Mp6OBjhEx8h1Wx09OzJdISQM+k47NpWz3Fl4rpidFddI4m+UaWyUtndhwYRt\nZLccbHOztEe3m1udo714H+BWyaONDqKn8rVlpa+41erNVuDvWbG4NDaeP2Na3OBnWXY4V8s2bdmi\nKLcRERyiOpEVV1VzvVKSPHsUK8IWR8seEwFWEEIJTwgp4QQ5LCvCkoJKRCoqUElRhWVDkGN5WNyZ\nUEoMrFYUNVtQZAqCkKggpUApCoFBQTCkKggaklUpKDE9QFkesZCCgsjfHsUMCyNQIePHwVYRhWEE\nlY6qnjlaY5WMlYd7JWNkafex4IKylLCD4S78DGjFUSZbFagXbS6GkjpXE85LqXUJO1fMz8V/Q52T\n96XNJOfQuNzA28wBqSAF3I1NB0uziuaHDfa5Xex1yuX/AE1AXOWjgC0TpseTsdC7HPU+WrPj53I/\nK7NwjCDjrDYqOiYI6OkpKOMbBHSU0FKz+7CxoXIlAQgSMoTCBYQjnQfH7ECKWUyUkAVJVZSIQSSk\nEJDKCighMFBQGFx97sVHWs8nW0lJVx7jHV08NSzHQWTMcP8AZcgEYQdaXjgE0Uqc+UsdC3O/zYTU\nZ+jzSRgC4OTiu6HndbJG+xtxuX/VUErulLCDpyn4sGhzSCbU52MH0rjcyPgKkAhfT2fgX0YpcGKw\n2okbQZ6VlWQeY5q9c5X3gCoIMdHTRxNEcTGRMG6OJjY2D3MYAAspTSKBAJoSKBJePHwQ44Q1A0IQ\nUEkp4TKCfH1IJSKZHj6kiglSVRCkoIIUO8farKnCDA9Yis7gsbggsK271jyqBQZW+PYqBUNPj9qo\nIMjUOSaU8IALIxQFbdyClLlQScEGJwWPnWZwUFqBsCyYSaFYQMJgJApgIDCeEJdCBkJhTlNAwgFB\nSQCYSBTQBQhJAwhwSJQgSMIATCBEJKiUkCcEgmUggaChNBJQAmUggMqkBPKBEKgFOVQQCEZQCgSS\nCUIEQk1UhuPHQgAEJ5SQGFKopZQIpKvH2qSggqSrcoQSVCsqCgxPKhZHLGQgTCqaVoqOOBpD1Oxd\nmuH+vTHHC0i6lYuzXD/XoN7GlWCtEuWHpF1KxdluHeCY442kXUrF2W4d4IN8QEwFodyyNI+pWHst\nw7wT5ZGkfUrD2W4d4IN8MK2laGcsjSPqVh7LcO8ExxydI+pWHstw7wQb7bMePqSC0LHHL0j6lYey\n3DvBHLM0j6lYey3DvBBvphSAtDuWZpH1KwdluHeCOWZpH1KwdluHeCDfIBUB/stCxxzNI+pWHstw\n7wRyzdI+pWDstw7wQb6Zwm0rQo8c3SPqVh7LcO8EcszSPqVg7LcO8EG+yS0K5ZuknUrD2W494I5Z\nmkfUrD2W4d4IN8yqC0J5ZekfUrD2W4d4J8szSPqVh7LcO8EG+oKa0I5ZekfUrD2W4d4KuWbpJ1Kw\n9luHeCDfUIytCeWbpJ1Kw9luHeCfLN0j6lYOy3DvBBvqmtCeWdpJ1KwdluPeCOWdpJ1KwdluHeCD\nfRMrQrlnaSdSsPZbj3gjlnaSdSsPZbj3gg31x+hC0K5Z2knUrB2W494I5Z2knUrB2W4d4IN9UZWh\nPLN0j6lYOy3HvBHLN0k6lYOy3DvBBvqkAVoZyzdI+pWDstx7wQOObpJ1Kw9luPeCDfQBGFoXyzdJ\nOpWDstw7wRyzdJOpWDstx7wQb6Y/2RhaF8s3SPqVg7LcO8EuWZpH1Kw9luHeCDfVC0K5ZuknUrD2\nW4d4I5ZukfUrD2W4d4IN9kErQnlm6SdSsHZbj3gg8czSPqVh7LcO8EG+uUi5aF8szSPqVh7Lce8E\nuWXpH1Kw9luHeCDfQlGVoXyy9I+pWHstw7wRyy9I+pWHstw7wQb6ZTC0KHHL0j6lYey3DvBPlmaR\n9SsPZbh3gg31BRhaFcszSPqVg7LcO8Ecs3STqVh7LcO8EG+pU6y0M5ZuknUrB2W4d4JcszSPqVh7\nLcO8EG+gUlaG8svSPqVh7LcO8EcsvSPqVh7LcO8EG+DlGFolyytI+pWHstw7wS5ZOkfUrD2W4d4I\nN7SpK0T5ZGkfUrD2W4d4KTxxtIupWLstw7wQb0uWNzVoyeOJpF1KxdmuH+vS5YWkXUrF2a4d4INc\n0IQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQg\nEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQg\nEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQg\nEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQg\nEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQgEIQg/9k=\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"600\"\n",
       "            height=\"350\"\n",
       "            src=\"https://www.youtube.com/embed/j7K3s_vi_1Y\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x7efff8273128>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "YouTubeVideo(\"j7K3s_vi_1Y\", width=600, height=350)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Answer: Depends on who you ask\n",
    "From Nature News' [1,500 scientists lift the lid on reproducibility](http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970)\n",
    "<center><img src=\"http://www.nature.com/polopoly_fs/7.36716.1469695923!/image/reproducibility-graphic-online1.jpeg_gen/derivatives/landscape_630/reproducibility-graphic-online1.jpeg\" /></center>\n",
    "\n",
    "Side note: the Wikipedia article on the [replication crisis](https://en.wikipedia.org/wiki/Replication_crisis) does a decent job pulling together the current state of things."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "From Nature News' [1,500 scientists lift the lid on reproducibility](http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970)\n",
    "<table>\n",
    "<tr>\n",
    "<td><img src=\"http://www.nature.com/polopoly_fs/7.36719.1464174488!/image/reproducibility-graphic-online4.jpg_gen/derivatives/landscape_630/reproducibility-graphic-online4.jpg\" width=\"500px\" /></td>\n",
    "<td><img src=\"http://www.nature.com/polopoly_fs/7.36727.1464174506!/image/reproducibility-graphic-online5.jpg_gen/derivatives/landscape_630/reproducibility-graphic-online5.jpg\" width=\"500px\" /></td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example: Estimating the Reproducibility of Psychological Science \n",
    "\n",
    "Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. Doi: [10.1126/science.aac4716](https://login.ezproxy.lib.utah.edu/login?url=http://dx.doi.org/10.1126/science.aac4716)\n",
    "* Over 270 contributors worked on 100 replication studies based on studies from 3 psychology journals\n",
    "* 97% of original studies reported statistically significant results\n",
    "* 36% of replication studies reported statistically significant results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "**Abstract**: Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects (Mr = .197, SD = .257) were half the magnitude of original effects (Mr = .403, SD = .188), representing a substantial decline. Ninety-seven percent of original studies had significant results (p < .05). Thirty-six percent of replications had significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and, if no bias in original results is assumed, combining original and replication results left 68% with significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Additional examples of irreproducible research\n",
    "\n",
    "* Cold fusion (see [1](https://en.wikipedia.org/wiki/Cold_fusion), [2](https://en.wikipedia.org/wiki/University_of_Utah_College_of_Science#History), [3](http://search.ebscohost.com.ezproxy.lib.utah.edu/login.aspx?direct=true&db=aph&bquery=JN+%26quot%3bNew+Scientist%26quot%3b+AND+DT+20030322+AND+%26quot%3bLost+in+Limbo%26quot%3b&type=0&site=ehost-live)) --in 1989 two chemists claimed to have discovered cold fusion and a team of physicists couldn't replicate their findings. \n",
    "\n",
    "* [Retraction Watch](http://retractionwatch.com/) --blog that keeps track of \"retractions as a window into the scientific process\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Some good news: [Cancer studies pass reproducibility test](http://www.sciencemag.org/news/2017/06/cancer-studies-pass-reproducibility-test)\n",
    "\n",
    "**[Reproducibility Project: Cancer Biology](https://osf.io/e81xl/)**: a collaboration between Science Exchange and the Center for Open Science\n",
    "* Each paper has a Registered Report (peer reviewed and published prior to data collection)\n",
    "* Results are then published as a Replication Study.\n",
    "* The aim of the project is to provide evidence about reproducibility and to identify the factors that influence reproducibility more generally. \n",
    "* To date, 4 of the studies have reproduced important parts of the original papers, 1 study has not, and 2 studies could not be interpreted. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What is research misconduct?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Defined under Section IV.H of the University's [Policy on Research Misconduct](http://regulations.utah.edu/research/7-001.php):**\n",
    "\n",
    "\"Misconduct\" or \"Misconduct in Research\" means fabrication, falsification, plagiarism, or other practices that seriously deviate from those practices that are commonly accepted within the research community for proposing, conducting, or reporting research. It does not include honest error or honest difference in interpretations or judgments of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "1. Fabrication is making up results and recording or reporting the fabricated results.\n",
    "2. Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.\n",
    "3. Plagiarism is the appropriation of another person's ideas, processes, results, or words without giving appropriate credit and without specific approval, including those obtained through confidential review of others' research proposals and manuscripts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Take a moment to read the whole policy \n",
    "http://regulations.utah.edu/research/7-001.php\n",
    "\n",
    "* Is there anything you learned that you didn't know before? \n",
    "* Anything surprising? \n",
    "* Did it clarify any questions for you? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Recent example of research misconduct\n",
    "\n",
    "From Retraction Watch, [Volunteer researcher faked weeks’ worth of data](http://retractionwatch.com/2017/07/14/research-volunteer-faked-weeks-worth-data/)\n",
    "> A volunteer researcher at Florida Atlantic University fabricated the results of mouse experiments over a 14-day period in June, 2016. According to the [Office of Research Integrity], Alec Mirchandani made up the results of behavioral experiments to make it seem as if he had done the work, and falsified animal transfer logs, which affected research supported by the National Institute of Mental Health.\n",
    "\n",
    "Consequences? \n",
    "> [In a Voluntary Settlement Agreement with ORI], for the next two years, Mirchandani must be supervised if he conducts any research supported by the U.S. Public Health Service (PHS), which includes the National Institutes of Health. He’s also not eligible to serve on PHS peer review committees during that time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Guidelines and Recommendations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### National Academies of Sciences, Engineering, and Medicine \n",
    "\n",
    "**[Fostering Integrity in Research](https://doi.org/10.17226/21896)** identifies best practices in research and recommends practical options for discouraging and addressing research misconduct and detrimental research practices.\n",
    "\n",
    "<center><img src=\"https://www.nap.edu/cover/21896/450\" width=\"400px\" /></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### NSF\n",
    "**[A Framework for Ongoing and Future National Science Foundation Activities to Improve Reproducibility, Replicability, and Robustness in Funded Research](https://www.nsf.gov/attachments/134722/public/Reproducibility_NSFPlanforOMB_Dec31_2014.pdf)**\n",
    "\n",
    "NSF is addressing issues of reproducibility, replicability, and robustness of research at both the agency-wide level and within all directorates as appropriate. \n",
    "\n",
    "1. scientific milieu and question formulation\n",
    "2. data collection, sharing, and curation\n",
    "3. instrumentation, models and interpretation of findings\n",
    "4. research outputs\n",
    "5. questions of science reproducibility, replicability, and robustness"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### NIH \n",
    "**[Rigor and Reproducibility](https://grants.nih.gov/reproducibility/index.htm)**\n",
    "\n",
    "<center><a href=\"https://grants.nih.gov/reproducibility/documents/grant-guideline.pdf\"><img src=\"https://grants.nih.gov/reproducibility/documents/grant-guideline.gif\" /></a></center>\n",
    "\n",
    "One page guide: http://grants.nih.gov/grants/RigorandReproducibilityChart508.pdf\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**[Principles and Guidelines for Reporting Preclinical Research](https://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research)**\n",
    "\n",
    "NIH held a joint workshop in June 2014 with the Nature Publishing Group and Science on the issue of reproducibility and rigor of research findings, with journal editors representing over 30 basic/preclinical science journals in which NIH-funded investigators have most often published. The journal editors came to consensus on a set of principles to facilitate these goals, which a considerable number of journals have agreed to [endorse](https://www.nih.gov/sites/default/files/research-training/initiatives/reproducibility/rigor-reproducibility-endorsements.pdf).\n",
    "1. Rigorous statistical analysis\n",
    "2. Transparency in reporting: standards, replicates, statistics, randomization, blinding, sample-size estimation, inclusion/exclusion criteria\n",
    "3. Data and material sharing\n",
    "4. Considerations of refutations\n",
    "5. Consider establishing best practice guidelines for image-based data and description of biological material with enough information to uniquely identify the reagents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Reproducibility Enhancement Principles (REP)\n",
    "This section draws from Victoria Stodden, et. al [Enhancing reproducibility for computational methods](http://science.sciencemag.org.ezproxy.lib.utah.edu/content/354/6317/1240.full)\n",
    "\n",
    "* Share data, software, workflows, and details of the computational environment that generate published findings in open trusted repositories.\n",
    "* Persistent links should appear in the published article and include a permanent identifier for data, code, and digital artifacts upon which the results depend. \n",
    "* To enable credit for shared digital scholarly objects, citation should be standard practice.\n",
    "* To facilitate reuse, adequately document digital scholarly artifacts.\n",
    "* Use Open Licensing when publishing digital scholarly objects.\n",
    "* Journals should conduct a reproducibility check as part of the publication process and should enact the Transparency and Openness Promotion (TOP) standards at level 2 or 3.\n",
    "* To better enable reproducibility across the scientific enterprise, funding agencies should instigate new research programs and pilot studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "From Brian Nosek, et al. [Promoting an open research culture](http://science.sciencemag.org.ezproxy.lib.utah.edu/content/348/6242/1422.full)\n",
    "<center><img src=\"https://d2ufo47lrtsv5s.cloudfront.net/content/sci/348/6242/1422/F1.large.jpg\" width=\"850px\"/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Practical tips to get started\n",
    "This section draws from Christopher Gandrud's [Getting Started with Reproducible Research](https://www.r-bloggers.com/getting-started-with-reproducible-research-a-chapter-from-my-new-book/)\n",
    "\n",
    "* Document everything! How did you gather, analyze, and present the data and results? \n",
    "* Everything is a (text) file. Think \"future-proofing.\" \n",
    "* All files should be human readable. Comment your code frequently. \n",
    "* Explicitly tie your files together. \n",
    "* Have a plan to organize, store, and make your files available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Example Workflow \n",
    "From Christopher Gandrud's [Getting Started with Reproducible Research](https://www.r-bloggers.com/getting-started-with-reproducible-research-a-chapter-from-my-new-book/)\n",
    "<center><img src=\"https://4.bp.blogspot.com/-u3BaetbDxtM/UeR4f8924nI/AAAAAAAAGlU/fnl_KfDSSb8/s1600/Workflow.png\" /></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Additional Reading\n",
    "* [Why Most Published Research Findings Are False](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)\n",
    "* [How to Make More Published Research True](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001747) \n",
    "* [Nature Special: Challenges in irreproducible research](https://login.ezproxy.lib.utah.edu/login?url=http://www.nature.com/news/reproducibility-1.17552)\n",
    "* [Science Isn't Broken](https://fivethirtyeight.com/features/science-isnt-broken/#part1)\n",
    "\n",
    "\n",
    "## Additional Resources\n",
    "* [Reproducibility of Research](http://campusguides.lib.utah.edu/reproducibility)\n",
    "* [CCTS Study Design & Biostatistics Center](http://medicine.utah.edu/ccts/sdbc/)\n",
    "* [Creative Commons Licenses](https://creativecommons.org/licenses/)\n",
    "* [Conference Materials from Research Reproducibility 2016](https://library.med.utah.edu/e-channel/portfolio/research-reproducibility-conference/) \n",
    "\n",
    "## Funding Opportunities\n",
    "* [BD2K](https://commonfund.nih.gov/bd2k)\n",
    "* [Center for Open Science Preregistration Challenge](https://cos.io/prereg/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Exercise\n",
    "\n",
    "Start thinking about [Module 6: Putting It All Together](./6-Synthesis.ipynb)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}