Julia is Fast .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Julia is fast: Comparing performance using benchmarking techniques\n",
    "\n",
    "\n",
    "We often use benchmarks to compare languages. In this notebook we will try to explore benchmarking by taking example of simple sum function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sum` function\n",
    "\n",
    "Consider the function `sum(a)` which adds the elements of a sequence supplied to it i.e: \n",
    "$$\n",
    "\\mathrm{sum}(a) = \\sum_{i=1}^n a_i,\n",
    "$$\n",
    "\n",
    "where $n$ is the length of sequence $a$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000000-element Array{Float64,1}:\n",
       " 0.08705364460495324\n",
       " 0.7930448158242029\n",
       " 0.5490600547756832\n",
       " 0.32802035406416397\n",
       " 0.7634176909249002\n",
       " 0.5187306491888561\n",
       " 0.9455408768708209\n",
       " 0.9097727311507622\n",
       " 0.3316496181089985\n",
       " 0.27181569886761303\n",
       " 0.2574728978408143\n",
       " 0.8267245829526333\n",
       " 0.16609820293163913\n",
       " ⋮\n",
       " 0.7945897326470248\n",
       " 0.019332576319549766\n",
       " 0.8991274034971959\n",
       " 0.1541851749469909\n",
       " 0.22306811579255248\n",
       " 0.40685665882655786\n",
       " 0.31814366116506165\n",
       " 0.7929774084109056\n",
       " 0.690977428943341\n",
       " 0.15349026600824645\n",
       " 0.7707953396293554\n",
       " 0.32271611677727896"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Vector to store random numbers between 0-1.\n",
    "a = rand(10^6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500028.72163173417"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This should be nearly $0.5X{10}^6$ since $0.5$ is the mean entry here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmarking in a few ways in a few langugages\n",
    "\n",
    "We can use the `@time` macro to get the time required to execute that function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.000818 seconds (1 allocation: 16 bytes)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "500028.72163173417"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@time sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.001751 seconds (1 allocation: 16 bytes)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "500028.72163173417"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@time sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.000783 seconds (1 allocation: 16 bytes)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "500028.72163173417"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@time sum(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that `@time` may not produce the same result everytime so it is not our best choice for benchmarking!\n",
    "\n",
    "Instead we can use `BenchmarkTools.jl` package available in Julia to easily get more accurate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using Pkg\n",
    "# Pkg.add(\"BenchmarkTools\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "using BenchmarkTools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. `sum` function in C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "c_sum (generic function with 1 method)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using Libdl\n",
    "C_code = \"\"\"\n",
    "#include <stddef.h>\n",
    "double c_sum(size_t n,double *X){\n",
    "    double s = 0.0;\n",
    "    for(size_t i = 0;i<n;i++){\n",
    "        s += *(X+i);\n",
    "    }\n",
    "    return s;\n",
    "}\n",
    "\"\"\"\n",
    "# make a temporary file\n",
    "const Clib = tempname()\n",
    "\n",
    "open(`gcc -fPIC -O3 -msse3 -xc -shared -o $(Clib * \".\" * Libdl.dlext) -`, \"w\") do f\n",
    "    print(f,C_code)\n",
    "end\n",
    "\n",
    "# define Julia function to call the C function:\n",
    "c_sum(X::Array{Float64}) = ccall((\"c_sum\", Clib), Float64, (Csize_t, Ptr{Float64}), length(X),X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500028.7216317387"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Comparing the sum calulated by Julia's sum function and C's sum function\n",
    "c_sum(a) ≈ sum(a) # type \\approx and then <TAB> to get the ≈ symbol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$ \\approx $ is alias for `isapprox` i.e to compare if the supplied values are approximately equal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the difference between the two `sum` functions: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.540197551250458e-9"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_sum(a) - sum(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can benchmark the C code directly from Julia:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: \n",
       "  memory estimate:  0 bytes\n",
       "  allocs estimate:  0\n",
       "  --------------\n",
       "  minimum time:     1.704 ms (0.00% GC)\n",
       "  median time:      1.738 ms (0.00% GC)\n",
       "  mean time:        1.740 ms (0.00% GC)\n",
       "  maximum time:     3.174 ms (0.00% GC)\n",
       "  --------------\n",
       "  samples:          2852\n",
       "  evals/sample:     1"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_bench = @benchmark $c_sum($a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Benchmark tools** execute the same code sevral times to get the accurate result. We can see the summary of benchmarking above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fastest time for C: 1.706173 ms\n"
     ]
    }
   ],
   "source": [
    "println(\"Fastest time for C: $(minimum(c_bench.times)/10^6) ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a dictionary to store fastest time for different scenerios:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "ename": "MethodError",
     "evalue": "MethodError: no method matching Dict(::String, ::Float64)\nClosest candidates are:\n  Dict(::Any) at dict.jl:127",
     "output_type": "error",
     "traceback": [
      "MethodError: no method matching Dict(::String, ::Float64)\nClosest candidates are:\n  Dict(::Any) at dict.jl:127",
      "",
      "Stacktrace:",
      " [1] top-level scope at In[84]:1"
     ]
    }
   ],
   "source": [
    "sum_times = Dict(\"C\",(minimum(c_bench.times)/10^6)) # time in milliseconds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. `sum` function in Python\n",
    "We can use the `pycall` package to use Python's functions in Julia:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#using Pkg\n",
    "#Pkg.add(\"PyCall\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "using PyCall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "a_list = PyCall.array2py(a);@time PyCall.array2py(rand(10^8));@time PyCall.array2py(rand(10^8));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PyObject <built-in function sum>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pysum = pybuiltin(\"sum\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500028.7216317387"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pysum(a_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pysum(a_list) ≈ sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: \n",
       "  memory estimate:  48 bytes\n",
       "  allocs estimate:  3\n",
       "  --------------\n",
       "  minimum time:     8.649 ms (0.00% GC)\n",
       "  median time:      8.803 ms (0.00% GC)\n",
       "  mean time:        8.871 ms (0.00% GC)\n",
       "  maximum time:     14.262 ms (0.00% GC)\n",
       "  --------------\n",
       "  samples:          563\n",
       "  evals/sample:     1"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py_list_bench = @benchmark $pysum($a_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 2 entries:\n",
       "  \"C\"               => 1.70617\n",
       "  \"Python built-in\" => 8.65179"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_times[\"Python built-in\"] = minimum(py_list_bench.times)/1e6\n",
    "sum_times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Python `numpy`\n",
    "### Takes advantage and hardware \"SIMD\", but only works when it work.\n",
    "\n",
    "`numpy` is an optimized C library, callable from Python. It may be installed within Julia as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m\u001b[1m   Updating\u001b[22m\u001b[39m registry at `~/.juliapro/JuliaPro_v1.4.1-1/registries/JuliaPro`\n",
      "\u001b[32m\u001b[1m  Resolving\u001b[22m\u001b[39m package versions...\n",
      "\u001b[32m\u001b[1m   Updating\u001b[22m\u001b[39m `~/.juliapro/JuliaPro_v1.4.1-1/environments/v1.4/Project.toml`\n",
      " \u001b[90m [8f4d0f93]\u001b[39m\u001b[92m + Conda v1.4.1\u001b[39m\n",
      "\u001b[32m\u001b[1m   Updating\u001b[22m\u001b[39m `~/.juliapro/JuliaPro_v1.4.1-1/environments/v1.4/Manifest.toml`\n",
      "\u001b[90m [no changes]\u001b[39m\n"
     ]
    }
   ],
   "source": [
    "Pkg.add(\"Conda\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Conda\n",
    "#Conda.add(\"numpy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: \n",
       "  memory estimate:  48 bytes\n",
       "  allocs estimate:  3\n",
       "  --------------\n",
       "  minimum time:     785.514 μs (0.00% GC)\n",
       "  median time:      1.083 ms (0.00% GC)\n",
       "  mean time:        1.108 ms (0.00% GC)\n",
       "  maximum time:     5.941 ms (0.00% GC)\n",
       "  --------------\n",
       "  samples:          4405\n",
       "  evals/sample:     1"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numpy_sum = pyimport(\"numpy\").\"sum\"\n",
    "a_numpy = PyObject(a)\n",
    "\n",
    "py_numpy_bench = @benchmark $numpy_sum($a_numpy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500028.72163173446"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numpy_sum(a_numpy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numpy_sum(a_numpy) ≈ sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 4 entries:\n",
       "  \"C\"                   => 1.70617\n",
       "  \"Python numpy\"        => 0.785514\n",
       "  \"Python user-defined\" => 56.3724\n",
       "  \"Python built-in\"     => 8.65179"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_times[\"Python numpy\"] = minimum(py_numpy_bench.times)/1e6\n",
    "sum_times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Python, user-defined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PyObject <function py_sum at 0x7f4db0072268>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py\"\"\"\n",
    "def py_sum(a):\n",
    "    sum = 0.0\n",
    "    for i in a:\n",
    "        sum+=i\n",
    "    return sum\n",
    "\"\"\"\n",
    "\n",
    "usrpy_sum = py\"py_sum\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: \n",
       "  memory estimate:  48 bytes\n",
       "  allocs estimate:  3\n",
       "  --------------\n",
       "  minimum time:     56.372 ms (0.00% GC)\n",
       "  median time:      58.517 ms (0.00% GC)\n",
       "  mean time:        59.845 ms (0.00% GC)\n",
       "  maximum time:     81.006 ms (0.00% GC)\n",
       "  --------------\n",
       "  samples:          84\n",
       "  evals/sample:     1"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "py_usr = @benchmark $usrpy_sum($a_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "500028.7216317387"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usrpy_sum(a_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usrpy_sum(a_list) ≈ sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 4 entries:\n",
       "  \"C\"                   => 1.70617\n",
       "  \"Python numpy\"        => 0.791941\n",
       "  \"Python user-defined\" => 56.3724\n",
       "  \"Python built-in\"     => 8.65179"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_times[\"Python user-defined\"] = minimum(py_usr.times)/1e6\n",
    "sum_times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Julia (built-in)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets try Julia's built-in sum function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "sum(a::<b>AbstractArray</b>; <i>dims</i>) in Base at <a href=\"https://github.com/JuliaLang/julia/tree/381693d3dfc9b7072707f6d544f82f6637fc5e7c/base/reducedim.jl#L652\" target=\"_blank\">reducedim.jl:652</a>"
      ],
      "text/plain": [
       "sum(a::AbstractArray; dims) in Base at reducedim.jl:652"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@which sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: \n",
       "  memory estimate:  0 bytes\n",
       "  allocs estimate:  0\n",
       "  --------------\n",
       "  minimum time:     682.417 μs (0.00% GC)\n",
       "  median time:      748.769 μs (0.00% GC)\n",
       "  mean time:        784.588 μs (0.00% GC)\n",
       "  maximum time:     2.353 ms (0.00% GC)\n",
       "  --------------\n",
       "  samples:          6224\n",
       "  evals/sample:     1"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jl_bench = @benchmark sum($a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 5 entries:\n",
       "  \"C\"                   => 1.70617\n",
       "  \"Python numpy\"        => 0.785514\n",
       "  \"Python user-defined\" => 56.3724\n",
       "  \"Python built-in\"     => 8.65179\n",
       "  \"Julia built-in\"      => 0.682417"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_times[\"Julia built-in\"] = minimum(jl_bench.times)/1e6\n",
    "sum_times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Julia user-defined"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mysum (generic function with 1 method)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function mysum(a)\n",
    "    sum = 0.0\n",
    "    for i in a\n",
    "        sum += i\n",
    "    end\n",
    "    sum\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BenchmarkTools.Trial: \n",
       "  memory estimate:  0 bytes\n",
       "  allocs estimate:  0\n",
       "  --------------\n",
       "  minimum time:     1.729 ms (0.00% GC)\n",
       "  median time:      1.762 ms (0.00% GC)\n",
       "  mean time:        1.772 ms (0.00% GC)\n",
       "  maximum time:     2.538 ms (0.00% GC)\n",
       "  --------------\n",
       "  samples:          2791\n",
       "  evals/sample:     1"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usrjl_bench = @benchmark mysum($a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Any,Any} with 6 entries:\n",
       "  \"C\"                   => 1.70617\n",
       "  \"Python numpy\"        => 0.785514\n",
       "  \"Python user-defined\" => 56.3724\n",
       "  \"Julia hand-written\"  => 1.7291\n",
       "  \"Python built-in\"     => 8.65179\n",
       "  \"Julia built-in\"      => 0.682417"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_times[\"Julia hand-written\"] = minimum(usrjl_bench.times)/1e6\n",
    "sum_times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Julia built-in............0.68\n",
      "Python numpy..............0.79\n",
      "C.........................1.71\n",
      "Julia hand-written........1.73\n",
      "Python built-in...........8.65\n",
      "Python user-defined......56.37\n"
     ]
    }
   ],
   "source": [
    "key = []\n",
    "val = []\n",
    "for (k,v) in sort(collect(sum_times), by=x -> x[2])\n",
    "    push!(key,k); push!(val,v)\n",
    "    println(rpad(k,20,\".\"),lpad(round(v,digits=2),10,\".\"))\n",
    "end"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.4.1",
   "language": "julia",
   "name": "julia-1.4"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}