6b. Optimization-Scipy-Brute.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scipy Optimization [Source](https://docs.scipy.org/doc/scipy/reference/optimize.html#module-scipy.optimize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SciPy optimize provides functions for minimizing (or maximizing) objective functions, possibly subject to constraints. It includes solvers for nonlinear problems (with support for both local and global optimization algorithms), linear programing, constrained and nonlinear least-squares, root finding and curve fitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Scalar Functions Optimization\n",
    "2. Local (Multivariate) Optimization\n",
    "3. Global Optimization\n",
    "  - **basinhopping** : Find the global minimum of a function using the basin-hopping algorithm\n",
    "  - **brute** : Minimize a function over a given range by brute force.\n",
    "  - **differential_evolution** : Finds the global minimum of a multivariate function.\n",
    "  - **shgo** : Finds the global minimum of a function using SHG optimization.\n",
    "  - **dual_annealing**: Find the global minimum of a function using Dual Annealing.\n",
    "\n",
    "\n",
    "4. Least-squares and Curve Fitting\n",
    "5. Root finding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Brute Method (Global Optimization)\n",
    "\n",
    "Minimize a function over a given range by brute force.\n",
    "\n",
    "Uses the “brute force” method, i.e., computes the function’s value at each point of a multidimensional grid of points, to find the global minimum of the function.\n",
    "\n",
    "The function is evaluated everywhere in the range with the datatype of the first call to the function, as enforced by the vectorize NumPy function. The value and type of the function evaluation returned when ```full_output=True``` are affected in addition by the finish argument (see Notes).\n",
    "\n",
    "The brute force approach is inefficient because the number of grid points increases exponentially - the number of grid points to evaluate is ```Ns ** len(x)```. Consequently, even with coarse grid spacing, even moderately sized problems can take a long time to run, and/or run into memory limitations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import numpy.linalg as la\n",
    "import scipy.optimize as sopt\n",
    "from scipy.optimize import minimize\n",
    "\n",
    "import matplotlib.pyplot as pt\n",
    "from mpl_toolkits.mplot3d import axes3d\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = (2, 3, 7, 8, 9, 10, 44, -1, 2, 26, 1, -2, 0.5)\n",
    "\n",
    "def f1(z, *params):\n",
    "    x, y = z\n",
    "    a, b, c, d, e, f, g, h, i, j, k, l, scale = params\n",
    "    return (a * x**2 + b * x * y + c * y**2 + d*x + e*y + f)\n",
    "\n",
    "def f2(z, *params):\n",
    "    x, y = z\n",
    "    a, b, c, d, e, f, g, h, i, j, k, l, scale = params\n",
    "    return (-g*np.exp(-((x-h)**2 + (y-i)**2) / scale))\n",
    "\n",
    "def f3(z, *params):\n",
    "    x, y = z\n",
    "    a, b, c, d, e, f, g, h, i, j, k, l, scale = params\n",
    "    return (-j*np.exp(-((x-k)**2 + (y-l)**2) / scale))\n",
    "\n",
    "def f(z, *params):\n",
    "    return f1(z, *params) + f2(z, *params) + f3(z, *params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "rranges = (slice(-4, 4, 0.25), slice(-4, 4, 0.25))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import optimize\n",
    "resbrute = optimize.brute(f, rranges, args=params, full_output=True,\n",
    "                          finish=optimize.fmin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-3.4085818767996527"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resbrute[0]  # global minimum\n",
    "resbrute[1]  # function value at global minimum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. https://andreask.cs.illinois.edu/cs357-s15/public/demos/12-optimization/Steepest%20Descent.html\n",
    "2. https://scipy-lectures.org/advanced/mathematical_optimization/auto_examples/plot_gradient_descent.html\n",
    "3. https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html\n",
    "4. https://scipy-cookbook.readthedocs.io/index.html\n",
    "5. http://folk.ntnu.no/leifh/teaching/tkt4140/._main000.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}