scalapy-intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7c2adb17-7745-4702-a2b2-a87da9d6ec8f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# ScalaPy tutorial\n",
    "In this notebook I will show how to use Scalapy -- a library for using python from Scala.\n",
    "\n",
    "Important Links:\n",
    "- Main Site: [https://scalapy.dev/](https://scalapy.dev/)\n",
    "- Github: [https://github.com/scalapy/scalapy](https://github.com/scalapy/scalapy)\n",
    "- Facade Examples\n",
    "    - [OpenAI Gym](https://github.com/cric96/scalapy-gym)\n",
    "    - [Tensorflow](https://github.com/shadaj/scalapy-tensorflow)\n",
    "    - [NumPy](https://github.com/shadaj/scalapy-numpy)\n",
    " \n",
    " ScalaPy is released for Scala 2.11 - 3 under the organization `dev.scalapy` on Maven Central, the last version is `0.5.3`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3bf86847-304c-4443-8a57-b0652af8233c",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36m$ivy.$                                \n",
       "\u001b[39m"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import $ivy.`dev.scalapy::scalapy-core:0.5.3`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9868a64a-ecd4-4fc0-973b-b0cb2cba4d4f",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overview\n",
    "Scalapy serves as a bridge, enabling seamless invocation of Python functions directly from Scala code. \n",
    "\n",
    "This library harnesses the scala.Dynamic trait to provide dynamic typing functionalities akin to those found in scala.js, thereby streamlining the integration process. The scala macro system performs the majority of the type conversions.\n",
    "\n",
    "At present, the library is compatible with Python versions 3.6 through 3.10. While an initial working version exists for scala 3.0, it lacks comprehensive support for numerous facade definition features, such as curried methods.\n",
    "\n",
    "Consequently, for the purposes of this tutorial, we will be utilizing scala version 2.13."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7f9cb44",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Main Concepts\n",
    "ScalaPy is structured upon several key abstractions, each tailored to facilitate seamless interaction between Scala and Python. These include:\n",
    "\n",
    "- `py.Any`: This is a Scala type designed to encapsulate any Python object. It serves as the foundational type from which any facade you create should inherit.\n",
    "- `py.Dynamic`: Another Scala type, `py.Dynamic` has the capability to hold a Python object and allows for the invocation of Python methods in an unsafe manner. This is particularly useful for manipulating Python objects within Scala code.\n",
    "- `py.Module`: This represents a Python module in Scala. It is the go-to type for accessing Python functions and classes, serving as a bridge to Python’s modular architecture.\n",
    "- `py.global`: As a specialized instance of `py.Module`, `py.global` mirrors the global namespace in Python. It provides a pathway to Python’s global functions and facilitates their accessibility within Scala.\n",
    "- `py.readwrite.Writer` and `py.readwrite.Reader`: These are typeclasses that act as translators between Scala and Python objects. They allow for the conversion of Scala objects to Python objects and vice versa, thus enabling a smooth bidirectional flow of data.\n",
    "\n",
    "These core concepts are fundamental to ScalaPy's design, ensuring that Scala developers can interact with Python's rich ecosystem in a type-safe and intuitive manner.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6ccda7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Walkthrough\n",
    "In this section, we will walk through the basic steps required to use ScalaPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c09e4b53-eb87-4b21-84e0-019e13bde78d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36mme.shadaj.scalapy.py\n",
       "\u001b[39m"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// Entrypoint\n",
    "import me.shadaj.scalapy.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "72bf9952",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mres2\u001b[39m: \u001b[32mDynamic\u001b[39m = 100"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// Access to global Python function \n",
    "py.Dynamic.global.int(\"100\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef017f25",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sequence conversion\n",
    "ScalaPy provides a number of implicit conversions to facilitate the conversion of Scala sequences to Python sequences and vice versa. These conversions are defined in the `py.SeqConverters` package.\n",
    "There are mainly two types of conversions:\n",
    "- copy: This conversion creates a copy of the original sequence. Any changes made to the copy will not affect the original sequence.\n",
    "- proxy: This conversion creates a proxy of the original sequence. Any changes made to the proxy will affect the original sequence.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "af8b1e7d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36mme.shadaj.scalapy.py.SeqConverters // Conversion for Scala-Python list \n",
       "// Convert scala list to python list\n",
       "\u001b[39m\n",
       "\u001b[36melements\u001b[39m: \u001b[32mList\u001b[39m[\u001b[32mInt\u001b[39m] = \u001b[33mList\u001b[39m(\u001b[32m10\u001b[39m, \u001b[32m20\u001b[39m, \u001b[32m30\u001b[39m)\n",
       "\u001b[36mpythonList\u001b[39m: \u001b[32mAny\u001b[39m = [10, 20, 30]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// Immutable copy\n",
    "import me.shadaj.scalapy.py.SeqConverters // Conversion for Scala-Python list \n",
    "// Convert scala list to python list\n",
    "val elements = 10 :: 20 :: 30 :: Nil\n",
    "val pythonList = elements.toPythonCopy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c8af97b",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### On python Dynamic\n",
    "You Use python list as Dynamic\n",
    "since it is possible to convert any python object to other object if a Reader exists in the context.\n",
    "Any `py.Any` can be converted `to py.Dynamic`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cc164200",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36munsafeList\u001b[39m: \u001b[32mDynamic\u001b[39m = [100, 20, 30]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val unsafeList = pythonList.as[py.Dynamic]\n",
    "unsafeList.bracketUpdate(0, 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "382656be",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mres5\u001b[39m: \u001b[32mList\u001b[39m[\u001b[32mInt\u001b[39m] = \u001b[33mList\u001b[39m(\u001b[32m10\u001b[39m, \u001b[32m20\u001b[39m, \u001b[32m30\u001b[39m)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// The original sequence is not affected\n",
    "elements"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "74562175",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mmutableList\u001b[39m: \u001b[32mcollection\u001b[39m.\u001b[32mmutable\u001b[39m.\u001b[32mSeq\u001b[39m[\u001b[32mInt\u001b[39m] = \u001b[33mArrayBuffer\u001b[39m(\u001b[32m1\u001b[39m, \u001b[32m2\u001b[39m, \u001b[32m3\u001b[39m)\n",
       "\u001b[36mproxy\u001b[39m: \u001b[32mpy\u001b[39m.\u001b[32mDynamic\u001b[39m = <SequenceProxy object at 0x7f290192cfd0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val mutableList = scala.collection.mutable.Seq[Int](1, 2, 3)\n",
    "// Proxy to the original object, the changes mutableList will be reflected\n",
    "val proxy = mutableList.toPythonProxy.as[py.Dynamic]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "3430f27b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mres35_1\u001b[39m: \u001b[32mpy\u001b[39m.\u001b[32mDynamic\u001b[39m = 100"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mutableList(0) = 100 // updating the original object change the value in the proxy\n",
    "proxy.bracketAccess(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "0612bb61",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mres36\u001b[39m: \u001b[32mSeq\u001b[39m[\u001b[32mInt\u001b[39m] = \u001b[33mList\u001b[39m(\u001b[32m100\u001b[39m, \u001b[32m100\u001b[39m, \u001b[32m30\u001b[39m)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// It is possible to convert back a python list to scala list\n",
    "pythonList.as[Seq[Int]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "49b2a39d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mmutablePythonList\u001b[39m: \u001b[32mcollection\u001b[39m.\u001b[32mmutable\u001b[39m.\u001b[32mSeq\u001b[39m[\u001b[32mInt\u001b[39m] = \u001b[33mSeq\u001b[39m(\u001b[32m100\u001b[39m, \u001b[32m100\u001b[39m, \u001b[32m100\u001b[39m)\n",
       "\u001b[36mres38_2\u001b[39m: \u001b[32mpy\u001b[39m.\u001b[32mAny\u001b[39m = [100, 100, 100]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// Python list can be converted also to scala mutable list\n",
    "// In this way, it is possible to update the original python list\n",
    "val mutablePythonList = pythonList.as[scala.collection.mutable.Seq[Int]]\n",
    "mutablePythonList(2) = 100\n",
    "pythonList "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a99c320",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### On PyQuote\n",
    "\n",
    "Occasionally, accessing specific Python functions directly within Scala is required. \n",
    "In such scenarios, quoting code is a practical approach. \n",
    "This method allows you to write Python code directly in Scala and interpolate Python objects using the `$` symbol. \n",
    "This integration technique enables seamless interaction between Scala and Python, facilitating the use of Python's extensive libraries and functions within a Scala environment.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "c7735996",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36mpy.PyQuote\u001b[39m\n",
       "\u001b[36mres39_1\u001b[39m: \u001b[32mpy\u001b[39m.\u001b[32mDynamic\u001b[39m = [101, 101, 101]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "import py.PyQuote\n",
    "py\"list(map(lambda x: x + 1, $pythonList))\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a5ed2b5",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Modules \n",
    "As mentioned earlier, `py.Module` is the primary type for accessing Python functions and classes. It is the ScalaPy equivalent of a Python module.\n",
    "There are two kinds of modules:\n",
    "- Dynamic/Unsafe: This module is created by calling `py.module(\"module\")`, and it returns a `py.Dynamic` object. This module is unsafe because it does not check if the module exists or not.\n",
    "- StaticModule: This module is mainly a typed facade for a Python module. It is created by extending `py.StaticModule` and defining the methods and attributes of the module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "9b29f565",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mos\u001b[39m: \u001b[32mpy\u001b[39m.\u001b[32mModule\u001b[39m = <module 'os' from '/usr/lib/python3.8/os.py'>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// Use python module\n",
    "val os = py.module(\"os\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "84d0bf92",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mres41\u001b[39m: \u001b[32mpy\u001b[39m.\u001b[32mDynamic\u001b[39m = /home"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "454a8005",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Safe Module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "036dcc93",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defined \u001b[32mobject\u001b[39m \u001b[36mos\u001b[39m"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@py.native // necessary to mark the trait as a native python module\n",
    "object os extends py.StaticModule(\"os\") {\n",
    "  // It converts the python object to String because in the context exists a Reader[String]\n",
    "  def getcwd(): String = py.native\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "3fac72cb",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mres45\u001b[39m: \u001b[32mString\u001b[39m = \u001b[32m\"/home\"\u001b[39m"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae9af62e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Typed Facade\n",
    "It is also possible to create a typed facade for any Python class. \n",
    "This is done by extending py.Object and defining the methods and attributes of the class.\n",
    "The facade can then be used using as[FacadeType] on a py.Any object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "bd318085",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defined \u001b[32mtrait\u001b[39m \u001b[36mPyString\u001b[39m"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@py.native trait PyString extends py.Object {\n",
    "  def count(subsequence: String): Int = py.native\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "47547871",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mstring\u001b[39m: \u001b[32mPyString\u001b[39m = 0123456789"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val string = py.module(\"string\").digits.as[PyString]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "ff6ed574",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36mres48\u001b[39m: \u001b[32mInt\u001b[39m = \u001b[32m0\u001b[39m"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "string.count(\"hello1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46544cd9",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Advanced Facade\n",
    "It is also possible to create a facade using generic programming, type bounds, and type classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "7e59e01d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[32mimport \u001b[39m\u001b[36mpy.PyBracketAccess\u001b[39m\n",
       "\u001b[32mimport \u001b[39m\u001b[36mme.shadaj.scalapy.readwrite.{Reader, Writer}\u001b[39m\n",
       "defined \u001b[32mtrait\u001b[39m \u001b[36mSafePythonList\u001b[39m"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import py.PyBracketAccess\n",
    "import me.shadaj.scalapy.readwrite.{Reader, Writer}\n",
    "// A complex example, it is possible to use generics in facade only if http://127.0.0.1:8888/lab?token=73dda80fbe123ad9d75007ca98900c061a27233772c0a437Reader and Writer exist.\n",
    "@py.native\n",
    "trait SafePythonList[A] extends py.Object {\n",
    "  @PyBracketAccess\n",
    "  def apply(index: Int)(implicit reader: Reader[A]): A = py.native\n",
    "\n",
    "  @PyBracketAccess\n",
    "  def update(index: A, newValue: Int)(implicit writer: Writer[A]): Unit = py.native\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "d8e396b3",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[36msafeList\u001b[39m: \u001b[32mSafePythonList\u001b[39m[\u001b[32mInt\u001b[39m] = [10, 2, 3]\n",
       "\u001b[36mres50_1\u001b[39m: \u001b[32mInt\u001b[39m = \u001b[32m1\u001b[39m"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val safeList = py\"[1, 2, 3]\".as[SafePythonList[Int]]\n",
    "safeList(0)\n",
    "safeList(0) = 10\n",
    "// Illegal -> safeList(0) = \"10\""
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Scala",
   "language": "scala",
   "name": "scala"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".sc",
   "mimetype": "text/x-scala",
   "name": "scala",
   "nbconvert_exporter": "script",
   "version": "2.13.10"
  },
  "rise": {
   "theme": "moon",
   "transition": "zoom"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}