Skip to content

index

Jump to bottom Edit New page
Wes Turner edited this page Jun 24, 2024 · 185 revisions 
.. index:: westurner/tools

.. index:: Tools

Tools

See Also: https://westurner.github.io/wiki/projects#tools

.. index:: Packages

Packages

A software package is an archive of files with a manifest that lists the files included. Often, the manifest contains file checksums and a signature.

Many packaging tools make a distinction between source and/or binary packages.

Some packaging tools provide configuration options for:

  • Scripts to run when packaging
  • Scripts to run at install time
  • Scripts to run at uninstal time
  • Patches to apply to the "vanilla" source tree, as might be obtained from a version control repository

There is a package maintainer whose responsibilities include:

  • Testing new upstream releases
  • Vetting changes from release to release
  • Repackaging upstream releases
  • Signing new package releases

Packaging lag refers to how long it takes a package maintainer to repackage upstream releases for the target platform(s).

.. index:: Anaconda

Anaconda

Anaconda is a maintained distribution of :ref:`Conda` packages for many languages; especially :ref:`Python`.

Note

https://en.wikipedia.org/wiki/Anaconda_(installer) (1999) is the installer for :ref:`RPM`-based :ref:`Linux` distributions; which is also written in :ref:`Python` (and :ref:`C`).

.. index:: APT

APT

APT ("Advanced Packaging Tool") is the core of :ref:`Debian` package management.

  • An APT package repository serves :ref:`DEB` packages created with :ref:`Dpkg`.

  • An APT package repository can be accessed from a local filesystem or over a network protocol ("apt transports") like HTTP, HTTPS, RSYNC, FTP, and BitTorrent (debtorrent).

    An example of APT usage (e.g. to maintain an updated :ref:`Ubuntu` :ref:`Linux` system):

apt-get update
apt-get upgrade
apt-get dist-upgrade

apt-cache show bash
apt-get install bash

apt-get --help
man apt-get
man sources.list
.. index:: AUR

AUR

AUR (:ref:`Arch` User Repository) contains :ref:`PKGBUILD` packages which can be installed by :ref:`pacman`.

.. index:: Bower

Bower

Bower is "a package manager for the web" (:ref:`Javascript` packages) built on :ref:`NPM`.

.. index:: BUILD

BUILD

A BUILD file describes a :ref:`Pants Build` build.

.. index:: Cabal

Cabal

Cabal is a package manager for :ref:`Haskell` packages.

Hackage is the community Cabal package index: https://hackage.haskell.org/

.. index:: Conda Package

.. index:: Conda

Conda

Conda is a package build, environment, and distribution system written in :ref:`Python` to install packages written in any language.

  • Conda was originally created for the Anaconda Python Distribution, which installs packages written in :ref:`Python`, R, :ref:`Javascript`, :ref:`Ruby`, :ref:`C`, :ref:`Fortran`
  • Conda packages are basically tar archives with build, and optional link/install and uninstall scripts.
  • conda-build generates conda packages from conda recipes with a meta.yaml, a build.sh, and/or a build.bat.
  • Conda recipes reference and build from a source package URI OR a :ref:`VCS <vcs>` URI and revision; and/or custom build.sh or build.bat scripts.
  • conda skeleton can automatically create conda recipes from PyPI (Python), CRAN (R), and CPAN (Perl)
  • conda skeleton-generated recipes can be updated with additional metadata, scripts, and source URIs (as separate patches or consecutive branch commits of e.g. a conda-recipes repository in order to get a diff of the skeleton recipe and the current recipe).
  • Conda (and :ref:`Anaconda`) packages are hosted by https://binstar.org, which hosts free public and paid private Conda packages.
    • Anaconda Server is an internal "Private, Secure Package Repository" that "supports over 100 different repositories, including PyPI, CRAN, conda, and the Anaconda repository."

To create a fresh conda env:

# Python 2.7
conda create -n science --yes python readline conda-env

# Python 3.X
conda create -n science3 --yes python=3 readline conda-env

Work on a conda env:

source activate exmpl2
conda list
source deactivate

conda-env writes to and creates environments from environment.yml files which list conda and :ref:`pip` packages.

Work with conda envs and environment.yml files:

# Install conda-env globally (in the "root" conda environment)
conda install -n root conda-env

# Create a conda environment with ``conda-create`` and install conda-env
conda create -n science python=3 readline conda-env pip

# Install some things with conda (and envs/science/bin/pip)
# https://github.com/westurner/notebooks/blob/gh-pages/install.sh
conda search pandas; conda info pandas
conda install blaze dask bokeh odo \
              sqlalchemy hdf5 h5py \
              scikit-learn statsmodels \
              beautiful-soup lxml html5lib pandas qgrid \
              ipython-notebook
pip install -e git+https://github.com/rdflib/rdflib@master#egg=rdflib
pip install arrow sarge structlog

# Export an environment.yml
#source deactivate
conda env export -n science | tee environment.yml

# Create an environment from an environment.yml
conda env create -n projectname -f ./environment.yml

To install a conda package from a custom channel:

conda install -c pydanny cookiecutter   # OR pip install cookiecutter

The conda-forge custom channel packages are built with :ref:`continuous integration` on multiple platforms:

Sources:

See also: :ref:`Anaconda`, :ref:`conda-forge` (:ref:`conda-smithy`)

.. index:: conda-forge

conda-forge

# create a conda package recipe from a pypi package
cd $VIRTUAL_ENV/src
conda skeleton pypi jupyterthemes
ls -ld jupyterthemes/
edit jupyterthemes/meta.yaml
# - git repo tags || pypi releases

# create a conda-forge feedstock from a conda recipe
## https://github.com/conda-forge/conda-smithy#making-a-new-feedstock
cd $VIRTUAL_ENV/src
ls -ld jupyterthemes
conda-smithy init jupyterhemes
ls jupyterthemes-feedstock/

# build a conda-forge feedstock with docker
# FROM condaforge/linux-anvil
cat ./scripts/run_docker_build.sh
./scripts/run_docker_build.sh
./ci_support/run_docker_build.sh
.. index:: conda-smithy

conda-smithy

.. index:: DEB

DEB

DEB is the :ref:`Debian` software package format.

DEB packages are built with :ref:`dpkg` and often hosted in an :ref:`APT` package repository.

.. index:: Dpkg

Dpkg

Dpkg is a collection of tools for creating and working with :ref:`DEB` packages.

.. index:: dnf

dnf

dnf is a an open source package manager written in :ref:`Python`.

  • dnf was introduced in :ref:`Fedora` 18.
  • dnf is the default package manager in :ref:`Fedora` 22; replacing :ref:`yum`.
    • [ ] yum errors if TODO package is installed (* :ref:`salt` provider)
    • [ ] repoquery redirects with an error to dnf repoquery
    • See dnf help (and man dnf)
  • dnf integrates with the Anaconda system installer.
  • :ref:`dnf` supports Delta RPM packages (DRPM), which often significantly reduce the required amount of network transfer required to regularly retrieve and upgrade to the latest repository packages.
.. index:: ebuild

ebuild

ebuild is a software package definition format.

.. index:: emerge

emerge

emerge is the primary CLI tool used for installing packages built from :ref:`ebuilds <ebuild>` [from :ref:`Portage`].

.. index:: fpm

fpm

fpm (effing package management) is a tool for building many types of software packages from many other types of software packages (e.g. :ref:`DEB`. :ref:`RPM`, :ref:`Python Packages`); often more easily than working with the actual package manager.

  • fpm package source types include: dir rpm gem python empty tar deb cpan npm osxpkg pear pkgin virtualenv zip.
  • fpm target package types include: rpm deb solaris puppet dir osxpkg p5p puppet sh tar zip
.. index:: Brew

.. index:: Homebrew

Homebrew

Homebrew is a package manager (brew) for :ref:`OSX`.

.. index:: NPM

.. index:: Node Package Manager

NPM

NPM is a :ref:`Javascript` package manager created for :ref:`Node.js`.

  • an NPM package is defined by a package.json :ref:`JSON` file.
  • NPM packages are installed with the npm CLI utility.
  • :ref:`Bower` builds upon NPM.
.. index:: NuGet

NuGet

NuGet is an open source package manager for :ref:`Windows`.

.. index:: pacman

pacman

Pacman is an open source package manager which installs .pkg.tar.xz files for :ref:`Arch` Linux.

.. index:: PEX

PEX

PEX (Python Executable) is a :ref:`ZIP`-based software package archive format with an executable header.

.. index:: PKGBUILD

PKGBUILD

PKGBUILD is a shell script containing the build information for an :ref:`AUR` :ref:`Arch` :ref:`linux` software package.

.. index:: Portage

Portage

Portage is a package management and repository system written in :ref:`Python` initially just for :ref:`Gentoo` :ref:`Linux`.

.. index:: Ports

Ports

A Ports collection contains Sources (e.g. archived releases and patch sets) and :ref:`Makefiles <make>` designed to compile software :ref:`packages` for particular :ref:`operating systems <operating system>` distributions' kernel and standard libraries usually for a particular platform.

.. index:: RPM

RPM

RPM (RPM Package Manager, :ref:`RedHat` Package Manager) is a :ref:`package` format and a set of commandline utilities written in :ref:`C` and :ref:`Perl`.

  • RPM packages can be installed with rpm, :ref:`yum`, :ref:`dnf`.

  • RPM pacage can be built with tools like rpmbuild and fpm

  • Python packages can be built into RPM packages with :ref:`setuptools' <setuptools>` bdist_rpm, fpm

  • List contents of RPM packages (archives) with e.g. less and lesspipe:

    less ~/path/to/local.rpm   # requires lesspipe to be configured

  • RPM Packages are served by and retrieved from repositories by tools like :ref:`yum` and :ref:`dnf`:

Note

There's not yet a :ref:`debtorrent` for :ref:`RPM`, :ref:`YUM`, :ref:`DNF`.

.. index:: Egg

.. index:: Python Egg

.. index:: Python Packages

Python Packages

A :ref:`Python` Package is a collection of source code and package data files.

  • Python packages have dependencies: they depend on other packages
  • Python packages can be served from a package index
  • :ref:`PyPI` is the community Python Package Index
  • A Python package is an archive of files (.zip (.egg, .whl), .tar, .tar.gz,) containing a setup.py file containing a version string and metadata that is meant for distribution.
  • An source dist (sdist) package contains source code (every file listed in or matching a pattern in a MANIFEST.in text file).
  • A binary dist (bdist, bdist_egg, bdist_wheel) is derived from an sdist and may be compiled and named for a specific platform.
  • sdists and bdists are defined by a setup.py file which contains a call to a distutils.setup() or setuptools.setup() function.
  • The arguments to the setup.py function are things like version, author, author_email, and homepage; in addition to package dependency strings required for the package to work (install_requires), for tests to run (tests_require), and for optional things to work (extras_require).
  • A package dependency string can specify an exact version (==) or a greater-than (>=) or less-than (<=) requirement for each package.
  • Package names are looked up from an index server (--index), such as :ref:`PyPI`, and or an HTML page (--find-links) containing URLs containing package names, version strings, and platform strings.
  • easy_install (:ref:`setuptools`) and :ref:`pip` can install packages from: the local filesystem, a remote index server, or a local index server.
  • easy_install and pip read the install_requires (and extras_require) attributes of setup.py files contained in packages in order to resolve a dependency graph (which can contain cycles) and install necessary packages.

.

Note

:ref:`JSON-LD-` for package metadata and environment build metadata could be helpful.

.. index:: distutils
Distutils

Distutils is a collection of tools for common packaging needs.

.. index:: setuptools
Setuptools

Setuptools is a :ref:`Python package <python packages>` for working with other :ref:`Python Packages`.

  • Setuptools builds upon :ref:`distutils`

  • Setuptools is widely implemented

  • Most Python packages are installed by setuptools (by :ref:`Pip`)

  • Setuptools can be installed by downloading ez_setup.py and then running python ez_setup.py; or, setuptools can be installed with a system package manager (apt, yum)

  • Setuptools installs a script called easy_install which can be used to install packages from the local filesystem, a remote index server, a local index server, or an HTML page

  • easy_install pip installs :ref:`Pip` from PyPI

  • Like easy_install, :ref:`Pip` installs python packages, with a number of additional configuration options

  • Setuptools can build :ref:`RPM` and :ref:`DEB` packages from python packages, with some extra configuration:

    python setup.py bdist_rpm --help
python setup.py --command-packages=stdeb.command bdist_deb --help

.. index:: Pip
Pip

Pip is a tool for installing, upgrading, and uninstalling :ref:`Python` packages.

pip help
pip help install
pip --version

sudo apt-get install python-pip
pip install --upgrade pip

pip install libcloud
pip install -r requirements.txt
pip uninstall libcloud
  • Pip stands upon :ref:`distutils` and :ref:`setuptools`.
  • Pip retrieves, installs, upgrades, and uninstalls packages.
  • Pip can list installed packages with pip freeze (and pip list).
  • Pip can install packages as 'editable' packages (pip install -e) from version control repository URLs which must begin with vcs+, end with #egg=<usuallythepackagename>, and may contain an @vcstag tag (such as a branch name or a version tag).
  • Pip installs packages as editable by first cloning (or checking out) the code to ./src (or ${VIRTUAL_ENV}/src if working in a :ref:`virtualenv`) and then running setup.py develop.
  • Pip configuration is in ${HOME}/.pip/pip.conf.
  • Pip can maintain a local cache of downloaded packages, which can lessen the load on package servers during testing.
  • Pip skips reinstallation if a package requirement is already satisfied.
  • Pip requires the --upgrade and/or --force-reinstall options to be added to the pip install command in order to upgrade or reinstall.
  • At the time of this writing, the latest stable pip version is 1.5.6.

Warning

With :ref:`Python` 2, pip is preferable to :ref:`setuptools`'s easy_install because pip installs backports.ssl_match_hostname in order to validate HTTPS certificates (by making sure that the certificate hostname matches the hostname from which the DNS resolved to).

Cloning packages from source repositories over ssh:// or https://, either manually or with pip install -e avoids this concern.

There is also a tool called :ref:`peep` which requires considered-good SHA256 checksums to be specified for every dependency listed in a requirements.txt file.

For more information, see: https://legacy.python.org/dev/peps/pep-0476/#python-versions

.. glossary::

   Pip Requirements File
      Plaintext list of packages and package URIs to install.

      Requirements files may contain version specifiers (``pip >= 1.5``)

      Pip installs Pip Requirement Files::

         pip install -r requirements.txt
         pip install --upgrade -r requirements.txt
         pip install --upgrade --user --force-reinstall -r requirements.txt

      An example ``requirements.txt`` file::

         # install pip from the default index (PyPI)
         pip
         --index=https://pypi.python.org/simple --upgrade pip

         # Install pip 1.5 or greater from PyPI
         pip >= 1.5

         # Git clone and install pip as an editable develop egg
         -e git+https://github.com/pypa/[email protected]#egg=pip

         # Install a source distribution release from PyPI
         # and check the MD5 checksum in the URL
         https://pypi.python.org/packages/source/p/pip/pip-1.5.5.tar.gz#md5=7520581ba0687dec1ce85bd15496537b

         # Install a source distribution release from Warehouse
         https://warehouse.python.org/packages/source/p/pip/pip-1.5.5.tar.gz

         # Install an additional requirements.txt file
         -r requirements/more-requirements.txt


.. index:: Peep
Peep

Peep works just like :ref:`pip`, but requires SHA256 checksum hashes to be specified for each package in requirements.txt file.

.. index:: Python Package Index

.. index:: PyPI
PyPI

PyPI-legacy is the original Python Package Index. PyPI is now powered by :ref:`Warehouse`

.. index:: Warehouse
Warehouse

Warehouse is the "Next Generation Python Package Repository".

All packages uploaded to :ref:`PyPI` are also available from Warehouse.

.. index:: Devpi
Devpi

Devpi is a server and client solution for :ref:`Python package <python packages>` mirroring, hosting, and testing.

.. index:: Python Wheel

.. index:: Wheel
Wheel
  • Wheel is a newer, PEP-based standard (.whl) with a different metadata format, the ability to specify (JSON) digital signatures for a package within the package, and a number of additional speed and platform-consistency advantages.
  • Wheels can be uploaded to PyPI.
  • Wheels are generally faster than traditional Python packages.

Packages available as wheels are listed at https://pythonwheels.com/.

.. index:: Ruby Gem

.. index:: RubyGems

RubyGems

RubyGems is a package manager for :ref:`Ruby` packages ("Gems").

.. index:: Yum

Yum

Yum is a tool for installing, upgrading, and uninstalling :ref:`RPM` packages.

.. index:: Version Control Systems

.. index:: Distributed Version Control Systems

Version Control Systems

Version Control Systems (VCS) --- or Revision Control Systems (RCS) --- are designed to solve various problems in change management.

  • VCS store code in a repository.
  • Changes to one or more files are called changesets, commits, or revisions
  • Changesets are comitted or checked into to a repository.
  • Changesets are checked out from a repository
  • Many/most VCS differentiate between the repository and a working directory, which is currently checked out to a specific changeset identified by a revision identifier; possibly with uncommitted local changes.
  • A branch is forked from a line of development and then merged back in.
  • Most projects designate a main line of development referred to as a trunk, master, or default branch.
  • Many projects work with feature and release branches, which, ideally, eventually converge by being merged back into trunk. (see: :ref:`HubFlow` for an excellent example of branching)
  • Traditional VCS are centralized on a single point-of-failure.
  • Some VCS have a concept of locking to prevent multiple peoples' changes from colliding
  • Distributed Version Control Systems (DVCS) (can) clone all revisions of every branch of a repository every time. *
  • DVCS changesets are pushed to a different repository
  • DVCS changesets are pulled from another repository into a local clone or copy of a repository
  • Teams working with DVCS often designate a central repository hosted by a project forge service like SourceForge, GNU Savannah, GitHub, or BitBucket.
  • Contributors send patches which build upon a specific revision, which can be applied by a maintainer with commit access permissions.
  • Contributors fork a new branch from a specific revision, commit changes, and then send a pull request, which can be applied by a maintainer with commit access permissions.
.. index:: CVS

CVS

CVS (cvs) is a centralized version control system (VCS) written in :ref:`C`.

CVS predates most/many other VCS.

.. index:: SVN

.. index:: Subversion

svn: Subversion

Apache Subversion (svn) is a centralized revision control system (VCS) written in :ref:`C`.

To checkout a revision of a repository with svn:

svn co https://svn.apache.org/repos/asf/subversion/trunk subversion
.. index:: bzr

.. index:: Bazaar

bzr: Bazaar

GNU Bazaar (bzr) is a distributed revision control system (DVCS, RCS, VCS) written in :ref:`Python` and :ref:`C`.

https://launchpad.net hosts Bazaar repositories; with special support from the bzr tool in the form of lp: URIs like lp:bzr.

To clone a repository with bzr:

bzr branch lp:bzr
.. index:: Git

Git

Git (git) is an open source distributed version control system for tracking a branching and merging repository of file revisions written in :ref:`C` (DVCS, VCS, RCS).

To clone a repository with git:

git clone https://github.com/git/git
cd ./git
git status; git remote -av; git reflog;

git help help; git help reflog
  • :ref:`jj` auto-saves and is compatible with git
.. index:: GitFlow

GitFlow

GitFlow is a named branch workflow for :ref:`git` with master, develop, feature, release, hotfix, and support branches (git flow).

Gitflow branch names and prefixes are configured in .git/config; the defaults are:

GitFlow Branch Names
Branch Name Description (and Code Labels)
master Stable trunk (latest release)
develop Development main line
feature/<name> New features for the next release (e.g. ENH, PRF)
release/<name> In-progress release branches (e.g. RLS)
hotfix/<name> Fixes to merge to both master and develop (e.g. BUG, TST, DOC)
support/<name>

"What is the 'support' branch?"

https://github.com/nvie/gitflow/wiki/FAQ

Creating a new release with :ref:`Git` and :ref:`GitFlow`:

git clone ssh://[email protected]/westurner/dotfiles
# git checkout master
# git checkout -h
# git help checkout (man git-checkout)
# git flow [<cmd> -h]
# git-flow [<cmd> -h]

git flow init
## Update versiontag in .git/config to prefix release tags with 'v'
git config --replace-all gitflow.prefix.versiontag v
cat ./.git/config
# [gitflow "prefix"]
# feature = feature/
# release = release/
# hotfix = hotfix/
# support = support/
# versiontag = v
#

## feature/ENH_print_hello_world
git flow feature start ENH_print_hello_world
#git commit, commit, commit
git flow feature
git flow feature finish ENH_print_hello_world   # ENH<TAB>

## release/0.1.0
git flow release start 0.1.0
#git commit (e.g. update __version__, setup.py, release notes)
git flow release finish 0.1.0
git flow release finish 0.1.0
git tag | grep 'v0.1.0'
.. index:: HubFlow

HubFlow

GitFlow is a named branch workflow for :ref:`git` with master, develop, feature, release, hotfix, and support branches (git flow).

HubFlow is a fork of :ref:`GitFlow` that adds useful commands for working with :ref:`Git` and GitHub pull requests.

HubFlow branch names and prefixes are configured in .git/config; the defaults are:

HubFlow Branch Names
Branch Name Description (and Code Labels)
master Stable trunk (latest release)
develop Development main line
feature/<name> New features for the next release (e.g. ENH, PRF)
release/<name> In-progress release branches (e.g. RLS)
hotfix/<name> Fixes to merge to both master and develop (e.g. BUG, TST, DOC)

Creating a new release with :ref:`Git` and :ref:`HubFlow`:

git clone ssh://[email protected]/westurner/dotfiles
# git checkout master
# git checkout -h
# git help checkout (man git-checkout)
# git hf [<cmd> -h]
# git-hf [<cmd> -h]

git hf init
## Update versiontag in .git/config to prefix release tags with 'v'
git config --replace-all hubflow.prefix.versiontag v
#cat .git/config # ...
# [hubflow "prefix"]
# feature = feature/
# release = release/
# hotfix = hotfix/
# support = support/
# versiontag = v
#
git hf update
git hf pull
git hf pull -h

## feature/ENH_print_hello_world
git hf feature start ENH_print_hello_world
#git commit, commit
git hf pull
git hf push
#git commit, commit
git hf feature finish ENH_print_hello_world   # ENH<TAB>

## release/0.1.0
git hf release start 0.1.0
## commit (e.g. update __version__, setup.py, release notes)
git hf release finish 0.1.0
git hf release finish 0.1.0
git tag | grep 'v0.1.0'

The GitFlow HubFlow illustrations are very helpful for visualizing and understanding any DVCS workflow: https://datasift.github.io/gitflow/IntroducingGitFlow.html.

GitFlow Release / Master Branch Merge Diagram
GitFlow Hotfix to Master and Develop Branches Merge Diagram
Numbered GitFlow Workflow Diagram 
.. index:: Hg

.. index:: Mercurial

hg: Mercurial

Mercurial (hg) is a distributed revision control system written in :ref:`Python` and :ref:`C` (DVCS, VCS, RCS).

To clone a repository with hg:

hg clone https://www.mercurial-scm.org/repo/hg
.. index:: jujutsu

.. index:: jj

jj: jujutsu

jj (jujutsu) is an open source :ref:`Version Control System` that works with :ref:`git`.

  • jj auto-saves changes

    From the jj docs > Concepts > Working Copy : https://martinvonz.github.io/jj/latest/working-copy/ :

    Unlike most other VCSs, Jujutsu will automatically create commits from the working-copy contents when they have changed. Most jj commands you run will commit the working-copy changes if they have changed. The resulting revision will replace the previous working-copy revision.

.. index:: GitHub

GitHub

:ref:`Git` repos, Issues, Pull Requests, Wikis, Pages, Actions, Project Boards, Webhooks

.. index:: GitHub CI

.. index:: GitHub Actions

GitHub Actions

.. index:: GitHub Classroom

GitHub Classroom

.. index:: GitLab

GitLab

.. index:: GitLab CI

GitLab CI

GitLab CI is an open source :ref:`Continuous Integration` system which runs commands in containers per tasks defined in a build :ref:`YAML` file on project events like git push, new Pull Request branch, new Issue.

.. index:: Gitea

Gitea

Gitea is an open source project forge site written in :ref:`Go`; with :ref:`Git` repositories (repos), Wiki repos, Issues, Pull Requests; and :ref:`Continuous Integration` to run the build script (:ref:`YAML`) on events like git push, new_issue, and new_pr;

:ref:`Package` repositories for released build artifacts, :ref:`Container` repository

.. index:: Gitea Actions

Gitea Actions

.. index:: Project Templates

Project Templates

.. index:: Exemplar Projects

Exemplar Projects

seeAlso:

.. index:: cookiecutter

cookiecutter

Cookiecutter creates projects (files and directories) from project templates written in :ref:`Jinja2` for projects written in :ref:`Python` and other languages.

.. index:: yeoman

yeoman

Languages

Programming Languages

.. index:: Programming Paradigms

Programming Paradigms

.. index:: Lightweight Markup Languages

Lightweight Markup Language

.. index:: BBCode
BBCode

BBCode is a :ref:`Lightweight markup language` often used by bulletin boards and forums.

.. index:: Markdown
Markdown

Markdown is a :ref:`Lightweight markup language` which can be parsed and transformed to valid :ref:`HTML-`.

  • GitHub and BitBucket support Markdown in Issue Descriptions, Wiki Pages, and Comments
  • :ref:`Jupyter Notebook` supports Markdown in Markdown cells
.. index:: CommonMark
CommonMark

:ref:`CommonMark` is one effort to standardize :ref:`Markdown`.

.. index:: MyST Markdown

.. index:: MyST
MyST Markdown

MyST Markdown is :ref:`CommonMark` :ref:`Markdown` with support for :ref:`Sphinx` roles and directives.

.. index:: MediaWiki Markup
MediaWiki Markup

MediaWiki Markup is a :ref:`Lightweight markup language` "WikiText" which can be parsed and transformed to valid :ref:`HTML-`.

  • Wikipedia is built on MediaWiki, which supports MediaWiki Markup.
.. index:: RD

.. index:: Ruby Document Format
RD

RD is a :ref:`Lightweight markup language` for documenting :ref:`Ruby` code and programs.

.. index:: Rdoc
RDoc

RDoc is a tool and a :ref:`Lightweight markup language` for generating :ref:`HTML-` and command-line documentation for :ref:`Ruby` projects.

To not build RDoc docs when installing a :ref:`Gem <RubyGems>`:

gem install --no-rdoc --no-ri
gem install --no-document
gem install -N
.. index:: ReStructuredText
ReStructuredText

ReStructuredText (ReST, RST) is a :ref:`Lightweight markup language` commonly used for narrative documentation and inline Python, C, Java, etc. docstrings which can be parsed, transformed, and published to valid :ref:`HTML-`, ePub, LaTeX, PDF.

:ref:`Sphinx` is built on :ref:`Docutils`, the primary implementation of ReStructuredText.

:ref:`Pandoc` also supports a form of ReStructuredText.

.. glossary::

   ReStructuredText Directive
      Actionable blocks of ReStructuredText

      | Docs: http://docutils.sourceforge.net/docs/ref/rst/directives.html

      ``include``, ``contents``, and ``index`` are all
      ReStructuredDirectives:

      .. code-block:: rest

          .. include:: goals.rst

          .. contents:: Table of Contents
           :depth: 3

           .. index:: Example 1
           .. index:: Sphinx +
           .. _example-1:

           Sphinx +1
           ==========
           This refs :ref:`example 1 <example-1>`.

           Similarly, an explicit link to this anchor `<#example-1>`__

           And an explicit link to this section `<#sphinx-1>`__
           (which is otherwise not found in the source text).


           .. index:: Example 2
           .. _example 2:

           Example 2
           ==========

           This links to :ref:`example-1` and :ref:`example 2`.

           (`<#example-1>`__, `<#example-2>`__)

           And this also links to `Example 2`_.

          .. include:: LICENSE

       .. note:: ``index`` is a :ref:`Sphinx` Directive,
           which will print an error to the console when building
           but will otherwise silently dropped
           by non-Sphinx ReStructuredText parsers
           like :ref:`Docutils` (GitHub) and :ref:`Pandoc`.

   ReStructuredText Role
      RestructuredText role extensions

      | Docs: http://docutils.sourceforge.net/docs/ref/rst/roles.html

      ``:ref:`` is a :ref:`Sphinx` RestructuredText Role:

      .. code-block:: rest

          A (between files) link to :ref:`example 2`.




.. index:: C

C

C is a third-generation programming language which affords relatively low-level machine access while providing helpful abstractions.

Every :ref:`Windows` kernel is written in C.

The GNU/:ref:`Linux` kernel is written in C and often compiled by :ref:`GCC` or :ref:`Clang` for a particular architecture (see: man uname)

The :ref:`OSX` kernel is written in C.

:ref:`Libc` libraries are written in C.

Almost all of the projects linked here, at some point, utilize code written in C.

.. index:: Libc
Libc

A libc is a standard library of :ref:`C` routines.

Libc implementations:

.. index:: GNU Libc

.. index:: Glibc
Glibc

Glibc is the GNU :ref:`C` Library (:ref:`libc`).

Many :ref:`Linux` packages and the :ref:`GNU/Linux <linux>` kernel build from Glibc.

.. index:: BSD Libc
BSD Libc

BSD libc are a superset of :ref:`POSIX`.

:ref:`OSX` builds from BSD libc.

:ref:`Android` :ref:`Bionic` is a BSD libc.

.. index:: Bionic
Bionic

Bionic is the :ref:`Android` :ref:`libc`, which is a :ref:`BSD Libc <bsd-libc>`.

.. index:: C++

C++

C++ is a free and open source third-generation programming language which adds object orientation and a standard library to :ref:`C`.

.. index:: Standard Template Library
Standard Template Library
.. index:: libstdc++
libstdc++

libstdc++ is the free and open source GNU :ref:`C++` :ref:`Standard Template Library`.

libc++

libc++ (libcxx) is the free and open source :ref:`LLVM` :ref:`C++` :ref:`Standard Template Library`.

.. index:: Microsoft STL
Microsoft STL

Microsoft STL is Microsoft's free and open source implementation of the :ref:`C++` :ref:`Standard Template Library`.

  • Microsoft Visual C++ typically builds with the Microsoft STL.
.. index:: Boost
Boost

Boost is a free and open source set of :ref:`C++` libraries for doing lots of things in C++.

.. index:: Fortran

Fortran

Fortran (or FORTRAN) is a third-generation programming language frequently used for mathematical and scientific computing.

Some of the :ref:`SciPy` libraries build optimized mathematical Fortran routines.

.. index:: Haskell

Haskell

Haskell is a free and open source strongly statically typed purely functional programming language.

:ref:`Cabal` is the Haskell package manager.

:ref:`Pandoc` is written in Haskell.

.. index:: Go

Go

Go is a free and open source statically-typed :ref:`C`-based third generation language.

.. index:: Java

Java

Java is a third-generation programming language which is compiled into code that runs in a virtual machine (JVM) written in :ref:`C` for many different operating systems.

.. index:: JVM
JVM

A JVM ("Java Virtual Machine") runs :ref:`Java` code (classes and JARs).

.. index:: Javascript

Javascript

Javascript (JS) is a free and open source third-generation programming language designed to run in an interpreter; now specified as :ref:`ECMAScript`.

All major web browsers support Javascript.

Client-side (web) applications can be written in Javascript.

Server-side (web) applications can be written in Javascript, often with :ref:`Node.js`, :ref:`NPM`, and :ref:`Bower` packages.

Note

Java and JavaScript are two distinctly different languages and developer ecosystems.

.. index:: ECMAScript
ECMAScript

ECMAScript (ES) is an evolving, formally-specified, weakly-typed scripting language from which :ref:`Javascript` and ActionScript are derived.

.. index:: Babel
Babel

Babel is a :ref:`Javascript` (:ref:`ECMAScript`) compiler that transforms ES6 (ES2015) and beyond into browser-compatible JS.

.. index:: Node.js
Node.js

Node.js is a free and open source framework for :ref:`Javascript` applications written in :ref:`C`, :ref:`C++`, and :ref:`Javascript`.

.. index:: Jinja2

Jinja2

Jinja2 is a free and open source templating engine written in :ref:`Python`.

:ref:`Sphinx` and :ref:`Salt` are two projects that utilize Jinja2.

.. index:: Perl

Perl

Src: git git://perl5.git.perl.org/perl.git

Perl is a free and open source, dynamically typed, :ref:`C`-based third-generation programming language.

Many of the :ref:`Debian` system management tools are or were originally written in Perl.

.. index:: Python

Python

Python is a free and open source dynamically-typed, :ref:`C`-based third-generation programming language.

As a multi-paradigm language with support for functional and object-oriented code, Python is often utilized for system administration and scientific software development.

The Python community is generously supported by a number of sponsors and the Python Infrastructure Team:

.. index:: CPython
CPython

CPython is the reference :ref:`Python` language implementation written in :ref:`C`.

CPython can interface with other :ref:`C` libraries through a number of interfaces:

There are other implementations of Python:

.. index:: Cython
Cython

Cython is a superset of :ref:`CPython` which adds static type definitions; making :ref:`CPython` code faster, in many cases.

.. index:: NumPy
NumPy

NumPy is a library of array-based mathematical functions implemented in :ref:`C` and :ref:`Python`.

NumPy and other languages:

.. index:: SciPy
SciPy

SciPy is a set of science and engineering libraries for :ref:`Python`, primarily written in :ref:`C`.

.. index:: SciPy Stack
SciPy Stack

Python Distributions

.. index:: Scipy Stack Docker Containers
SciPy Stack Docker Containers
.. index:: Jupyter Docker Stacks
Jupyter Docker Stacks

See also:

.. index:: PyPy
PyPy

PyPy is a JIT LLVM compiler for :ref:`Python` code written in RPython -- a restricted subset of :ref:`CPython` syntax -- which compiles to :ref:`C`, and is often faster than :ref:`CPython` for many types of purposes.

.. index:: NumPyPy
NumPyPy

NumPyPy is a port of :ref:`NumPy` to :ref:`PyPy`:

.. index:: Python 3
Python 3

Python 3 made a number of incompatible changes, requiring developers to update and review their Python 2 code in order to "port to" Python 3.

Python 2 will be supported in "no-new-features" status for quite some time.

Python 3 Wall of Superpowers tracks which popular packages have been ported to support Python 3: https://python3wos.appspot.com/

There are a number of projects which help bridge the gap between the two language versions:

See also: :ref:`Anaconda`

.. index:: awesome-python-testing
awesome-python-testing
.. index:: Tox
Tox

Tox is a build automation tool designed to build and test Python projects with multiple language versions and environments in separate :ref:`virtualenvs <virtualenv>`.

Run the py27 environment:

tox -v -e py27
tox --help

.. index:: Ruby

Ruby

Ruby is a free and open source dynamically-typed programming language.

:ref:`Vagrant` is written in Ruby.

.. index:: Rust

Rust

Rust is a free and open source strongly typed multi-paradigm programming language.

.. index:: Scala

Scala

Scala is a free and open source object-oriented and functional :ref:`programming language` which compiles to :ref:`JVM` (and :ref:`LLVM`) bytecode.

.. index:: TypeScript

TypeScript

TypeScript is a free and open source :ref:`Programming Language` developed as a superset of :ref:`Javascript` with optional additional features like static typing and native object-oriented code.

.. index:: WebAssembly

WebAssembly

WebAssembly (wasm) is a safe (sandboxed), efficient low-level :ref:`programming language` (abstract syntax tree) and binary format for the web.

  • WebAssembly is initially derived from asm.js and PNaCL.
  • WebAssembly is an industry-wide effort.
  • :ref:`LLVM` can generate WebAssembly from e.g. :ref:`C` and :ref:`C++` code.
.. index:: YAML

YAML

YAML ("YAML Ain't Markup Language") is a concise data serialization format.

Most :ref:`Salt` states and pillar data are written in YAML. Here's an example top.sls file:

base:
 '*':
   - openssh
 '*-webserver':
   - webserver
 '*-workstation':
   - gnome
   - i3
.. index:: Compilers

Compilers

.. index:: Interpreters

Interpreter

.. index:: Source-to-Source Compiler

Source-to-Source compiler

.. index:: ROSE
ROSE
.. index:: Binutils

.. index:: GNU Binutils

Binutils

GNU Binutils are a set of utilities for working with assembly and binary.

:ref:`GCC` utilizes GNU Binutils to compile the GNU/:ref:`Linux` kernel and userspace.

GAS, the GNU Assembler (as) assembles ASM code for linking by the GNU linker (ld).

.. index:: Clang

Clang

Clang is a compiler front end for :ref:`C`, :ref:`C++`, and Objective C/++. Clang is part of the :ref:`LLVM` project.

.. index:: GCC

.. index:: GNU Compiler Collection

GCC

Src: git ssh://gcc.gnu.org/git/gcc.git

The GNU Compiler Collection started as a Free and Open Source compiler for :ref:`C`.

.. index:: GNU Linker

.. index:: ld

GNU Linker

The GNU Linker is the GNU implementation of the ld command for linking object files and libraries.

.. index:: LLVM

LLVM

LLVM "Low Level Virtual Machine" is a reusable compiler infrastructure with frontends for many languages.

.. index:: Operating Systems

Operating Systems

.. index:: POSIX

POSIX

POSIX ("Portable Operating System Interface") is a set of standards for :ref:`Shells`, :ref:`Operating Systems`, and APIs.

.. index:: GNU/Linux

.. index:: Linux

Linux

GNU/Linux ("Linux") is a free and open source operating system kernel written in :ref:`C`.

uname -a; echo "Linux"
uname -o; echo "GNU/Linux"
.. index:: Linux Distributions

Linux Distributions

A Linux Distribution is a collection of :ref:`Packages` compiled to work with a :ref:`GNU/Linux <linux>` kernel and a :ref:`libc`.

.. index:: Arch
Arch

Arch Linux is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`AUR` packages.

.. index:: Debian
Debian

Debian is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`DEB` packages.

.. index:: Ubuntu
Ubuntu

Ubuntu is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`DEB` packages which are often derived from :ref:`Debian` packages.

.. index:: Fedora
Fedora

Fedora is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`RPM` packages.

.. index:: RedHat

.. index:: RedHat Enterprise Linux

.. index:: RHEL
RedHat

RedHat Enterprise Linux ("RHEL") is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`RPM` packages.

.. index:: CentOS
CentOS

CentOS is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`RPM` packages which is derived from :ref:`RHEL <redhat>`.

.. index:: Scientific Linux
Scientific Linux

Scientific Linux is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`RPM` packages which is derived from :ref:`CentOS`. which is derived from :ref:`RHEL <redhat>`.

.. index:: Oracle Linux
Oracle

Oracle Linux is a :ref:`Linux Distribution <linux-distributions>` that is built from :ref:`RPM` packages which is derived from :ref:`RHEL <redhat>`.

.. index:: Gentoo
Gentoo

Gentoo is a :ref:`Linux Distribution <linux-distributions>` built on :ref:`Portage`.

.. index:: ChromiumOS
ChromiumOS

ChromiumOS is a :ref:`Linux Distribution <linux-distributions>` built on :ref:`Portage`.

.. index:: Crouton
Crouton

Crouton ("Chromium OS Universal Chroot Environment") installs and debootstraps a :ref:`Linux Distribution <linux-distributions>` (i.e. :ref:`Debian` or :ref:`Ubuntu`) within a :ref:`ChromiumOS` or :ref:`ChromeOS` chroot.

.. index:: ChromeOS
ChromeOS

ChromeOS is a :ref:`Linux Distribution <linux-distributions>` built on :ref:`ChromiumOS` and :ref:`Portage`.

.. index:: CoreOS
CoreOS

CoreOS is a :ref:`Linux Distribution <linux-distributions>` for highly available distributed computing.

CoreOS schedules redundant :ref:`docker` images with fleet and systemd according to configuration stored in etcd, a key-value store with a D-Bus interface.

  • CoreOS runs on very many platforms
  • CoreOS does not provide a package manager
  • CoreOS schedules Docker
  • CoreOS -- Operating System
  • etcd -- Consensus and Discovery
  • rkt -- Container Runtime
  • fleet -- Distributed init system (etcd, systemd)
  • flannel -- Networking
.. index:: Linux Notes
Linux Notes
Linux Dual Boot
  • [ ] GRUB chainloader to partition boot record
    • Ubuntu and Fedora GRUB try to autodiscover Windows partitions
.. index:: Android
Android
.. index:: Android SDK
Android SDK
.. index:: Android Studio
Android Studio
.. index:: Apple OSX

.. index:: OS X

.. index:: OSX

OS X

OS X is a UNIX operating system based upon the Mach kernel from NeXTSTEP, which was partially derived from NetBSD and FreeBSD.

OS X GUI support is built from XFree86/X.org :ref:`X11`.

OS X maintains forks of many POSIX BSD and GNU tools like bash, readlink, and find.

:ref:`Homebrew` installs and maintains packages for OS X.

uname; echo "Darwin"
.. index:: iOS
iOS

iOS is a closed source UNIX operating system based upon many components of :ref:`OSX` adapted for phones and then tablets.

  • iOS powers iPhones and iPads
  • You must have a Mac with :ref:`OSX` and XCode to develop and compile for iOS.
OSX Notes
OSX Reinstall
  • [ ] Generate installation media
  • [ ] Reboot to recovery partition
  • [ ] Adjust partitions
  • [ ] Format?
  • [ ] Install OS
  • [ ] (wait)
  • [ ] Manual time/date/language config
  • [ ] Run workstation provis scripts
OSX Fresh Install
  • [ ] Generate / obtain installation media
  • [ ] Boot from installation media
  • [ ] Manual time/date/language config
  • [ ] Run workstation provis scripts
OSX Dual Boot
.. index:: Windows

Windows

Microsoft Windows is a NT-kernel based operating system.

  • There used to be a POSIX compatibility mode.
  • Chocolatey maintains a set of :ref:`NuGet` packages for Windows.
.. index:: Windows Subsystem for Linux

.. index:: WSL
Windows Subsystem for Linux

Windows Subsystem for Linux (WSL) is a binary compatibility layer which allows many Linux programs to be run on Windows 10+.

The Windows Subsystem for Linux lets developers run a GNU/Linux environment -- including most command-line tools, utilities, and applications -- directly on Windows, unmodified, without the overhead of a virtual machine.
.. index:: Windows Sysinternals
Windows Sysinternals

Windows Sysinternals is a group of tools for working with :ref:`Windows`.

.. index:: WSUS Offline Update
WSUS Offline Update

WSUS Offline Update is a free and open source software tool for generating offline :ref:`Windows` upgrade CDs / DVDs containing the latest upgrades for Windows, Office, and .Net.

  • Bandwidth costs: Windows Updates (WSUS) in GB * n_machines (see also: Debtorrent, :ref:`Packages`)
  • "Slipstreaming" an installation ISO is one alternative way to avoid having to spend hours upgrading a factory reinstalled ("reformatted") :ref:`Windows` installation
Windows Notes

A few annotated excerpts from this Chocolatey :ref:`NuGet` :ref:`PowerShell` script https://gist.github.com/westurner/10950476#file-cinst_workstation_minimal-ps1

cinst GnuWin
cinst sysinternals      # Process Explorer XP
cinst 7zip
cinst curl
Windows Dual Boot
  • [ ] Windows MBR chain loads to partition GRUB (Linux)
  • [ ] Ubuntu WUBI .exe Linux Installer (XP, 7, 8*)
    • It's now better to install to a separate partition from a bootable ISO
UEFI
.. index:: Configuration Management

Configuration Management

.. index:: Ansible

Ansible

Ansible is a :ref:`Configuration Management` tool written in :ref:`Python` which runs idempotent Ansible Playbooks written in :ref:`YAML` markup with :ref:`Jinja2` variables for managing one or more physical and virtual machines running various operating systems over SSH.

.. index:: Cobbler

Cobbler

Cobbler is a machine image configuration, repository mirroring, and networked booting server with support for DNS, DHCP, TFTP, and PXE.

See also: crowbar, :ref:`OpenStack` Ironic bare-metal deployment

.. index:: JuJu

Juju

Juju is a :ref:`Configuration Management` tool written in :ref:`Python` which runs Juju Charms written in :ref:`Python` on one or more systems over SSH, for managing one or more physical and virtual machines running :ref:`Ubuntu`.

.. index:: osquery

osquery

osquery is a tool for reading and querying many sources of system data with SQL for :ref:`OSX` and :ref:`Linux`.

.. index:: Puppet

Puppet

Puppet is a :ref:`Configuration Management` system written in :ref:`Ruby` which runs Puppet Modules written in Puppet DSL or :ref:`Ruby` for managing one or more physical and virtual machines running various operating systems.

.. index:: Salt

Salt

Salt is a :ref:`Configuration Management` system written in :ref:`Python` which runs Salt Formulas written in :ref:`YAML`, :ref:`Jinja2`, :ref:`Python` for managing one or more physical and virtual machines running various operating systems.

.. index:: Software Build Tools

.. index:: Build Automation Tools

Build Automation Tools

.. index:: Autotools

GNU Autotools

GNU Autotools (GNU Build System) are a set of tools for software build automation: autoconf, automake, libtool, and gnulib.

  • The traditional ./configure --help; make; make install build workflow comes from the GNU Build System.
  • Autoconf uses a configure.ac configure include file in generating a configure script that checks for platform and software dependencies and caches the results in a config.status script, which generates config.h C header file that caches the results
  • Automake uses a Makefile.am to generate a Makefile.in makefile include file that generates a :ref:`GNU Make` Makefile
  • GNU Coding Standards define a number of standard configuration variables: CC, CFLAGS, CXX, CXXFLAGS, LDFLAGS, CPPFLAGS which tools such as :ref:`GNU Make` automatically add to e.g. :ref:`GCC` (and :ref:`ld`) build program arguments
$ # autoconf  # configure.ac -> ./configure
$ ./configure --help
$ ./configure --prefix=/usr/local --with-this-or-that
make
.. index:: Bake

Bake

Bake is a free and open source software build automation tool similar in form and function to :ref:`Make`.

.. index:: BUILD

BUILD

A number of tools use (incompatible) BUILD files to describe software builds:

.. index:: Google Blaze

.. index:: Blaze

Blaze

Blaze is an internal software build automation tool developed by Google.

.. index:: Bazel

Bazel

Bazel is a free and open source software build automation tool developed as a rewrite of Google :ref:`Blaze`.

  • A WORKSPACE or WORKSPACE.bazel file indicates the root of a Bazel workspace.
  • Bazel uses :ref:`BUILD` (or BUILD.bazel) files to describe builds.
  • :ref:`Buck` was released before :ref:`Bazel` was open-sourced.
.. index:: BuckBuild

.. index:: Buck

Buck

  • Buck uses BUCK files to describe builds.
.. index:: CMake

CMake

CMake is a free and open source software build automation tool.

  • CMake generates build configurations for a number of tools: Unix Makefiles, Ninja, Visual Studio
.. index:: Gradle

Gradle

Gradle is a build tool for the :ref:`Java` :ref:`JVM` which builds a directed acyclic graph (DAG).

.. index:: Grunt

Grunt

Grunt is a build tool written in :ref:`Javascript` which builds a directed acyclic graph (DAG).

.. index:: Gulp

Gulp

Gulp is a build tool written in :ref:`Javascript` which builds a directed acyclic graph (DAG).

.. index:: Jake

Jake

Jake is a :ref:`Javascript` build tool written in :ref:`Javascript` (for :ref:`Node.js`) similar to :ref:`Make` or :ref:`Rake`.

.. index:: Make

Make

GNU Make is a classic, ubiquitous software build automation tool designed for file-based source code compilation which builds a directed acyclic graph (DAG).

:ref:`Bash`, :ref:`Python`, and the GNU/:ref:`Linux` kernel are all built with Make.

Make build task chains are represented in a Makefile.

Pros

  • Simple, easy to read syntax
  • Designed to build files on disk (see: .PHONY)
  • Nesting: make -C <path> <taskname>
  • Variable Syntax: $(VARIABLE_NAME) or ${VARIABLE_NAME}
  • Bash completion: make <tab>
  • Python: Initially parseable with disutils.text_file
  • Logging: command names and values print to stdout (unless prefixed with @)

Cons

  • Platform Portability: make is not installed everywhere
  • Global Variables: parametrization with shell scripts
VARIABLE_NAME="value" make test
make test VARIABLE_NAME="value"

# ...
export VARIABLE_NAME="value"
make test
.. index:: Pants

.. index:: Pants Build

Pants Build

Pants Build is a build tool for :ref:`JVM` [:ref:`Java`, :ref:`Scala`, :ref:`Android`], :ref:`C++`, :ref:`Go`, :ref:`Haskell`, :ref:`Node`, and :ref:`Python` [:ref:`CPython`] software projects.

.. index:: Virtualization

Virtualization

.. index:: Cgroups

Cgroups

Cgroups are a :ref:`Linux` mechanism for containerizing groups of processes and resources.

.. index:: libcontainer

libcontainer

libcontainer is a library built by :ref:`Docker` to replace :ref:`LXC`.

Libcontainer provides a native :ref:`Go` implementation for creating containers with namespaces, :ref:`cgroups`, capabilities, and filesystem access controls.

—https://github.com/opencontainers/runc/tree/master/libcontainer

.. index:: OCI

.. index:: Open Container Initiative

Open Container Initiative

The Open Container Initiative (OCI) is a Linux Foundation collaborative project dedicated to developing a working, portable software container specification.

.. index:: runC

runC

runC is a container abstraction

runc is a CLI tool for spawning and running containers according to the :ref:`OCI` specification.
.. index:: Docker

Docker

Docker is an OS virtualization project written in :ref:`Go` which utilizes :ref:`Linux` containers -- first :ref:`LXC` now :ref:`libcontainer` / :ref:`runC` -- to partition process workloads across one or more host systems.

.. glossary::

    Dockerfile
        A ``Dockerfile`` contains the instructions needed to
        create a docker image.

    Docker container
        A Docker container is an instance of a :term:`Docker Image`
        with configuration.

    Docker API
        The Docker API is an interface of management commands for
        provisioning and managing containers.

        :term:`Docker Machine`, :term:`Docker Swarm`, and
        :term:`Docker Universal Control Plane` all implement the Docker
        API; so the ``docker`` client works equally well with each
        implementation.

    Docker Machine
        Docker Machine is the container management application
        which implements the :term:`Docker API`.

    Docker Swarm
        Docker Swarm is a cluster management system for Docker
        containers hosted on one or more :term:`Docker Machines <Docker
        Machine>`

    Docker Universal Control Plane
        Docker Universal Control Plane is an enterprise-grade cluster
        management solution with a web dashboard and external authentication
        which implements the :term:`Docker API`.

    Docker Compose
        Docker Compose is a Python application for defining and managing
        services (:term:`Docker containers <docker container>`) and
        networks with a ``docker-compose.yml`` :ref:`YAML` configuration
        file.

    Docker Image
        A Docker Image is an archived container filesystem with
        configuration which is usually defined by a :term:`Dockerfile`.

    Docker Hub
        Docker Hub is a cloud-based registry service for
        :term:`Docker Images <Docker Image>`.

    Docker Cloud
        Docker Cloud is the hosting service offered by Docker.
  • Docker images build from a :term:`Dockerfile`
  • A Dockerfile can subclass another Dockerfile (to add, remove, or change configuration)
  • Dockerfile support a limited number of commands
  • Docker is not intended to be a complete :ref:`configuration management system <configuration management>`
  • Ideally, a Docker image requires minimal configuration once built
  • Docker images can be hosted by https://hub.docker.com/
  • docker run -it ubuntu/16.04 downloads the image from https://hub.docker.com/_/ubuntu/, creates a new instance (docker ps), and spawns a root :ref:`Shell <shells>` with a UUID name (by default).
  • There are a number of ways to "Schedule" [redundant] persistent containers that launch on boot with :ref:`Docker`
    • Docker Swarm is the Docker-native way to run a cluster of containers. To a client app, Docker Swarm looks just like Docker Machine because it implements the Docker API.
    • :ref:`Kubernetes` is one project which uses Docker to schedule redundant, optionally geodistributed, :ref:`LXC` containers (in "Pods").

:ref:`Salt` can install and manage docker, docker images and containers:

.. index:: CNCF

.. index:: Cloud Native Computing Foundation

Cloud Native Computing Foundation

The Cloud Native Computing Foundation (CNCF) is a foundation for cloud and container industry collaboration.

.. index:: Kubernetes

Kubernetes

Kubernetes (k8s) is a highly-available distributed cluster scheduler which works with groups of :ref:`Docker` containers called Pods.

.. index:: k3s

k3s

k3s is a lightweight :ref:`Kubernetes` distribution which runs on x86-64, ARM6, ARM7; only requires 512Mb of RAM; and is distributed as a single :ref:`Go` binary.

  • You've heard of k8s? This is k8s - 5.
.. index:: KVM

KVM

KVM is a full virtualization platform with support for Intel VT and AMD-V; which supports running various guest operating systems, each with their own kernel, on a given host machine.

.. index:: Libcloud

Libcloud

Apache libcloud is a :ref:`Python` library which abstracts and unifies a large number of Cloud APIs for Compute Resources, Object Storage, Load Balancing, and DNS.

:ref:`Salt` :term:`salt cloud` depends upon libcloud.

.. index:: Libvirt

Libvirt

Libvirt is a system for platform virtualization with various :ref:`Linux` hypervisors.

.. index:: LXC

LXC

LXC ("Linux Containers"), written in :ref:`C`, builds upon :ref:`Linux` :ref:`Cgroups` to provide containerized OS chroots (all running under :ref:`the host kernel <linux>`).

LXC is included in recent :ref:`Linux` kernels.

.. index:: LXD

LXD

LXD, written in :ref:`Go`, builds upon :ref:`LXC` to provide a system-wide daemon and an :ref:`OpenStack` Nova hypervisor plugin.

.. index:: Mesos

Mesos

Apache Mesos is a highly-available distributed datacenter operating system, for which there are many different task/process/service schedulers.

Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively.
.. index:: Mesosphere

Mesosphere

.. index:: OpenStack

OpenStack

OpenStack is a platform of infrastructure services for running a cloud datacenter (a private or a public cloud).

OpenStack makes it possible for end-users to create a new virtual machine from the available pool of resources.

rdfs:seeAlso: :ref:`openstack-devstack`, :ref:`Libcloud`

.. index:: OpenStack DevStack

OpenStack DevStack

OpenStack DevStack is a default development configuration for :ref:`OpenStack`.

There are many alternatives to and implementations of OpenStack DevStack:

.. index:: Packer

Packer

Packer generates machine images for multiple platforms, clouds, and hypervisors from a parameterizable template.

.. glossary::

   Packer Artifact
      Build products: machine image and manifest

   Packer Template
      JSON build definitions with optional variables and templating

   Packer Build
      Task defined by a JSON file containing build steps
      which produce a machine image

   Packer Builder
      Packer components which produce machine images
      for one of many platforms:

      - :ref:`VirtualBox`
      - :ref:`Docker`
      - :ref:`OpenStack`
      - GCE
      - EC2
      - VMware
      - QEMU (:ref:`KVM`, Xen)
      - https://www.packer.io/docs/templates/builders.html

   Packer Provisioner
      Packer components for provisioning machine images at build time

      - Shell scripts
      - File uploads
      - ansible
      - chef
      - solo
      - puppet
      - salt

   Packer Post-Processor
      Packer components for compressing and uploading built machine images




.. index:: Vagrant

Vagrant

Vagrant is a tool written in :ref:`Ruby` for creating and managing virtual machine instances with CPU, RAM, Storage, and Networking.

vagrant help
vagrant status
vagrant init ubuntu/trusty64
vagrant up
vagrant ssh
$EDITOR Vagrantfile
vagrant provision
vagrant halt
vagrant destroy

.. glossary::

   vagrantfile
      vagrant script defining a team of one or more
      virtual machines and networks.

      create a vagrantfile::

         vagrant init [basebox]
         cat vagrantfile

      start virtual machines and networks defined in the vagrantfile::

         vagrant status
         vagrant up

   Vagrant Box
      Vagrant base machine virtual machine image.

      There are many baseboxes for various operating systems.

      Essentially a virtual disk plus CPU, RAM, Storage, and Networking
      metadata.

      Locally-stored and cached vagrant boxes can be listed with::

         vagrant help box
         vagrant box list

      A running vagrant environment can be packaged into a new box with::

         vagrant package

      :ref:`Packer` generates :ref:`VirtualBox` Vagrant Boxes
      with a Post-Processor.

   Vagrant Cloud
      Vagrant-hosted public Vagrant Box storage.

      Install a box from Vagrant cloud::

         vagrant init ubuntu/trusty64
         vagrant up
         vagrant ssh

   Vagrant Provider
      A driver for running Vagrant Boxes with a hypervisor or in a cloud.

      The Vagrant :ref:`VirtualBox` Provider is well-supported.

      With Plugins: https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins

      See also: :ref:`libcloud`.

   Vagrant Provisioner
      Set of hooks to install and run shell scripts and
      configuration managment tools over ``vagrant ssh``.

      Vagrant up runs ``vagrant provision`` on first invocation of
      ``vagrant up``.

      ::

         vagrant provision

Note

Vagrant configures a default NFS share mounted at /vagrant.

Note

Vagrant adds a default NAT Adapter as eth0; presumably for DNS, the default route, and to ensure vagrant ssh connectivity.

.. index:: VirtualBox

VirtualBox

Src: svn svn://www.virtualbox.org/svn/vbox/trunk

Oracle VirtualBox is a platform virtualization package for running one or more guest VMs (virtual machines) within a host system.

VirtualBox:

  • runs on many platforms: :ref:`Linux`, OSX, Windows
  • has support for full platform NX/AMD-v virtualization
  • requires matching kernel modules

:ref:`Vagrant` scripts VirtualBox.

.. index:: Shells

Shells

.. index:: Bash

Bash

GNU Bash, the Bourne-again shell, is an open source command-line program written in :ref:`C` for running commands in a text-based terminal.

A few commands to try when learning to shell with Bash:

echo $SHELL; echo "$SHELL"; echo "${SHELL}"
type bash
bash --help
help help
help type
apropos bash
info bash
man bash

man man
info info  # [down arrow] and then [enter] to select, or 'n' for next

  • Bash works with unix command outputs and return codes: a program returns nonzero when there is an error:

    true;  echo $?  # 0
false; echo $?  # 1
echo "Hello" && echo " World!"  # Hello World!
false || echo "World!"          # World!

  • Functions: Bash supports functions with arguments that can print to standard out and/or return an integer return code:

    function add_a {
   echo "$1 + $2 = $(( $1 + $2 ))"
}
add_b () {
   echo "$1 + $2 = $(( $1 + $2 ))"
}
add_xy () {
   echo "$x + $y = $(( $x + $y ))"
}
add_a 3 5       # "3 + 5 = 8"
add_b 3 5       # "3 + 5 = 8"

x=3 y=5 add_xy  # "3 + 5 = 8"
x=3; y=5;
add_xy          # "3 + 5 = 8"

output=$(add_a 3 5)
echo "${output}"

help test
help [
help [[
help return

test "$(add_a 3 5)" == "3 + 5 = 8" && echo 'OK'

test_add_a () {
   if [[ "$(add_a 3 5)" == "3 + 5 = 8" ]]; then
       echo 'OK'
       return 0
   else
       echo 'Test failed'
       return 1
   fi
}
test_add_a

help trap
help exit

  • Portability: sh (sh, bash, dash, zsh) shell scripts are mostly compatible; though bash supports some features that other shells do not.

  • Logging: You can configure bash to print commands and arguments as bash executes scripts:

    set -x  # print commands and arguments
set -v  # print source

Bash reads various configuration files at startup time:

/etc/profile
/etc/bash.bashrc
/etc/profile.d/*.sh
${HOME}/.profile        /etc/skel/.profile   # PATH=+$HOME/bin  # umask
${HOME}/.bash_profile   # empty. preempts .profile
${HOME}/.bashrc

Bash and various :ref:`Operating Systems`:

  • Linux: Bash is almost always installed as the default shell on Linux boxes.
  • Mac:
    • MacOS includes Bash 3.2.
    • You can brew install bash to get a more recent version. (:ref:`homebrew`)
  • Windows:
    • :ref:`Windows Subsystem for Linux` (WSL) installs Linux distributions which include bash.
    • You can also install bash on Windows by installing git with choco install git -y (:ref:`Chocolatey`)
    • You can also install bash on Windows by installing MSYS2 (Mingw) or Cygwin with choco install msys2 or choco install cygwin

While Bash is ubiquitous, shell scripts are loose with quoting; which makes shell scripts flexible but dangerous and thus often avoided in favor of other languages:

## Shell script quoting example 1:

# This prints a newline
echo $(echo "-e a\nb")

# This prints "-e a\nb"
echo "$(echo "-e a\nb")"

This isn't an issue with e.g. :ref:`Python` (a popular language that's also useful for system administration).

import subprocess
print(subprocess.check_output(['echo', "-e a\nb"])
print(subprocess.check_output('echo "-e a\nb"', shell=True))

# Though, note that Python subprocess shell=True is a security risk:
# - avoid shell=True
# - pass the command as a list of already-tokenized arguments
# - use something like sarge (or ansible) instead of shell=True

:ref:`IPython` is one of many alternatives to Bash.

.. index:: Readline

Readline

.. index:: IPython

.. index:: ipython

IPython

IPython is an interactive REPL and distributed computation framework written in :ref:`Python`.

## Formatting expression output with the Python interpreter
1 + 1
x = 1+1
print("1 + 1 = 2")
print('1 + 1 = %d' % (x))
print('1 + 1 = {0}'.format(x))   # Python 2.7+
print('1 + 1 = {x}'.format(x=x)) # Python 2.7+
print(f'1 + 1 = {x}')            # Python 3.6+
print(f'{1 + 1 = }')             # Python 3.8+

## IPython
!ipython --help                # run `$SHELL -c 'ipython --help'`
!python -m IPython --help      # run `ipython --help`

?                              # print IPython help within IPython

%lsmagic
%<tab>                         # list magic commands and aliases
%paste?                        # help for the %paste magic command
%logstart?                     # help for the %logstart magic command
%logstart -o logoutput.log.py  # log input and output to a file

import json
json?                          # print(json.__doc__)
json??                         # print(inspect.getsource(json))

## IPython shell
!cat ./README.rst; echo $PWD   # run shell commands
lines = !ls -al                # capture shell command output
print(lines[0:])
%run -i -t example.py          # run a script with timing info,
                               # in the local namespace
%run -d example.py             # run a script with pdb
%pdb on                        # automatically run pdb on Exception

  • If a kernel is not specified, IPython uses the ipykernel Jupyter kernel.

  • To use other kernels with IPython, you must install jupyter_console and a kernel:

    pip install jupyter_console  # conda install -y jupyter_console

ipython console --kernel python  # ipykernel
jupyter console --kernel python  # ipykernel
# <Ctrl-D> | <Ctrl-C> | "exit()"

pip install bash_kernel  # conda install -y bash_kernel
jupyter console --kernel bash
# "exit"

conda install -y -c conda-forge xeus-cling
jupyter console --kernel xcpp11
# <Ctrl-D> (<Ctrl-Z> on Windows)

conda install -y nodejs; npm install -g ijavascript; ijsinstall
jupyter console --kernel javascript
# <Ctrl-D>

conda install -y nodejs; npm install -g jp-babel; jp-babel-install
jupyter console --kernel babel
# <Ctrl-D>

conda install -y nodejs; npm install -g itypescript; its --install=local
jupyter console --kernel typescript
# <Ctrl-D>

jupyter kernelspec list

  • There are very many Jupyter kernels: https://github.com/jupyter/jupyter/wiki/Jupyter-kernels

  • :ref:`Jupyter Notebook` and :ref:`Jupyter Lab` are built atop IPython: a Jupyter notebook file is a :ref:`JSON` file with an .ipynb extension which contains inputs and text and binary outputs.

.. index:: PowerShell

.. index:: Windows PowerShell

PowerShell

Windows PowerShell is a shell for :ref:`Windows`.

.. index:: ZSH

ZSH

.. index:: Shell Utilities

Shell Utilities

.. index:: Awk

Awk

AWK is a pattern programming language for matching and transforming text.

.. index:: Grep

Grep

Grep is a commandline utility for pattern-based text matching.

.. index:: Htop

Htop

Htop is a commandline task manager; like top extended.

.. index:: Pyline

Pyline

Pyline is an open source :ref:`POSIX` command-line utility for streaming line-based processing in :ref:`Python` with regex and output transform features similar to :ref:`grep`, :ref:`sed`, and :ref:`awk`.

.. index:: Pyrpo

Pyrpo

Pyrpo is an open source :ref:`POSIX` command-line utility for locating and generating reports from :ref:`Git`, :ref:`Mercurial`, :ref:`Bazaar`, and :ref:`Subversion` repositories.

.. index:: Sed

Sed

GNU Sed is an open source :ref:`POSIX` command-line utility for transforming text.

Note

BSD Sed

Use <Ctrl-V><tab> for explicit tabs (as \t does not work)

Use \\\n or '$'\n for newlines (as \n does not work)

sed -E should be consistent extended regular expressions between GNU Sed (e.g. Linux) and BSD Sed (FreeBSD, OSX).

OR: brew install gnu-sed

See: https://unix.stackexchange.com/questions/101059/sed-behaves-different-on-freebsd-and-on-linux

See: https://superuser.com/questions/307165/newlines-in-sed-on-mac-os-x

.. index:: Web Shells

Web Shells

.. index:: IPython Notebook

.. index:: IPython notebook

IPython Notebook

:ref:`IPython` Notebook (now :ref:`Jupyter Notebook`) is an open source web-based shell written in :ref:`Python` and :ref:`Javascript` for interactive and literate computing with IPython notebooks composed of raw, markdown, or code input and plaintext- or rich- output cells.

To start IPython Notebook (assuming the _SRC variable as defined in :ref:`Venv`):

pip install ipython[notebook]
# pip install -e git+https://github.com/ipython/[email protected]#egg=ipython
# https://github.com/ipython/ipython/releases

mkdir $_SRC/notebooks; cd $_SRC/notebooks
ipython notebook

ipython notebook --notebook-dir="${_SRC}/notebooks"

# With HTTPS (TLS/SSL)
ipython notebook \
 --ip=127.0.0.1 \
 --certfile=mycert.pem \
 --keyfile=privkey.pem \
 --port=8888 \
 --browser=web  # (optional) westurner/dotfiles/scripts/web

 # List supported options
 ipython notebook --help

Warning

IPython Notebook runs code and shell commands as the user the process is running as, on a remote or local machine.

Reproducible :ref:`SciPy Stack <scipystack>` IPython Notebook / :ref:`Jupyter Notebook` servers implement best practices like process isolation and privilege separation with e.g. :ref:`Docker` and/or :ref:`Jupyter` Hub.

Note

IPython Notebook is now :ref:`Jupyter Notebook`.

Jupyter Notebook runs Python notebooks with :ref:`ipykernel`, the :ref:`IPython` :ref:`Python` kernel from :ref:`IPython Notebook`

.. index:: Jupyter kernels
Jupyter kernels
.. index:: ipykernel
ipykernel
.. index:: ipython_nose
ipython_nose

ipython_nose is an extension for :ref:`IPython Notebook` (and likely :ref:`Jupyter Notebook`) for discovering and running test functions starting with test_ (and unittest.TestCase test classes with names containing Test) with Nose.

  • ipython_nose is not (yet?) uploaded to PyPI
  • to install ipython_nose from GitHub (with :ref:`Pip` and :ref:`Git`):
pip install -e git+https://github.com/taavi/ipython_nose#egg=ipython_nose

See also:

.. index:: nosebook

nosebook

nosebook is a tool for finding and running tests in :ref:`nbformat` :ref:`IPython Notebooks <ipython notebooks>` and :ref:`Jupyter Notebooks <jupyter notebook>` with nose.

See also:

.. index:: Jupyter

.. index:: Project Jupyter

Jupyter

Project Jupyter expands upon components like :ref:`IPython` and :ref:`IPython Notebook` to provide a multi-user web-based shell for many languages (:ref:`Python`, :ref:`Ruby`, :ref:`Java`, :ref:`Haskell`, Julia, R).

IPython Jupyter comparison (adapted from http://jupyter.org)
:ref:`IPython` :ref:`Jupyter`
  • Interactive Python shell
  • Python kernel for Jupyter
  • Interactive Parallel Python
  • Rich REPL Protocol
  • :ref:`Jupyter Notebook` (format, environment, conversion)
  • JupyterHub (multi-user notebook server)
  • JupyterHub authenticators (MediaWiki OAuth, GitHub OAuth)
  • JupyterHub spawners (Docker, Sudo, Remote, Docker Swarm)
 
.. index:: Jupyter Notebook
Jupyter Notebook

:ref:`Jupyter` Notebook is an open source shell webapp written in :ref:`Python` and :ref:`Javascript` for interactive and literate computing with Jupyter notebooks composed of raw, markdown, or code input and plaintext- or rich- output cells.

To start IPython Notebook (assuming the _SRC variable as defined in :ref:`Venv`):

pip install ipython[notebook]
# pip install -e git+https://github.com/ipython/[email protected]#egg=ipython
# https://github.com/ipython/ipython/releases

mkdir $_SRC/notebooks; cd $_SRC/notebooks
ipython notebook

ipython notebook --notebook-dir="${_SRC}/notebooks"

# With HTTPS (TLS/SSL)
ipython notebook \
 --ip=127.0.0.1 \
 --certfile=mycert.pem \
 --keyfile=privkey.pem \
 --port=8888 \
 --browser=web  # (optional) westurner/dotfiles/scripts/web

 # List supported options
 ipython notebook --help

Warning

IPython Notebook runs code and shell commands as the user the process is running as, on a remote or local machine.

Reproducible :ref:`SciPy Stack <scipystack>` IPython Notebook / :ref:`Jupyter Notebook` servers implement best practices like process isolation and privilege separation with e.g. :ref:`Docker` and/or :ref:`Jupyter` Hub.

Note

:ref:`JupyterLab` (a mostly-rewrite) adds e.g. tabs and undo (and a new extension API) to :ref:`Jupyter Notebook`.

.. index:: JupyterLab
JupyterLab

JupyterLab is an open-source web-based tabbed :ref:`IDE <IDE>` written in :ref:`Python`, :ref:`Javascript`, and :ref:`TypeScript` for working with :ref:`<jupyter notebooks> Jupyter Notebook`, terminals, text editing, undo, extensions.

  • You can edit Jupyter notebooks with JupyterLab.
  • Installing JupyterLab also installs :ref:`Jupyter Notebook`. (which doesn't support tabs or the new extension API)
  • A few UI differences between JupyterLab and Jupyter Notebook:
    • JupyterLab has tabbed editing: you can open files, notebooks, and terminals in tabs
    • JupyterLab has a sidebar with a file selector pane
  • Installing JupyterLab does not install any :ref:`scipystack` or other packages.
.. index:: JupyterLab (install)
Install JupyterLab

Install JupyterLab With :ref:`pip`:

python -m pip install jupyterlab

Install JupyterLab with :ref:`conda`:

conda install -c conda-forge -y jupyterlab
.. index:: Hosting JupyterLab

.. index:: JupyterLab (hosting)
Hosting JupyterLab

You can host JupyterLab yourself:

.. index:: Hosted JupyterLab

.. index:: JupyterLab (hosted)
Hosted JupyterLab

There are many providers of hosted :ref:`JupyterLab` and :ref:`Jupyter Notebook`; where they run Jupyter in a shell or a VM on their servers for you and you connect over your internet connection.

.. index:: JupyterHub
JupyterHub

JupyterHub makes it easy to serve :ref:`Jupyter Notebook` and/or :ref:`Jupyter Lab` for multiple users on one or more servers.

  • JupyterHub spawns individual Jupyter Notebook / JupyterLab server instances for logged-in users.
  • JupyterHub enables users to log-in with Authenticator backends: system users, LDAP, SSO, OAuth (e.g. Google accounts)
  • If so configured, JupyterHub can launch additional servers to serve one or more Notebook/Lab :ref:`Docker` containers and then shut those down when they're idle or, for example, when a course session is complete.
.. index:: nbconvert
nbconvert

nbconvert is the code that converts (transforms) an .ipynb notebook (:ref:`nbformat` :ref:`JSON <json->`) file into an output representation (e.g. :ref:`HTML`, HTML slides (:ref:`reveal.js`), :ref:`LaTeX`, PDF, ePub, Mobi).

  • nbconvert is included with :ref:`Jupyter Notebook` and :ref:`JupyterLab`

    pip install nbconvert
# pip install -e git+https://github.com/jupyter/nbconvert@master#egg=nbconvert

jupyter nbconvert --to html mynotebook.ipynb
.. index:: reveal.js
reveal.js

reveal.js is a :ref:`Javascript` and :ref:`HTML` library for slide presentations served from an HTML file.

  • Reveal.js slides can be in a 1-dimensional or a 2-dimensional arrangement.

  • You can generate reveal.js slides from Jupyter notebooks in two ways: with nbconvert --to slides or with the GUI: "File" > "Export Notebok As..." > "Export Notebook to reveal.js slides"

    jupyter nbconvert --to slides mynotebook.ipynb

    Note

    Presentation content that doesn't fit on a slide is hidden and unscrollable: only put a slide worth of data in each cell for a Jupyter reveal.js presentation.

    Alternatives to presenting notebooks as reveal.js slides:

    • Increase the browser font size (Jupyter Notebook)
    • "View" > "Presentation Mode" (JupyterLab)
    • Select a keyboard shortcut set use the "Select Cell Below" / "Select Cell Above" keyboard shortcuts to highlight cells and scroll them into view
      • Press "<Escape>"
      • Press "j" to "Select Cell Below"
      • Press "k" to "Select Cell Above"

  • The :ref:`RISE` extension also generates reveal.js slides.

.. index:: RISE
RISE

RISE is a :ref:`Jupyter Notebook` and :ref:`JupyterLab` extension that generates live :ref:`reveal.js` presentations from Jupyter notebooks.

  • Install the RISE extension
  • Click the RISE button to generate a live :ref:`reveal.js` slide presentation wherein you can execute cells on the slides with "Ctrl-Enter" and "Shift-Enter" just like you can in the Notebook interface.
.. index:: nbformat
nbformat

The :ref:`Jupyter notebook` (.ipynb) format is a versioned :ref:`JSON <json->` format for storing metadata and input/output sequences.

Usually, when the nbformat changes, notebooks are silently upgraded to the new version on the next save.

Note

nbformat v3 and above add a kernelspec attribute to the nbformat :ref:`JSON <json->`, because .ipynb files can now contain code for languages other than :ref:`Python`.

  • nbformat does not specify any schema for the user-supplied metadata dict (TODO: :ref:`nbmeta`), so JSON that conforms to an externally managed :ref:`JSON-LD <json-ld->` @context would work.
.. index:: nbgrader
nbgrader

nbgrader is a solution for centrally receiving and grading :ref:`Jupyter notebooks <jupyter notebook>`.

  • You mark notebook cells as TODO

See also:

.. index:: nbviewer
nbviewer

(nbviewer) is an application for serving read-only versions of Jupyter notebooks from HTTP URLs.

  • When you enter a URL, GitHub username, GitHub username/repo, or Gist ID into the text box at https://nbviewer.jupyter.org/ and click 'Go!' (or press Enter), nbviewer nbconverts the notebook to HTML or shows a file browser and branch/tag selector for the git repo.
  • You do not need to look up the raw GitHub URL for the notebook, because nbviewer automatically rewrites the GitHub /blob/ file URL to a raw.githubusercontent.com URL.
  • GitHub now also renders static .ipynb files, CSV, SVG, and PDF. However, GitHub does not execute any JS in the notebook due to security concerns (XSS)
  • GitLab renders Jupyter notebooks with JS.
.. index:: runipy
runipy

runipy runs :ref:`Jupyter notebooks <jupyter notebook>` from a :ref:`Shell <shells>` commandline, generates HTML reports, and can write errors to stderr.

:ref:`Jupyter notebook <Jupyter Notebook>` manual test review process:

# - run Jupyter Notebook server
!jupyter notebook
# - Browser
#     - navigate to / upload / drag and drop the notebook
        !web http://localhost:8888   # or https://
#     - (optional) click 'TODO Restart Kernel'
#     - (optional) click 'Cell' > 'All Output' > 'Clear'
#     - click 'Cell' > 'Run All'
#     - [wait] <Jupyter Kernel runs notebook>
#     - visually seek for the first ERRoring cell (scroll)
#     - review the notebook
        for (i, o) in notebook_cells:
            human.manually_review((i, o))
# - Compare the files on disk with the most recent commit (HEAD)
!git status && git diff
!git diff mynotebook.ipynb
# - Commit the changes
!git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb

:ref:`Jupyter notebook <Jupyter Notebook>` TODO review process:

# - run Jupyter Notebook server
!jupyter notebook
# - Browser
#     - navigate to / upload / drag and drop the notebook
        !web http://localhost:8888   # or https://
#     - (optional) click 'TODO Restart Kernel'
#     - (optional) click 'Cell' > 'All Output' > 'Clear'
#     - click 'Cell' > 'Run All'
#     - [wait] <Jupyter Kernel runs notebook>
#     - visually seek for the first ERRoring cell (scroll)
#     - review the notebook
        for (i, o) in notebook_cells:
            human.manually_review((i, o))
# - Compare the files on disk with the most recent commit (HEAD)
!git status && git diff
!git diff mynotebook.ipynb
# - Commit the changes
!git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb

:ref:`Jupyter notebook <Jupyter Notebook>` runipy review process:

# - runipy the Jupyter notebook
!runipy mynotebook.ipynb
# - review stdout and stderr from runipy
# - review in browser (optional; recommended)
#     - navigate to the converted HTML
        !web ./mynotebook.ipynb.html
#     - visually seek for the first WEEoring cell (scroll)
#     - review the notebook
        for (i, o) in notebook_cells:
            human.manually_review((i, o))
# - Compare the files on disk with the most recent commit (HEAD)
!git status && git diff
!git diff mynotebook.ipynb*
# - Commit the changes
!git-add-commit "TST: mynotebook: tests for #123" ./mynotebook.ipynb*

.. index:: jupyter_contrib_nbextensions
jupyter_contrib_nbextensions
.. index:: NBPresent
NBPresent
remix your :ref:`Jupyter Notebooks <jupyter notebook>` as interactive slideshows
.. index:: Anaconda Jupyter Notebook Extensions
Anaconda Jupyter Notebook Extensions
.. index:: CoCalc

.. index:: SageMathCloud

CoCalc

.. index:: Google Colab

.. index:: Colab

Google Colab

Google Colab is a hosted :ref:`Jupyter Notebook` system.

  • Colab has a number of packages installed in the default environment. If you want additional packages, you need to !pip install them once when you first open the notebook.
  • Colab is forked from a previous version of Jupyter Notebook, and so does not have some newer Jupyter Notebook or any Jupyter Lab features.
  • :ref:`ipywidgets` are not yet implemented on Colab.
  • Colab saves to Google Drive.
  • Colab instances are free and can use some GPU time if needed.
  • There is a Colab Pro.
  • Google AI Platform Notebooks hosts :ref:`JupyterLab` notebooks: https://cloud.google.com/ai-platform-notebooks 
.. index:: Dotfiles

Dotfiles

Dotfiles are userspace :ref:`shell <shells>` configuration in files that are often prefixed with "dot" (e.g. ~/.bashrc for :ref:`Bash`)

.. index:: Dotvim

Dotvim

Dotvim is a conjunction / contraction of :ref:`Dotfiles` and :ref:`Vim` (in reference to a ~/.vim/ directory and/or a ~/.vimrc).

.. index:: Venv

Venv

Venv is a tool for making working with :ref:`Virtualenv`, :ref:`Virtualenvwrapper`, :ref:`Bash`, :ref:`ZSH`, :ref:`Vim`, and :ref:`IPython` within a project context very easy.

Venv defines standard :ref:`fhs` and :ref:`Python` paths, environment variables, and aliases for routinizing workflow.

Venv paths and cdaliases
var name description

cdaliases

Bash: cdhelp

IPython: %cdhelp

Vim: :Cdhelp

 example path
HOME user home directory

Bash/ZSH: cdh, cdhome

IPython: %cdh, %cdhome

Vim: :Cdh, :Cdhome

 ~/
__WRK workspace root cdwrk (ibid.) ~/-wrk
WORKON_HOME virtualenvs root cdwh, cdworkonhome, cdve ~/-wrk/-ve27
CONDA_ENVS_PATH condaenvs root cdch, cdcondahome ~/-wrk/-ce27
VIRTUAL_ENV virtualenv root cdv, cdvirtualenv ~/-wrk/-ve27/dotfiles
_BIN virtualenv executables cdb, cdbin ~/-wrk/-ve27/dotfiles/bin
_ETC virtualenv configuration cd, cdetc ~/-wrk/-ve27/dotfiles/etc
_LIB virtualenv lib directory cdl, cdlib ~/-wrk/-ve27/dotfiles/lib
_LOG virtualenv log directory cdlog ~/-wrk/-ve27/dotfiles/var/log
_SRC virtualenv source repositories cds, cdsrc ~/-wrk/-ve27/dotfiles/src
_WRD virtualenv working directory cdw, cdwrd ~/-wrk/-ve27/dotfiles/src/dotfiles

To generate this venv config:

python -m dotfiles.venv.ipython_config --print-bash dotfiles
venv.py --print-bash dotfiles
venv --print-bash dotfiles docs
venv --print-bash dotfiles ~/path
venv --print-bash ~/-wrk/-ve27/dotfiles ~/path

To generate a default venv config with a prefix of /:

venv --print-bash --prefix=/

To launch an interactive shell within a venv:

venv --run-bash dotfiles
venv -xb dotfiles

Note

pyvenv is the :ref:`Virtualenv` -like functionality now included in :ref:`Python >= 3.3 <python3>` (python3 -m venv)

Python pyvenv docs: https://docs.python.org/3/library/venv.html

.. index:: Virtualenv

Virtualenv

Virtualenv is a tool for creating reproducible :ref:`Python` environments.

Virtualenv sets the shell environment variable $VIRTUAL_ENV when active.

Virtualenv installs a copy of :ref:`Python`, :ref:`Setuptools`, and :ref:`Pip` when a new virtualenv is created.

A virtualenv is activated by source-ing ${VIRTUAL_ENV}/bin/activate.

Paths within a virtualenv are more-or-less :ref:`FHS <fhs>` standard paths, which makes virtualenv structure very useful for building chroot and container overlays.

A standard virtual environment:

bin/           # pip, easy_install, console_scripts
bin/activate   # source bin/activate to work on a virtualenv
include/       # (symlinks to) dev headers (python-dev/python-devel)
lib/           # libraries
lib/python2.7/distutils/
lib/python2.7/site-packages/  # pip and easy_installed packages
local/         # symlinks to bin, include, and lib
src/           # editable requirements (source repositories)

# also useful
etc/           # configuration
var/log        # logs
var/run        # sockets, PID files
tmp/           # mkstemp temporary files with permission bits
srv/           # local data

:ref:`Virtualenvwrapper` wraps virtualenv.

echo $PATH; echo $VIRTUAL_ENV
python -m site; pip list

virtualenv example               # mkvirtualenv example
source ./example/bin/activate    # workon example

echo $PATH; echo $VIRTUAL_ENV
python -m site; pip list

ls -altr $VIRTUAL_ENV/lib/python*/site-packages/**  # lssitepackages -altr

Note

Python 3.3+ now also contain a script called venv, which performs the same functions and works similarly to virtualenv: https://docs.python.org/3/library/venv.html.

.. index:: Virtualenvwrapper

Virtualenvwrapper

Virtualenvwrapper is a tool which extends virtualenvwrapper.

Virtualenvwrapper provides a number of useful shell commands and python functions for working with and within :ref:`virtualenvs <virtualenv>`, as well as project event scripts (e.g. postactivate, postmkvirtualenv) and two filesystem configuration variables useful for structuring development projects of any language within :ref:`virtualenvs <virtualenv>`: $PROJECT_HOME and $WORKON_HOME.

Virtualenvwrapper is sourced into the shell:

# pip install --user --upgrade virtualenvwrapper
source ~/.local/bin/virtualenvwrapper.sh

# sudo apt-get install virtualenvwrapper
source /etc/bash_completion.d/virtualenvwrapper

echo $PROJECT_HOME; echo ~/workspace             # venv: ~/-wrk
cd $PROJECT_HOME                                 # venv: cdp; cdph
echo $WORKON_HOME;  echo ~/.virtualenvs          # venv: ~/-wrk/-ve27
cd $WORKON_HOME                                  # venv: cdwh; cdwrk

mkvirtualenv example
workon example                                   # venv: we example

cdvirtualenv; cd $VIRTUAL_ENV                    # venv: cdv
echo $VIRTUAL_ENV; echo ~/.virtualenvs/example   # venv: ~/-wrk/-ve27/example

mkdir src ; cd src/                              # venv: cds; cd $_SRC

pip install -e git+https://github.com/westurner/dotfiles#egg=dotfiles

cd src/dotfiles; cd $VIRTUAL_ENV/src/dotfiles    # venv: cdw; cds dotfiles
head README.rst

                                                 # venv: cdpylib
cdsitepackages                                   # venv: cdpysite
lssitepackages

deactivate
rmvirtualenv example

lsvirtualenvs; ls -d $WORKON_HOME                # venv: lsve; lsve 'ls -d'
.. index:: Window Managers

Window Managers

.. index:: Compiz

Compiz

Compiz is a window compositing layer for :ref:`X11` which adds lots of cool and productivity-enhancing visual capabilities.

Compiz works with :ref:`Gnome`, :ref:`KDE`, and :ref:`Qt` applications.

.. index:: f.lux

f.lux

f.lux is a userspace utility for gradually adjusting the blue color channel throughout the day; or as needed.

  • A similar effect can be accomplished with the :ref:`X11` xgamma command (e.g. for :ref:`Linux` platforms where the latest f.lux is not yet available). A few keybindings from an :ref:`i3wm` configuration here:

    # [...] #L105
set $xgamma_reset    xgamma -gamma 1.0
set $xgamma_soft     xgamma -bgamma 0.6 -ggamma 0.9 -rgamma 0.9
set $xgamma_soft_red xgamma -bgamma 0.4 -ggamma 0.6 -rgamma 0.9
# [...] #L200
## Start, stop, and reset xflux
#  <alt> [         -- start xflux
bindsym $mod+bracketleft    exec --no-startup-id $xflux_start
#  <alt> ]         -- stop xflux
bindsym $mod+bracketright   exec --no-startup-id $xflux_stop
#  <alt><shift> ]  -- reset gamma to 1.0
bindsym $mod+Shift+bracketright  exec --no-startup-id $xgamma_reset
#  <alt><shift> [  -- xgamma -bgamma 0.6 -ggamma 0.9 -rgamma 0.9
bindsym $mod+Shift+bracketleft exec --no-startup-id $xgamma_soft
#  <alt><shift> \  -- xgamma -bgamma -0.4 -ggamma 0.4 -rgamma 0.9
bindsym $mod+Shift+p exec --no-startup-id $xgamma_soft_red

.. index:: Gnome

Gnome

.. index:: i3wm

i3wm

i3wm is a tiling window manager for :ref:`X11` (:ref:`Linux`) with extremely-configurable :ref:`Vim`-like keyboard shortcuts.

i3wm works with :ref:`Gnome`, :ref:`KDE`, and :ref:`Qt` applications.

.. index:: KDE

KDE

KDE is a GUI framework built on :ref:`Qt`.

KWin is the main KDE window manager for :ref:`X11`.

.. index:: Qt

Qt

Qt is a Graphical User Interface toolkit for developing applications with Android, iOS, :ref:`OSX`, Windows, Embedded :ref:`Linux`, and :ref:`X11`.

.. index:: Wayland

Wayland

Wayland is a display server protocol for GUI window management.

Wayland is an alternative to :ref:`X11` servers like XFree86 and X.org.

The reference Wayland implementation, Weston, is written in :ref:`C`.

.. index:: X Window System

.. index:: X11

X11

Src: git git://anongit.freedesktop.org/git/xorg/

X Window System (X, X11) is a display server protocol for window management (drawing windows on the screen).

Most UNIX and :ref:`Linux` systems utilize XFree86 or the newer X.org X11 window managers.

:ref:`Gnome`, :ref:`KDE`, :ref:`I3wm`, :ref:`OSX`, and :ref:`Compiz` build upon X11.

.. index:: Browsers

Browsers

.. index:: Blink

Blink

Blink is a :ref:`web browser <browsers>` layout engine written in :ref:`C++` which was forked from :ref:`WebKit`.

.. index:: Chrome

Chrome

Google Chrome is a Web Browser built from the open source :ref:`Chromium` browser.

  • Google Chrome is now based on :ref:`Blink`.
  • Google Chrome was based on :ref:`WebKit`.
  • Google Chrome includes and updates Adobe Flash, pdf.js

See also: :ref:`ChromeOS`.

.. index:: Chromium

Chromium

The Chromium Projects include the :ref:`Chromium` Browser and :ref:`chromiumos`.

.. index:: Chrome DevTools
Chrome DevTools

How to open Chrome (and Firefox) DevTools:

  • Right-click > "Inspect Element"
  • Linux: <ctrl><shift>i
  • OSX: <option><command>i

DevTools Emulation

.. index:: Chrome Extensions
Chrome Extensions

Accessibility

Safety

Content

Tab

Development

Vim

.. index:: pbm
pbm
.. index:: Chrome Android

Chrome Android

Extensions

Chrome Android does not support extensions.

.. index:: Wandroid
Wandroid
.. index:: Firefox

Firefox

.. index:: Firefox Extensions
Firefox Extensions

Accessibility

Safety

Content

Tabs

Development

Vim

.. index:: Firefox Android

Firefox Android

.. index:: Firefox Android Extensions
Firefox Android Extensions
.. index:: Internet Explorer

Internet Explorer

Internet Explorer is the web browser included with :ref:`Windows`.

See also: :ref:`Microsoft Edge`

.. index:: Microsoft Edge

Microsoft Edge

Microsoft Edge will be replacing :ref:`Internet Explorer`.

.. index:: Opera

Opera

Opera is a multi-platform :ref:`web browser <browsers>` written in :ref:`C++`.

.. index:: Safari

Safari

Safari is the web browser included with :ref:`OSX`.

.. index:: Safari Extensions
Safari Extensions

Safety

Content

Development

.. index:: Safari iOS

Safari iOS

.. index:: WebKit

WebKit

WebKit is an open source :ref:`web browser <browsers>` written in :ref:`C++`.

.. index:: Browser Extensions

Browser Extensions

.. index:: Browser Extensions > Accessibility

Accessibility Extensions

.. index:: Accessibility Developer Tools
Accessibility Developer Tools
.. index:: ChromeVox
ChromeVox
.. index:: Dark Reader
Dark Reader
.. index:: Deluminate
Deluminate
.. index:: GitHub Dark Theme
GitHub Dark Theme
.. index:: High Contrast
High Contrast
.. index:: NASA Night Launch
NASA Night Launch
.. index:: ShadowFox
ShadowFox
.. index:: Spectrum
Spectrum
.. index:: Stylish
Stylish
.. index:: Tiësto
Tiësto

The Tiësto Chrome Theme is a Dark Theme for Chrome.

.. index:: Browser Extensions > Safety

.. index:: Safety Extensions

Safety Extensions

.. index:: Facebook Container
Facebook Container
.. index:: Firefox Multi-Account Containers
Firefox Multi-Account Containers
.. index:: Ghostery
Ghostery
.. index:: HTTPS Everywhere
HTTPS Everywhere

HTTPS Everywhere is a browser extension that forces the browser to only connect over HTTPS to sites listed in its database.

.. index:: uBlock
uBlock
_repo="chrisaljoudi/ublock"
curl -Ls "https://api.github.com/repos/${_repo}/releases" > ./releases.json
cat releases.json \
    | grep browser_download_url \
    | pyline 'w and w[1][1:-1]' \
    | pyline --regex \
        '.*download/(.*)/(uBlock.(firefox.xpi|chromium.zip))$' \
        'rgx and rgx.group(1,2)'
.. index:: uBlock Origin
uBlock Origin
.. index:: Browser Extensions > Content

.. index:: Content Extensions

Content Extensions

.. index:: Hypothesis
Hypothesis

Hypothesis can also be included as a sidebar on a site:

<script async defer src="//hypothes.is/embed.js"></script>
.. index:: OpenLink Structured Data Sniffer
OpenLink Structured Data Sniffer
.. index:: Pocket
Pocket
.. index:: Zotero
Zotero

Zotero archives and tags resources with bibliographic metadata.

  • Zotero is really helpful for research.

  • Browsers other than Firefox connect to Zotero Standalone

  • Zotero can store a full-page archive of a given resource (e.g. HTML, PDF)

  • Zotero can store and synchronize data on Zotero's servers with Zotero File Storage

  • Zotero can store and synchronize data over WebDAV

  • Zotero can export a collection of resources' bibliographic metadata in one of many citation styles ("CSL") (e.g. MLA, APA, [Journal XYZ])

  • Zotero can export a collection of resources' bibliographic metadata as RDF

  • There are a number of plugins and integrations with Zotero:

    https://www.zotero.org/support/plugins

.. index:: Zotero and Schema.org RDFa
[ ] Zotero and Schema.org RDFa

> How would I go about adding HTML + RDFa [1] and/or HTML + Microdata [2] export templates with Schema.org classes and properties to Zotero?

.. index:: Browser Extensions > Tabs

.. index:: Tab Extensions

Tab Extensions

.. index:: better-onetab
better-onetab
.. index:: FoxyTab
FoxyTab
.. index:: OneTab
OneTab
.. index:: Snipe
Snipe
.. index:: Tabs Outliner
Tabs Outliner
.. index:: Tree Style Tab
Tree Style Tab
.. index:: Browser Extensions > Development

.. index:: Development Extensions

Development Extensions

.. index:: AngularJS Batarang
AngularJS Batarang
.. index:: FireBug
FireBug
.. index:: FireLogger
FireLogger
.. index:: FireLogger for Chrome
FireLogger for Chrome

See: FireLogger

.. index:: JSONView
JSONView
.. index:: ng-inspector for AngularJS

.. index:: AngularJS ng-inspector
ng-inspector for AngularJS
.. index:: Postman
Postman
.. index:: Preact Devtools
Preact Devtools
.. index:: React Developer Tools
React Developer Tools
.. index:: Refined Github
Refined Github
.. index:: Responsive Web Design Tester
Responsive Web Design Tester

See also: :ref:`Chrome DevTools <chrome-devtools>` Emulation

.. index:: Requirify
Requirify

Requirify adds NPM modules to the local namespace (e.g. from Chrome DevTools JS console).

> require() npm modules in the browser console

.. index:: local-requirify
Local-requirify

Require local NPM modules with Requirify

.. index:: Web Developer Extension
Web Developer

Web Developer Extension, originally just for Firefox, adds many useful developer tools and bookmarklets in a structured menu.

.. index:: Window Resizer
Window Resizer

See also: :ref:`Chrome DevTools <chrome-devtools>` Emulation

.. index:: Vim Extensions

Vim Extensions

.. index:: Vimium
Vimium

Vimium is a Chrome Extension which adds :ref:`vim`-like functionality.

Vimium shortcuts
function vimium shortcut
help ?
jump to link in current/New tab f / F
copy link to clipboard yf
open clipboard link in current/New tab p / P
...   
.. index:: Vimperator
Vimperator

Vimperator connects a JS shell with VIM command interpretation to the Firefox API, with :ref:`vim`-like functionality.

  • vimperatorrc can configure settings in about:config
  • Vimperator stopped working after Firefox 57
.. index:: Wasavi
Wasavi

Wasavi converts the focused textarea to an in-page editor with :ref:`vim`-like functionality.

.. index:: Web Servers

Web Servers

.. index:: ACME

ACME

.. index:: LetsEncrypt
LetsEncrypt
.. index:: Apache HTTPD

Apache HTTPD

Apache HTTPD is a scriptable, industry-mainstay HTTP server written in :ref:`C` and :ref:`C++`.

.. index:: BusyBox HTTPD

BusyBox HTTPD

busybox httpd --help
busybox httpd -p 8082

See also: :ref:`python http.server`

.. index:: Caddy

Caddy

.. index:: Netcat

Netcat web server

# Serve a file over HTTP then close
{ printf 'HTTP/1.0 200 OK\r\nContent-Length: %d\r\n\r\n' "$(wc -c < some.file)"; cat some.file; } | nc -l 8082

# Serve the date over HTTP then start another server
while true ; do nc -l -p 8082 -c 'echo -e "HTTP/1.1 200 OK\n\n $(date -Is)"'; done &

# Make an HTTP request with netcat
printf "GET / HTTP/1.0\r\nHost: localhost\r\n\r\n" | nc localhost 8082

# Make an HTTP request with curl
curl localhost:8082
curl -v localhost:8082

# Make an HTTP request with wget
wget -O - localhost:8082
wget -d -O - localhost:8082
from urllib.request import urlopen
resp = urlopen("http://localhost:8082")
assert resp.code == 200
assert resp.headers.get_content_type() == 'text/plain'
body = resp.read()
print(body)
.. index:: ncat
ncat
nc --help
ncat --help
.. index:: Nginx

Nginx

Nginx is a scriptable, lightweight HTTP server written in :ref:`C`.

.. index:: Python http.server
Python http.server
python -m http.server --help
python -m http.server --directory . 8082
python -m http.server --directory . --cgi 8082

See also: :ref:`pgs`

.. index:: Tengine

Tengine

Tengine is a fork of :ref:`Nginx` with many useful modules and features bundled in.

.. index:: Traefik

Traefik

.. index:: Kubernetes Ingress

Kubernetes Ingress

.. index:: Documentation

Documentation Tools

.. index:: Docutils

Docutils

Docutils is a :ref:`Python` library which 'parses" :ref:`ReStructuredText` lightweight markup language into a doctree (~DOM) which can be serialized into HTML, ePub, MOBI, LaTeX, man pages, Open Document files, XML, JSON, and a number of other formats.

.. index:: Pandoc

Pandoc

Pandoc is a "universal" markup converter written in :ref:`Haskell` which can convert between HTML, :ref:`BBCode`, :ref:`Markdown`, :ref:`MediaWiki Markup <mediawiki markup>`, :ref:`ReStructuredText`, HTML, and a number of other formats.

.. index:: pgs

Pgs

pgs is an open source web application written in :ref:`Python` for serving static files from a :ref:`git` branch, or from the local filesystem.

pgs -p "${_WRD}/_build/html" -r gh-pages -H localhost -P 8082

  • pgs is written with the one-file Bottle web framework

  • compared to python -m SimpleHTTPServer localhost:8000 / python3 -m http.server localhost:8000 pgs has WSGI, the ability to read from a Git branch without real :ref:`git` bindings, and caching HTTP headers based on Git or filesystem mtimes.

  • pgs does something like :ref:`Nginx` try_files $.html

.. index:: Sphinx

Sphinx

Sphinx is a tool for working with :ref:`ReStructuredText` documentation trees and rendering them into HTML, PDF, LaTeX, ePub, and a number of other formats.

Sphinx extends :ref:`Docutils` with a number of useful markup behaviors which are not supported by other ReStructuredText parsers.

Most other ReStructuredText parsers do not support Sphinx directives; so, for example,

.. glossary::

   Sphinx Builder
      A Sphinx Builder transforms :ref:`ReStructuredText` into various
      output forms:

         * HTML
         * LaTeX
         * PDF
         * ePub
         * MOBI
         * JSON
         * OpenDocument (OpenOffice)
         * Office Open XML (MS Word)

      See: `Sphinx Builders <https://www.sphinx-doc.org/builders.html>`_

   Sphinx ReStructuredText
      Sphinx extends :ref:`ReStructuredText` with roles and directives
      which only work with Sphinx.

   Sphinx Directive
      Sphinx extensions of :ref:`Docutils` :ref:`ReStructuredText` directives.

      Most other ReStructuredText parsers do not support Sphinx directives.

      .. code-block:: rest

         .. toctree::

            readme
            installation
            usage

      See: `Sphinx Directives <https://sphinx-doc.org/rest.html#directives>`_

   Sphinx Role
      Sphinx extensions of :ref:`Docutils` :ref:`RestructuredText` roles

      Most other ReStructured

      .. code-block:: rest

            .. _anchor-name:

            A link to :ref:`anchor <anchor-name>`.



.. index:: jupyter-book
jupyter-book
.. index:: nbsphinx
nbsphinx
.. index:: Tinkerer
Tinkerer

Tinkerer is a very simple static blogging website generation tool written in :ref:`Python` which extends :ref:`Sphinx` and generates :ref:`HTML-` from :ref:`ReStructuredText`.

Static HTML pages generated with Tinkerer do not require a serverside application, and can be easily hosted with GitHub Pages or any other web hosting service.

.. index:: Backup Tools

Backup Tools

.. index:: Backup Ninja

Backup Ninja

Backup Ninja is an open source backup utility written in /etc/backup.d

  • BackupNinja supports :ref:`rdiff-backup`, :ref:`Duplicity`, and :ref:`rsync`.
  • BackupNinja can create and burn CD/DVD images.
  • BackupNinja can backup a number of relational databases (MySQL, PostgreSQL), maildirs, SVN repositories, Trac instances, and LDAP.
.. index:: Bup

bup

Bup (backup) is a backup system based on :ref:`git` packfiles and rolling checksums.

[Bup is a very] efficient backup system based on the git packfile format, providing fast incremental saves and global deduplication (among and within files, including virtual machine images).
  • AFAIU, like :ref:`git`, Bup does not preserve file permissions, Access Control Lists, or extended attributes (though some archive formats and snapshot images do).
.. index:: Clonezilla

Clonezilla

Clonezilla is an open source :ref:`Linux` distribution which is bootable from a CD/DVD/USB (a LiveCD, LiveDVD, LiveUSB) or PXE which contains a number of tools for disk imaging, disk cloning, filesystem backup and recovery; and a server :ref:`Linux` distribution for serving disk images to one or more computers over a LAN.

.. index:: Duplicity

Duplicity

Duplicity is an open source incremental file directory backup utility with GnuPG encryption, signatures, versions, and a number of actions for redundantly storing backups.

  • Duplicity can push offsite backups to/over a number of protocols and services (e.g. SSH/SCP/SFTP, S3, Google Cloud Storage, Rackspace Cloudfiles (OpenStack Swift)).
  • Duplicity stores data with tar archives and :ref:`rdiff`
  • :ref:`rdiff-backup` is similar to :ref:`duplicity`.
.. index:: FSArchiver

FSArchiver

FSAchiver is an open source filesystem backup (disk cloning) utility which can preserve file permissions, labels, and extended attributes.

  • FSArchiver can backup a filesysmet to a new or within an existing filesystem.
  • FSArchiver has special support for LVM.
  • FSArchiver supports password-based encryption.
.. index:: partclone

partclone

partclone is an open source utility for making compressed backups of the used blocks of partitions with each specific filesystem driver.

.. index:: partimage

partimage

Partimage is an open source utility for making complete sector-for-sector compressed backups of partitions over the network or to a local device.

.. index:: rclone

rclone

Rclone is an open source utility for managing files on cloud storages like local disk, SFTP, WebDAV, Dropbox, and Google Drive.

  • Rclone supports very many cloud storages
.. index:: rsync

rsync

rsync is an open-source file backup utility which can be used to make incremental backups using file deltas over the network or the local system.

  • rsync may appear to be stalled when it is actually calculating the full set of initial relative differences in order to minimize the amount of data transfer.

Note

rsync does not preserve file permissions by default.

To preserve file permissions with rsync:

man rsync

rsync -a    # rsync -rlptgoD
  rsync -r  # recursive (traverse into directories)
  rsync -l  # copy symlinks as links
  rsync -p  # preserve file permissions
  rsync -t  # preserve modification times
  rsync -g  # preserve group
  rsync -o  # preserve owner (requires superuser)
  rsync -D  # rsync --devices --specials
    rsync --devices   # preserve device files (requires superuser)
    rsync --specials  # preserve special files
rsync -A  # preserve file ACLs
rsync -X  # preserve file extended attributes

rsync -aAX  # rsync -a -A -X

rsync -v  # verbose
rsync -P  # rsync --partial --progress
  rsync --partial     # keep partially downloaded files
  rsync --progress    # show *per-file* progress and xfer speed

Note

rsync is picky about paths and trailing slashes.

# setUp
mkdir -p A/one B/one  # TODO
echo 'A' > A/one; echo 'B' > B/one
# tests
rsync A B
rsync A B/  --> B/A
rsync A/ B
rsync A/ B/
.. index:: rdiff
rdiff

rdiff is the open source relative delta algorithm of :ref:`rsync`.

.. index:: rsnapshot
rsnapshot

rsnapshot is an open source incremental file directory backup utility built with :ref:`rsync`.

.. index:: rdiff-backup

rdiff-backup

rdiff-backup is an open source incremental file directory backup utility.

  • Like :ref:`rsync`, rdiff-backup transmits file deltas instead of entire files.
  • Unlike :ref:`rsync`, rdiff-backup manages reverting to previous revisions.
.. index:: SystemRescueCD

SystemRescueCD

SystemRescueCD is a :ref:`Linux` distribution which is bootable from a CD/DVD/USB (a LiveCD) which contains a number of helpful utilities for system maintenance.

.. index:: Standards

Standards

.. index:: Cascading Style Sheets

.. index:: CSS

CSS

CSS (Cascading Style Sheets) define the presentational aspects of :ref:`HTML-` and a number of mobile and desktop web framworks.

  • CSS is designed to ensure separation of data and presentation. With javascript, the separation is then data, code, and presentation.
.. index:: Filesystem Hierarchy Standard

Filesystem Hierarchy Standard

The Filesystem Hierarchy Standard (FHS) is a well-worn industry-supported system file naming structure.

.. index:: HTTP

HTTP

.. index:: HTTPS
HTTPS
.. index:: HTML

HTML

.. index:: JSON

JSON

JSON is an object representation in :ref:`Javascript` syntax which is now supported by libraries written in many languages.

A list of objects with key and value attributes in JSON syntax:

[
{ "key": "language", "value": "Javascript" },
{ "key": "version", "value": 1 },
{ "key": "example", "value": true }
]

Machine-generated JSON is often not very readable, because it doesn't contain extra spaces or newlines. The :ref:`Python` JSON library contains a utility for parsing and indenting ("prettifying") JSON from the commandline

cat example.json | python -m json.tool

.. index:: JSON5

JSON5

JSON5 is JSON extended with support for a number of additional features: comments, trailing commas, IEEE 754 +/- infinity and NaN, hexadecimal numbers, leading and trailing decimal points, single-quoted strings, multiline strings, and escaped characters.

  • Regular JSON libraries do not support JSON5.
{
// comment
key:   [0, +1, 2., .3, NaN, +inf, -inf, 0xF, 'thing1', "thing2"],
"str": "this is a \
multi-line string", // trailing comma
}
.. index:: JSON-lines

JSON-lines

JSON-lines (newline-delimited JSON) is an informal spec for line-based processing of JSON e.g. for streaming records and unix pipes.

{"key": "red", "value": 1}
{"key": "green", "value": 2}
.. index:: JSONLD

.. index:: JSON-LD

JSON-LD

JSON-LD is a web standard for Linked Data in :ref:`JSON <json->`.

An example from the JSON-LD Playground (https://goo.gl/xxZ410):

{
   "@context": {
    "gr": "http://purl.org/goodrelations/v1#",
    "pto": "http://www.productontology.org/id/",
    "foaf": "http://xmlns.com/foaf/0.1/",
    "xsd": "http://www.w3.org/2001/XMLSchema#",
    "foaf:page": {
      "@type": "@id"
    },
    "gr:acceptedPaymentMethods": {
      "@type": "@id"
    },
    "gr:hasBusinessFunction": {
      "@type": "@id"
    },
    "gr:hasCurrencyValue": {
      "@type": "xsd:float"
    }
   },
   "@id": "http://example.org/cars/for-sale#tesla",
   "@type": "gr:Offering",
   "gr:name": "Used Tesla Roadster",
   "gr:description": "Need to sell fast and furiously",
   "gr:hasBusinessFunction": "gr:Sell",
   "gr:acceptedPaymentMethods": "gr:Cash",
   "gr:hasPriceSpecification": {
    "gr:hasCurrencyValue": "85000",
    "gr:hasCurrency": "USD"
   },
   "gr:includes": {
    "@type": [
      "gr:Individual",
      "pto:Vehicle"
    ],
    "gr:name": "Tesla Roadster",
    "foaf:page": "http://www.teslamotors.com/roadster"
   }
}
.. index:: MessagePack

MessagePack

MessagePack (msgpack) is a data interchange format with implementations in many languages.

.. index:: Text Editors

Text Editors

.. index:: Gedit

Gedit

Gedit is an open source text editor written in :ref:`C` and :ref:`Python` (:ref:`GTK`, GtkSourceView, and :ref:`Gnome`) that's available for :ref:`Linux`, :ref:`OSX`, and :ref:`Windows`).

  • Gedit supports tabbed editing.
  • Gedit plugins are written in :ref:`Python`.
  • Gedit is the default :ref:`Gnome` text editor; where it's called "Text Editor".
.. index:: Notepad++

Notepad++

Notepad++ is an open source text editor written in :ref:`C++` for :ref:`Windows` which has tabbed editing.

.. index:: IDEs

.. index:: Integrated Development Environments

IDEs

An IDE (Integrated Development Environment) is a software tool for developing software.

  • Most IDEs are source code :ref:`Text Editors`.
  • Some IDEs are visual development tools for various types of not code trees and graphs.
  • IDEs have a concept of a project, which may be defined in a config file in the current working directory or otherwise selected through the GUI.
  • An IDE has some sort of language server that understands the source code at a deeper level than syntax in order to do cool things like code completion and code refactorings like renaming a method in every file in the project.
  • https://en.wikipedia.org/wiki/Comparison_of_integrated_development_environments
.. index:: Emacs

.. index:: GNU Emacs

Emacs

GNU Emacs is an open source text editor written in Emacs :ref:`Lisp` and :ref:`C` that's available for :ref:`Linux`, :ref:`OSX`, and :ref:`Windows`.

.. index:: Spacemacs
Spacemacs

Print help:

SPC h SPC

.. index:: org-mode
org-mode

Org-mode is an open source document editing mode originally written in :ref:`Emacs Lisp` for :ref:`Emacs` that's now available in some form for a number of editors including :ref:`Vim`.

  • Org-mode makes it really easy to work with outlines in plain text documents.
  • The org-mode wikipedia page lists a number of org-mode implementations for other editors.
.. index:: org-babel
org-babel
.. index:: VSCode

VSCode

VSCode (Visual Studio Code) is an open source programmer's text editor written in :ref:`TypeScript`, :ref:`Javascript`, and :ref:`CSS` that's available for :ref:`Windows`, :ref:`Mac`, and :ref:`Linux`.

  • VSCode extensions are written in :ref:`Javascript`.
  • VSCode has collaborative editing features with multiple cursors.
  • VSCode and MS Visual Studio are different projects.
  • VSCode supports many of the Visual Studio keyboard shortcuts.
  • There is an official :ref:`Vim` extension for :ref:`VSCode`.
  • In VSCode, Ctrl+Space opens the context-sensitive Intellisense Code Completion
  • In VScode, Ctrl-p opens the quick open dialogue
  • IN VScode, Ctrl-Shift-p opens the command palette (which lists "all available commands based on your current context")

You can install VSCode by downloading from the Download page or with :ref:`Chocolatey`:

choco install vscode

.. index:: VScode Flatpak

VSCode Flatpak

.. index:: Vim

Vim

ViM (VI-iMproved) is an open source text editor written in :ref:`C` that's available on very many platforms.

  • Vim help can be accessed with :help and :help help (Press <esc>, Type :help help, Press Enter)
  • Vi is almost always installed on Linux and BSD boxes.
  • Vi is often included with :ref:`Busybox`.
  • Vi and Vim are installed with :ref:`OSX`.
  • Vi and Vim are installed by default with many :ref:`Linux Distributions`
  • Vim runs in a terminal, over SSH, and with a GUI window manager (Gvim, Macvim)
  • Vim configuration is written in the vim language.
  • Vim reads a few vimrc configuration files in sequence (:help vimrc)
  • GVim is Vim for :ref:`Gnome` window manager
  • GVim reads a few vimrc configuration files in sequence (:help gvimrc)
  • MacVim is Vim for :ref:`OSX`
  • One way to write changes and exit vim: :wq! (Press <esc>, Type :wq!, Press Enter)
  • There are many plugins for vim.
  • NERDTree is an example of a vim plugin: https://github.com/scrooloose/nerdtree (:help nerdtree)
  • :ref:`SpaceVim` and :ref:`westurner/dotvim` include the NERDtree plugin
  • Vim keyboard shortcuts are calling mappings.
  • Vim mappings are defined in a vimrc file.
  • Examples of vim mappings: \e opens NERDTree, \E opens NERDTree to the current file
  • Vim mappings can be defined for different vim modes: :map \e (command mode), :imap \e (insert mode) (:help modes)
  • Press i or a while in command mode to enter insert or append mode (:help vim-modes)
  • Press <Esc> to return to command mode

:ref:`Browser extensions` with vim-style keyboard shortcuts:

A number of web apps support vim-style keyboard shortcuts like j and k for up and down:

  • GMail (? for help)
  • Facebook (? for help)
  • Twitter (? for help)
.. index:: neovim
neovim

Neovim is an open source programmer's editor and :ref:`IDE` which is a rewrite of :ref:`Vim`.

.. index:: westurner/dotvim
westurner/dotvim

westurner/dotvim is a set of plugins and configuration defaults for :ref:`Vim`.

.. index:: SpaceVim
SpaceVim

SpaceVim is a set of plugins, configuration defaults, and keybindings for :ref:`Vim`.

.. index:: WRD R&D Documentation

WRD R&D Documentation

^top^

#top

Clone this wiki locally