From 066653bb6a11d7e3c472590bdfacabee2d248b50 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Wed, 18 Nov 2015 14:42:59 -0700 Subject: [PATCH 01/16] Create document outline For DM-4351. --- index.rst | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/index.rst b/index.rst index d7a966d..c716d21 100644 --- a/index.rst +++ b/index.rst @@ -37,4 +37,36 @@ Feel free to delete this instructional comment. -Write the docs; right here. +Technical notes (hereafter, *technotes*) are a new documentation medium for `LSST Data Management `_ staff to publish stand-alone documents that are native to the web. +Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable `digital object identifiers (DOIs) `_ through Zenodo_. + +This document describes the Technote platform and will be updated as the platform evolves. + +DM team members can create a new technote today by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap + +.. _Zenodo: https://zenodo.org + +.. _niche: + +Technotes' niche in the Data Management communication landscape +=============================================================== + + + +Proof of concept implementation +=============================== + +Roadmap for improvements +======================== + +Improved document creation and management automation +---------------------------------------------------- + +Improved presentation +--------------------- + +A document index +---------------- + +Metadata standard +================= From 6d25c0189078b32e3013a71f5ce32fdd93f7fb99 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Thu, 19 Nov 2015 17:02:42 -0700 Subject: [PATCH 02/16] Write section describing niche of technotes For DM-4351. --- index.rst | 75 ++++++++++++++++++++++++++----------------------------- 1 file changed, 36 insertions(+), 39 deletions(-) diff --git a/index.rst b/index.rst index c716d21..9d26167 100644 --- a/index.rst +++ b/index.rst @@ -1,42 +1,3 @@ -.. - Content of technical report. - - See http://docs.lsst.codes/en/latest/development/docs/rst_styleguide.html - for a guide to reStructuredText writing. - - Do not put the title, authors or other metadata in this document; - those are automatically added. - - Use the following syntax for sections: - - Sections - ======== - - and - - Subsections - ----------- - - and - - Subsubsections - ^^^^^^^^^^^^^^ - - To add images, add the image file (png, svg or jpeg preferred) to the - _static/ directory. The reST syntax for adding the image is - - .. figure:: /_static/filename.ext - :name: fig-label - :target: http://target.link/url - - Caption text. - - Run: ``make html`` and ``open _build/html/index.html`` to preview your work. - See the README at https://github.com/lsst-sqre/lsst-report-bootstrap or - this repo's README for more info. - - Feel free to delete this instructional comment. - Technical notes (hereafter, *technotes*) are a new documentation medium for `LSST Data Management `_ staff to publish stand-alone documents that are native to the web. Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable `digital object identifiers (DOIs) `_ through Zenodo_. @@ -51,6 +12,42 @@ DM team members can create a new technote today by following the instructions at Technotes' niche in the Data Management communication landscape =============================================================== +DM already has a myriad of communication channels. +For conversations we use HipChat in real-time, and the `Community forum`_ for asynchronous long-form discussions, notices and community engagement. +We track work on JIRA. +We document our code in software docs (the platform for which is being re-imagined by SQuaRE). +We describe the design and architecture of the DM subsystem in change-controlled design documents. +We publish articles in journals and conference proceedings where appropriate. +Finally, we have several Confluence wikis that traditionally served as a catch-all for DM documentation. +Despite this array of platforms, a use case was still unserved this technote platform was born organically to meet that need. + +In SQuaRE, the Science Quality and Reliability Engineering, group, we realized that we often wanted to write stand-alone documents to describe our work. +These documents didn't fit the conversational nature of `Community forum`_ topics, nor did they qualify for DM's change-controlled design document series, or fit into academic journals or the `arXiv`. +Traditionally, Confluence was meant to fill this niche, and indeed, the wikis are well used by DM. +In practice, however, wikis have served DM poorly. +Confluence pages are hard to find and discover. +The editing and version control experience is also clumsy from the perspective of DM developers used to working on GitHub. + +In the fall of 2015, we experimented with moving several DM design documents from Word documents and stored on Docushare to documents that are written in reStructuredText, built with Sphinx, hosted on GitHub, and deployed to the web with Read the Docs. +The impetus behind the transition was that documents written in plain text and hosted on GitHub were far more likely to be kept up-to-date by DM developers. + +Technotes were thus built on the same technology stack as the reStructuredText-based design documents, but re-purposed for general use by DM team members.\ [#]_ + +.. [#] In fact, this technote applies equally to describing the new reStructuredText-based design document platform as it does the technote platform. The only meaningful difference is that design documents must be approved by a control board, and are ultimately deposited in LSST's Docushare. + +Some of the possible applications for technotes are: + +- to report the results of a project, such as a data processing or software development experiment, +- to announce a new technology, serving as a high-level wrapper to software documentation, +- to propose an architecture, possibly becoming the subject of a request for comment (RFC). + +Although there is a clear need for technotes, there may inevitably be uncertainty in deciding which platform should be used for a given piece of content. +In particular, technotes are not a replacement for user-oriented documentation of software. +A technote can be written with how-to documentation in support of software experiments, once a software product is fully developed we should strive to provide proper user documentation. +That said, if the software product is backend infrastructure, a technote describing the product's architecture to DM colleagues along with 'README' files in the code may be entirely sufficient. + +.. _Community forum: https://community.lsst.org +.. _arXiv: http://arxiv.org Proof of concept implementation From 46f7de6cb3536806b607b13a7388b8f80e6fa427 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Thu, 19 Nov 2015 18:31:46 -0700 Subject: [PATCH 03/16] Describe design choices for technotes For DM-4351. --- index.rst | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/index.rst b/index.rst index 9d26167..1005c20 100644 --- a/index.rst +++ b/index.rst @@ -1,3 +1,5 @@ +:tocdepth: 1 + Technical notes (hereafter, *technotes*) are a new documentation medium for `LSST Data Management `_ staff to publish stand-alone documents that are native to the web. Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable `digital object identifiers (DOIs) `_ through Zenodo_. @@ -49,6 +51,50 @@ That said, if the software product is backend infrastructure, a technote describ .. _Community forum: https://community.lsst.org .. _arXiv: http://arxiv.org +Technote Platform Design Choices +================================= + +Although technotes are still in an early, minimally-viable state, (see :ref:`implementation`), they also demonstrate several design decisions that characterize the platform. + +Technotes are native to the web +------------------------------- + +Although a preponderance of astronomy literature is published as PDFs, and many management documents as Word documents, those formats are archaic solutions from an era when documents were read in print. +Modern HTML, CSS and JavaScript allow for better, more innovative, and frictionless reading experiences than PDFs and Word documents can provide. +Modern readers expect to be able to search for content, click and link, and read it immediately in the browser. + +Technotes are written in reStructuredText +----------------------------------------- + +A fundamental requirement of the technote platform is that documents are written in a plain text format, and are then compiled into a website so that content can be written in common text editors and hosted on GitHub for collaboration. +Many plain text formats have been invented for writing, however, not all formats are ideal for technotes. +For example, LaTeX, though common in astronomy, is tied too closely to PDF output and makes little sense for a web-native syntax. +Markdown is a hugely population format among developers, in part because of its intuitive simplicity. +However, this simplicity has prompted developers to add non-standard extensions to the language to their markup to add semantics and style to content. + +Instead, reStructuredText is an ideal markup format for technotes. +reStructuredText's syntax and build process was designed to be extended, and in fact, those extensions are made in DM's primary language, Python. +Because the Python community has adopted reStructuredText as its official markup language, a our technote platform can take advantage of excellent open source tools, such as the Sphinx build system and the Read the Docs documentation hosting site. +For these same reasons, reStructuredText and the Sphinx build toolchain are also used by DM's new software documentation platform. +This allows both documentation platforms to benefit from the same bespoke infrastructure development, such as SQuaRE's `documenteer`_ package. + +.. _documenteer: https://github.com/lsst-sqre/documenteer + +Technotes are single-page documents +----------------------------------- + +This limitation is made purposefully to limit the scope of technotes, make it possible to easily print a technote in its entirely, and allow readers to their browser's search feature to navigate a document. +We are designing technote presentation to accommodate on-page navigation within a long document. + +Technotes are citeable +---------------------- + +Since they are standalone, self-encapsulated documents, technotes can be easily archived and assigned digital object identifiers, DOIs. +DOIs allow technotes to be easily cited in scientific literature. +The benefit of a citeable-trail of DM implementation work is that LSST publications can more accurately describe DM's engineering work. +We use Zenodo_ as an archive and DOI provider, taking advantage of its GitHub integration. + +.. _implementation: Proof of concept implementation =============================== From 96a3a3114dc671819d6ff1d9813a390c3ffddc15 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Thu, 19 Nov 2015 21:43:50 -0700 Subject: [PATCH 04/16] Ignore _build/ directory --- .gitignore | 1 + 1 file changed, 1 insertion(+) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e35d885 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +_build From bb5c45c3c1663fabc0fec978b0f8273292822598 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Thu, 19 Nov 2015 23:08:59 -0700 Subject: [PATCH 05/16] Describe lsst-technote-bootstrap impl For DM-4351. --- index.rst | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/index.rst b/index.rst index 1005c20..6603024 100644 --- a/index.rst +++ b/index.rst @@ -7,6 +7,9 @@ This document describes the Technote platform and will be updated as the platfor DM team members can create a new technote today by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap +We currently use two technote series. The 'DMTN' is for general use within Data Management. +The 'SQR' technote series is used for SQuaRE science quality verification and infrastructure work. + .. _Zenodo: https://zenodo.org .. _niche: @@ -94,11 +97,74 @@ DOIs allow technotes to be easily cited in scientific literature. The benefit of a citeable-trail of DM implementation work is that LSST publications can more accurately describe DM's engineering work. We use Zenodo_ as an archive and DOI provider, taking advantage of its GitHub integration. +Technotes are versioned +----------------------- + +Unlike scientific publications that are published once, technotes can and should be updated when appropriate. +The full version history is maintained by git and published on GitHub. + .. _implementation: Proof of concept implementation =============================== +We released a mininum-viable product for creating publishing tech notes. +Authors can create a technote by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap. + +.. _lsst-technote-bootstrap: https://github.com/lsst-sqre/lsst-technote-bootstrap +.. _cookiecutter: http://cookiecutter.rtfd.org/ +.. _Jinja2: http://jinja.pocoo.org + +Project Automation +------------------ + +`lsst-technote-bootstrap`_ is built around the cookiecutter_ Python project. +cookiecutter_ allows code *projects* to be templated in the Jinja2_ template language. +Everything about the project can be templated: file contents, file names, and even directory structures. +By running + +.. code-block:: bash + + cookiecutter https://github.com/lsst-sqre/lsst-technote-bootstrap.git + +the author is prompted to answer questions that configure the document. +When that is done, the author is left with a working Sphinx-based documentation project that can be immediately built with a ``make html`` command. +This level of configuration automation is crucial to the adoption of tech notes, and :ref:`we intend to only increase this level of automation `. + +Document Build Configuration and Metadata +----------------------------------------- + +The Sphinx project prepared by `lsst-technote-bootstrap`_ appears conventional with the exception of how the Sphinx build is configured. +Most Sphinx projects have extensive ``conf.py`` files, which are ``execfile()``'d Python code that configure Sphinx and prepare the data available to document templates. +The Sphinx ``conf.py`` posed a maintenance threat to technotes: any infrastructural change to the Sphinx build system for technotes would require edits to the ``conf.py`` files of every technote and DM design document. +Our solution was to strip nearly all logic from the ``conf.py`` files, and centralize all configuration management in our documenteer_ Python package. +Now, single commits to documenteer_ are effectively deployed instantly to all technotes. + +Of course, individual technotes need custom configuration, such as title and authorship information. +We keep this in a ``metadata.yaml`` file in each technote repository. +By effectively refactoring metadata out of both ``conf.py`` *and* the reStructuredText content, it is easy to develop a standardized schema for describing technotes. +See :ref:`metadata`. +Such a schema opens opportunities for indexing DM's technote library. + +Deployment +---------- + +GitHub is the central infrastructure for hosting technotes. +The ``master`` branch is considered a live publication, but 'releases' can be made as well using git tags or the GitHub release feature. + +Technotes are published on `Read the Docs`_ a free and open-source platform for publishing Sphinx-based documentation, such as technotes. +`Read the Docs`_ integrates with GitHub to rebuild the technote's webpage whenever commits are pushed to the technote's ``master`` branch on GitHub. +We serve technotes as a subdomain of ``lsst.io``, e.g., http://sqr-000.lsst.io. + +.. _`Read the docs`: http://readthedocs.org + +Finally, major versions of the technote can be granted DOIs. +The technote repository can be connected to Zenodo_. +When a major version of a technote is completed, a GitHub Release can be made, and the contents of the technote repository are uploaded and archived on Zenodo_. +`Following our instructions `_, a citeable DOI can be conveniently obtained. + +.. _roadmap: + Roadmap for improvements ======================== @@ -111,5 +177,7 @@ Improved presentation A document index ---------------- +.. _metadata: + Metadata standard ================= From 4d25ed57cf0898a56fcb8c02b6851b04e54b501f Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Thu, 19 Nov 2015 23:51:44 -0700 Subject: [PATCH 06/16] Add a roadmap for technote platform. For DM-4351. --- index.rst | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/index.rst b/index.rst index 6603024..40edabb 100644 --- a/index.rst +++ b/index.rst @@ -171,12 +171,44 @@ Roadmap for improvements Improved document creation and management automation ---------------------------------------------------- +Although lsst-technote-bootstrap_ automates report creation, there are still many facets of technote authorship that would benefit from automation: + +#. additional automation of technote configuration, beyond what cookiecutter_ provides (such as dynamic date suggestions) +#. creation of a GitHub repository +#. creation and configuration of a Read The Docs project +#. provisioning of an ``lsst.io`` domain +#. syntax linting (Travis CI testing) +#. automation of releases and procurement of DOIs (leveraging ``metadata.yaml`` to automate the technote's deposition on Zenodo_) + +This likely demands a command line application to manage technotes, which incorporate lsst-technote-bootstrap_. +Likely the most challenging aspect will be automating the creation of a Read the Docs project, since project creation is not part of `RTD's API `_. + Improved presentation --------------------- +Technotes are currently published Read the Doc's default theme, with minor additions to incorporate metadata from ``metadata.yaml``. +A new HTML/CSS theme is needed to + +- establish a visual identity for DM documents +- provide allowances for navigation in long single page documents +- add facilities for styling elements created by an extended reStructuredText language (rather than retrofitting an existing theme) + +Extensions to reStructuredText +------------------------------ + +DM authors need a richer reStructuredText language to for technical writing. +One request is to implement a bibliographies of the same quality as are possible with LaTeX and natbib. +We can achieve this by developing Sphinx extensions within the documenteer_ package. +Development work done here will also benefit DM's software documentation. + A document index ---------------- +From Docushare and the Confluence wikis, we learned that documentation can be easily buried if not indexed from a central, authoritative, reliable and highly visible place. +We need to provide a documentation index for DM, likely as part of http://dm.lsst.org. +The page could be automatically updated by leveraging the GitHub API and individual documents' ``metadata.yaml`` information. +Ideally, the index would provide facilities for filtering or searching. + .. _metadata: Metadata standard From 6a8dc02f2cf169c0eb4bf6f20e196e33f8d6f831 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 20 Nov 2015 14:51:32 -0700 Subject: [PATCH 07/16] Describe metadata and propose extensions For DM-4351. --- index.rst | 195 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 180 insertions(+), 15 deletions(-) diff --git a/index.rst b/index.rst index 40edabb..2ebb561 100644 --- a/index.rst +++ b/index.rst @@ -10,8 +10,6 @@ DM team members can create a new technote today by following the instructions at We currently use two technote series. The 'DMTN' is for general use within Data Management. The 'SQR' technote series is used for SQuaRE science quality verification and infrastructure work. -.. _Zenodo: https://zenodo.org - .. _niche: Technotes' niche in the Data Management communication landscape @@ -51,9 +49,6 @@ In particular, technotes are not a replacement for user-oriented documentation o A technote can be written with how-to documentation in support of software experiments, once a software product is fully developed we should strive to provide proper user documentation. That said, if the software product is backend infrastructure, a technote describing the product's architecture to DM colleagues along with 'README' files in the code may be entirely sufficient. -.. _Community forum: https://community.lsst.org -.. _arXiv: http://arxiv.org - Technote Platform Design Choices ================================= @@ -81,8 +76,6 @@ Because the Python community has adopted reStructuredText as its official markup For these same reasons, reStructuredText and the Sphinx build toolchain are also used by DM's new software documentation platform. This allows both documentation platforms to benefit from the same bespoke infrastructure development, such as SQuaRE's `documenteer`_ package. -.. _documenteer: https://github.com/lsst-sqre/documenteer - Technotes are single-page documents ----------------------------------- @@ -111,10 +104,6 @@ Proof of concept implementation We released a mininum-viable product for creating publishing tech notes. Authors can create a technote by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap. -.. _lsst-technote-bootstrap: https://github.com/lsst-sqre/lsst-technote-bootstrap -.. _cookiecutter: http://cookiecutter.rtfd.org/ -.. _Jinja2: http://jinja.pocoo.org - Project Automation ------------------ @@ -174,10 +163,11 @@ Improved document creation and management automation Although lsst-technote-bootstrap_ automates report creation, there are still many facets of technote authorship that would benefit from automation: #. additional automation of technote configuration, beyond what cookiecutter_ provides (such as dynamic date suggestions) -#. creation of a GitHub repository -#. creation and configuration of a Read The Docs project +#. creation of a `GitHub`_ repository +#. creation and configuration of a `Read The Docs`_ project #. provisioning of an ``lsst.io`` domain -#. syntax linting (Travis CI testing) +#. reStructuredText and metadata linting (using `Travis CI `_ testing) +#. automatic local builds and browser updates (e.g., `Browsersync `_) #. automation of releases and procurement of DOIs (leveraging ``metadata.yaml`` to automate the technote's deposition on Zenodo_) This likely demands a command line application to manage technotes, which incorporate lsst-technote-bootstrap_. @@ -204,7 +194,7 @@ Development work done here will also benefit DM's software documentation. A document index ---------------- -From Docushare and the Confluence wikis, we learned that documentation can be easily buried if not indexed from a central, authoritative, reliable and highly visible place. +From experience with Docushare and the Confluence wikis, we learnt that documentation can be easily buried if not indexed from a central, authoritative, reliable and highly visible place. We need to provide a documentation index for DM, likely as part of http://dm.lsst.org. The page could be automatically updated by leveraging the GitHub API and individual documents' ``metadata.yaml`` information. Ideally, the index would provide facilities for filtering or searching. @@ -213,3 +203,178 @@ Ideally, the index would provide facilities for filtering or searching. Metadata standard ================= + +Here we document the available keys in the ``metadata.yaml`` schema. + +series: + A string identifying the technote series. + Possible values are ``'DMTN'`` for DM Technotes and ``'SQR'`` for SQuaRE Technotes. + Existing change-controlled document series can also be used, such as ``'LDM'``. + + Example: + + .. code-block:: yaml + + series: 'SQR' + +series_number: + Serial number of the document, as a string. + For the DMTN and SQR series we use three digit serial numbers (with leading zeros). + + Example: + + .. code-block:: yaml + + serial_number: '000' + +doc_id: + **Planned for deprecation.** This is a string that joins ``series`` and ``serial_number`` with a dash. + + Example: + + .. code-block:: yaml + + doc_id: 'SQR-000' + +authors: + Author names, ordered as a list. + Each author name should be formatted as 'First Last' + + Example: + + .. code-block:: yaml + + authors: + - 'Jonathan Sick' + - 'Frossie Economou' + + An extended syntax for the ``authors`` key is planned. + +version: + Version. Use Semvar, e.g., 1.0, including .dev, as necessary + This version string should correspond to the git tag when the document is published on Zenodo + + Example of a '1.0' release: + + .. code-block:: yaml + + version: '1.0' + + Example of an early development version: + + .. code-block:: yaml + + version: '0.1.dev' + +doi: + Digital Object Identifier (DOI). + Keep this DOI updated as new releases are pushed to Zenodo_. + + Example: + + .. code-block:: yaml + + doi: '10.5281/zenodo.12345' + + This field can be left commented (or omitted) if a DOI is unavailable: + + .. code-block:: yaml + + # doi: '10.5281/zenodo.#####' + +last_revised: + Document release date, as ``'YYYY-MM-DD'``. + + Example: + + .. code-block:: yaml + + '2015-11-18' + +copyright: + Copyright statement. + + Example: + + .. code-block:: yaml + + copyright: '2015, AURA/LSST' + +Planned metadata extensions +--------------------------- + +We plan to add the following fields to the ``metadata.yaml`` schema. + +description: + A short 1-2 sentence description for document indices. + +abstract: + An abstract, if available. + + Example: + + .. code-block:: yaml + + abstract: > + Write your paragraph + here with multiple lines. + + You can have multiple paragraphs too. + +url: + The canonical URL where the document is published by `Read the Docs`_. + + Example: + + .. code-block:: yaml + + url: 'http://sqr-000.github.io' + +docushare_url: + If a canonical version of the document is archived in Docushare, the URL can be provided. + + Example: + + .. code-block:: yaml + + docushare_url: 'https://docushare.lsstcorp.org/docushare/{{ path }}' + +github_url: + The document's URL on GitHub. + + Example: + + .. code-block:: yaml + + github_url: 'https://github.com/lsst-sqre/sqr-000' + +Leveraging ORCID for Author Information +--------------------------------------- + +The current authorship metadata is limited; the ``authors`` key is an ordered list of author names. +A better way to annotate authorship metadata would be through ORCID_ iDs, which unique identify researchers. +ORCID_ uses those identifiers to connect people to their work. + +A possible revised syntax for declaring authorship metadata would be + +.. code-block:: yaml + + authors: + - + name: Jonathan Sick + orcid: 0000-0003-3001-676X + - + name: Second Author + orcid: ####-####-####-#### + +ORCID_ iD integration would be used to improve the Zenodo_ submission process. + +.. _Zenodo: https://zenodo.org +.. _GitHub: https://github.com +.. _Community forum: https://community.lsst.org +.. _arXiv: http://arxiv.org +.. _documenteer: https://github.com/lsst-sqre/documenteer +.. _lsst-technote-bootstrap: https://github.com/lsst-sqre/lsst-technote-bootstrap +.. _cookiecutter: http://cookiecutter.rtfd.org/ +.. _Jinja2: http://jinja.pocoo.org +.. _ORCID: http://orcid.org/ From 2f064112354b4a42436dc33082fb72beb3b02090 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 20 Nov 2015 15:34:58 -0700 Subject: [PATCH 08/16] Edits of first draft For DM-4351. --- index.rst | 58 +++++++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 30 deletions(-) diff --git a/index.rst b/index.rst index 2ebb561..0234f0a 100644 --- a/index.rst +++ b/index.rst @@ -1,7 +1,7 @@ :tocdepth: 1 Technical notes (hereafter, *technotes*) are a new documentation medium for `LSST Data Management `_ staff to publish stand-alone documents that are native to the web. -Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable `digital object identifiers (DOIs) `_ through Zenodo_. +Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable with `digital object identifiers (DOIs) `_ provided through Zenodo_. This document describes the Technote platform and will be updated as the platform evolves. @@ -12,7 +12,7 @@ The 'SQR' technote series is used for SQuaRE science quality verification and in .. _niche: -Technotes' niche in the Data Management communication landscape +Technotes' Niche in the Data Management Communication Landscape =============================================================== DM already has a myriad of communication channels. @@ -22,16 +22,16 @@ We document our code in software docs (the platform for which is being re-imagin We describe the design and architecture of the DM subsystem in change-controlled design documents. We publish articles in journals and conference proceedings where appropriate. Finally, we have several Confluence wikis that traditionally served as a catch-all for DM documentation. -Despite this array of platforms, a use case was still unserved this technote platform was born organically to meet that need. +Despite this array of platforms, a use case was still unserved and this technote platform was conceived to meet that need. -In SQuaRE, the Science Quality and Reliability Engineering, group, we realized that we often wanted to write stand-alone documents to describe our work. -These documents didn't fit the conversational nature of `Community forum`_ topics, nor did they qualify for DM's change-controlled design document series, or fit into academic journals or the `arXiv`. +In SQuaRE, the Science Quality and Reliability Engineering group, we realized that we often wanted to write stand-alone documents to describe our work. +These documents didn't fit the conversational nature of `Community forum`_ topics, nor did they qualify for DM's change-controlled design document series, or fit into academic journals or the arXiv_. Traditionally, Confluence was meant to fill this niche, and indeed, the wikis are well used by DM. In practice, however, wikis have served DM poorly. Confluence pages are hard to find and discover. The editing and version control experience is also clumsy from the perspective of DM developers used to working on GitHub. -In the fall of 2015, we experimented with moving several DM design documents from Word documents and stored on Docushare to documents that are written in reStructuredText, built with Sphinx, hosted on GitHub, and deployed to the web with Read the Docs. +In the fall of 2015, we experimented with moving several DM design documents from Word documents stored on Docushare to documents that are written in `reStructuredText`_, built with `Sphinx`_, hosted on GitHub, and deployed to the web with `Read the Docs`_. The impetus behind the transition was that documents written in plain text and hosted on GitHub were far more likely to be kept up-to-date by DM developers. Technotes were thus built on the same technology stack as the reStructuredText-based design documents, but re-purposed for general use by DM team members.\ [#]_ @@ -41,14 +41,9 @@ Technotes were thus built on the same technology stack as the reStructuredText-b Some of the possible applications for technotes are: - to report the results of a project, such as a data processing or software development experiment, -- to announce a new technology, serving as a high-level wrapper to software documentation, +- to announce a new technology, serving as a high-level overview complementing software documentation, - to propose an architecture, possibly becoming the subject of a request for comment (RFC). -Although there is a clear need for technotes, there may inevitably be uncertainty in deciding which platform should be used for a given piece of content. -In particular, technotes are not a replacement for user-oriented documentation of software. -A technote can be written with how-to documentation in support of software experiments, once a software product is fully developed we should strive to provide proper user documentation. -That said, if the software product is backend infrastructure, a technote describing the product's architecture to DM colleagues along with 'README' files in the code may be entirely sufficient. - Technote Platform Design Choices ================================= @@ -59,16 +54,16 @@ Technotes are native to the web Although a preponderance of astronomy literature is published as PDFs, and many management documents as Word documents, those formats are archaic solutions from an era when documents were read in print. Modern HTML, CSS and JavaScript allow for better, more innovative, and frictionless reading experiences than PDFs and Word documents can provide. -Modern readers expect to be able to search for content, click and link, and read it immediately in the browser. +Modern readers expect to be able to search for content, click a link, and read it immediately in the browser. Technotes are written in reStructuredText ----------------------------------------- -A fundamental requirement of the technote platform is that documents are written in a plain text format, and are then compiled into a website so that content can be written in common text editors and hosted on GitHub for collaboration. +A fundamental requirement of the technote platform is that documents are written in a plain text format, and are then compiled into a website. This way, content is written in common text editors and hosted on GitHub for collaboration (the same workflow we already use for source code). Many plain text formats have been invented for writing, however, not all formats are ideal for technotes. For example, LaTeX, though common in astronomy, is tied too closely to PDF output and makes little sense for a web-native syntax. -Markdown is a hugely population format among developers, in part because of its intuitive simplicity. -However, this simplicity has prompted developers to add non-standard extensions to the language to their markup to add semantics and style to content. +`Markdown `_ is a hugely popular format among developers, in part because of its intuitive simplicity. +However, this simplicity has prompted developers to `add non-standard extensions to the format `_ to their markup to add semantics and style to content. Instead, reStructuredText is an ideal markup format for technotes. reStructuredText's syntax and build process was designed to be extended, and in fact, those extensions are made in DM's primary language, Python. @@ -79,7 +74,7 @@ This allows both documentation platforms to benefit from the same bespoke infras Technotes are single-page documents ----------------------------------- -This limitation is made purposefully to limit the scope of technotes, make it possible to easily print a technote in its entirely, and allow readers to their browser's search feature to navigate a document. +This limitation is made purposefully to limit the scope of technotes, make it possible to easily print a technote in its entirely, and allow readers to use their browser's search feature to navigate a document. We are designing technote presentation to accommodate on-page navigation within a long document. Technotes are citeable @@ -87,7 +82,7 @@ Technotes are citeable Since they are standalone, self-encapsulated documents, technotes can be easily archived and assigned digital object identifiers, DOIs. DOIs allow technotes to be easily cited in scientific literature. -The benefit of a citeable-trail of DM implementation work is that LSST publications can more accurately describe DM's engineering work. +The benefit of a citeable-trail of DM documentation is that LSST publications can more accurately describe DM's engineering work. We use Zenodo_ as an archive and DOI provider, taking advantage of its GitHub integration. Technotes are versioned @@ -98,13 +93,13 @@ The full version history is maintained by git and published on GitHub. .. _implementation: -Proof of concept implementation +Proof of Concept Implementation =============================== -We released a mininum-viable product for creating publishing tech notes. +We released a minimum-viable product for creating and publishing technotes. Authors can create a technote by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap. -Project Automation +Project automation ------------------ `lsst-technote-bootstrap`_ is built around the cookiecutter_ Python project. @@ -120,7 +115,7 @@ the author is prompted to answer questions that configure the document. When that is done, the author is left with a working Sphinx-based documentation project that can be immediately built with a ``make html`` command. This level of configuration automation is crucial to the adoption of tech notes, and :ref:`we intend to only increase this level of automation `. -Document Build Configuration and Metadata +Document build configuration and metadata ----------------------------------------- The Sphinx project prepared by `lsst-technote-bootstrap`_ appears conventional with the exception of how the Sphinx build is configured. @@ -170,13 +165,13 @@ Although lsst-technote-bootstrap_ automates report creation, there are still man #. automatic local builds and browser updates (e.g., `Browsersync `_) #. automation of releases and procurement of DOIs (leveraging ``metadata.yaml`` to automate the technote's deposition on Zenodo_) -This likely demands a command line application to manage technotes, which incorporate lsst-technote-bootstrap_. -Likely the most challenging aspect will be automating the creation of a Read the Docs project, since project creation is not part of `RTD's API `_. +This likely demands a command line application to manage technotes, which would incorporate lsst-technote-bootstrap_. +Likely the most challenging aspect will be automating the creation of a `Read the Docs`_ project, since project creation is not part of `RTD's API `_. Improved presentation --------------------- -Technotes are currently published Read the Doc's default theme, with minor additions to incorporate metadata from ``metadata.yaml``. +Technotes are currently published with Read the Doc's default theme (including minor additions to incorporate metadata from ``metadata.yaml``). A new HTML/CSS theme is needed to - establish a visual identity for DM documents @@ -186,8 +181,8 @@ A new HTML/CSS theme is needed to Extensions to reStructuredText ------------------------------ -DM authors need a richer reStructuredText language to for technical writing. -One request is to implement a bibliographies of the same quality as are possible with LaTeX and natbib. +DM authors need a richer reStructuredText language for technical writing. +One request is to implement citations and bibliographies of the same quality as are possible with LaTeX and natbib. We can achieve this by developing Sphinx extensions within the documenteer_ package. Development work done here will also benefit DM's software documentation. @@ -201,7 +196,7 @@ Ideally, the index would provide facilities for filtering or searching. .. _metadata: -Metadata standard +Metadata Standard ================= Here we document the available keys in the ``metadata.yaml`` schema. @@ -251,8 +246,8 @@ authors: An extended syntax for the ``authors`` key is planned. version: - Version. Use Semvar, e.g., 1.0, including .dev, as necessary - This version string should correspond to the git tag when the document is published on Zenodo + Use semantic versioning, e.g., '1.0', including '.dev', as necessary. + This version string should correspond to the git tag when the document is published on Zenodo_. Example of a '1.0' release: @@ -378,3 +373,6 @@ ORCID_ iD integration would be used to improve the Zenodo_ submission process. .. _cookiecutter: http://cookiecutter.rtfd.org/ .. _Jinja2: http://jinja.pocoo.org .. _ORCID: http://orcid.org/ +.. _reStructuredText: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html +.. _Sphinx: http://sphinx-doc.org +.. _Read the Docs: http://readthedocs.org From 85c4317e1b0478c6b74e9781d34feada9e3fd736 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 20 Nov 2015 15:42:59 -0700 Subject: [PATCH 09/16] Add travis checks for HTML builds For DM-4351. --- .travis.yml | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 0000000..9ac0f95 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,8 @@ +sudo: false +language: python +python: + - 3.4 +install: "pip install -r requirements.txt" +script: "make html" +notifications: + email: false From a517673d0452676d6f99d60b785399711a98b9b0 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Fri, 20 Nov 2015 15:58:10 -0700 Subject: [PATCH 10/16] Use wikipedia for DOI background link --- index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.rst b/index.rst index 0234f0a..fa2d97a 100644 --- a/index.rst +++ b/index.rst @@ -1,7 +1,7 @@ :tocdepth: 1 Technical notes (hereafter, *technotes*) are a new documentation medium for `LSST Data Management `_ staff to publish stand-alone documents that are native to the web. -Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable with `digital object identifiers (DOIs) `_ provided through Zenodo_. +Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable with `digital object identifiers (DOIs) `_ provided through Zenodo_. This document describes the Technote platform and will be updated as the platform evolves. From 0a97bdc914bf4eb7feb5b079b22527e1c3d7a94b Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 23 Nov 2015 17:41:44 -0700 Subject: [PATCH 11/16] Addressed most of timj's comments - better linking to software - use file and abbr roles when appropriate - be kinder to PDF allies - updated text on versioning For DM-4351. --- index.rst | 109 +++++++++++++++++++++++++++++------------------------- 1 file changed, 58 insertions(+), 51 deletions(-) diff --git a/index.rst b/index.rst index fa2d97a..55f3e52 100644 --- a/index.rst +++ b/index.rst @@ -1,14 +1,14 @@ :tocdepth: 1 -Technical notes (hereafter, *technotes*) are a new documentation medium for `LSST Data Management `_ staff to publish stand-alone documents that are native to the web. -Technotes are written in reStructuredText, version controlled on GitHub, built with the same stack as DM's software documentation, and made universally citeable with `digital object identifiers (DOIs) `_ provided through Zenodo_. +Technical notes (hereafter, *technotes*) are a new documentation medium for `LSST Data Management (DM) `_ staff to publish stand-alone documents that are native to the web. +Technotes are written in reStructuredText_, version controlled on GitHub_, built with the same stack as DM's software documentation, and made universally citeable with `digital object identifiers (DOIs) `_ provided through Zenodo_. This document describes the Technote platform and will be updated as the platform evolves. -DM team members can create a new technote today by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap +DM team members can create a new technote today by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap. -We currently use two technote series. The 'DMTN' is for general use within Data Management. -The 'SQR' technote series is used for SQuaRE science quality verification and infrastructure work. +We currently use two technote series. The :abbr:`DMTN (Data Management Technote)` is for general use within Data Management. +The :abbr:`SQR (SQuaRE Technote)` technote series is used for SQuaRE science quality verification and infrastructure work. .. _niche: @@ -21,22 +21,22 @@ We track work on JIRA. We document our code in software docs (the platform for which is being re-imagined by SQuaRE). We describe the design and architecture of the DM subsystem in change-controlled design documents. We publish articles in journals and conference proceedings where appropriate. -Finally, we have several Confluence wikis that traditionally served as a catch-all for DM documentation. +Finally, we have several Confluence_ wikis that traditionally served as a catch-all for DM documentation. Despite this array of platforms, a use case was still unserved and this technote platform was conceived to meet that need. In SQuaRE, the Science Quality and Reliability Engineering group, we realized that we often wanted to write stand-alone documents to describe our work. These documents didn't fit the conversational nature of `Community forum`_ topics, nor did they qualify for DM's change-controlled design document series, or fit into academic journals or the arXiv_. -Traditionally, Confluence was meant to fill this niche, and indeed, the wikis are well used by DM. +Traditionally, Confluence_ was meant to fill this niche, and indeed, the wikis are well used by DM. In practice, however, wikis have served DM poorly. -Confluence pages are hard to find and discover. -The editing and version control experience is also clumsy from the perspective of DM developers used to working on GitHub. +For readers, Confluence pages are hard to find and discover. +For authors, the editing and version control experience is also clumsy from the perspective of DM developers used to working on GitHub_. -In the fall of 2015, we experimented with moving several DM design documents from Word documents stored on Docushare to documents that are written in `reStructuredText`_, built with `Sphinx`_, hosted on GitHub, and deployed to the web with `Read the Docs`_. +In the fall of 2015, we experimented with moving several DM design documents from Word documents stored on Docushare_ to documents that are written in reStructuredText_, built with Sphinx_, hosted on GitHub_, and deployed to the web with `Read the Docs`_. The impetus behind the transition was that documents written in plain text and hosted on GitHub were far more likely to be kept up-to-date by DM developers. -Technotes were thus built on the same technology stack as the reStructuredText-based design documents, but re-purposed for general use by DM team members.\ [#]_ +Technotes were thus built on the same technology stack as the reStructuredText_-based design documents, but re-purposed for general use by DM team members.\ [#]_ -.. [#] In fact, this technote applies equally to describing the new reStructuredText-based design document platform as it does the technote platform. The only meaningful difference is that design documents must be approved by a control board, and are ultimately deposited in LSST's Docushare. +.. [#] In fact, this technote applies equally to describing the new reStructuredText_-based design document platform as it does the technote platform. The only meaningful difference is that design documents must be approved by a control board, and are ultimately deposited in LSST's Docushare_. Some of the possible applications for technotes are: @@ -52,9 +52,10 @@ Although technotes are still in an early, minimally-viable state, (see :ref:`imp Technotes are native to the web ------------------------------- -Although a preponderance of astronomy literature is published as PDFs, and many management documents as Word documents, those formats are archaic solutions from an era when documents were read in print. -Modern HTML, CSS and JavaScript allow for better, more innovative, and frictionless reading experiences than PDFs and Word documents can provide. -Modern readers expect to be able to search for content, click a link, and read it immediately in the browser. +Although a preponderance of astronomy literature is published as PDFs, and many management documents as Word documents, those formats are built on the assumption that documents are read in print. +These platforms are out of step with modern expectations that documents should be Googleable, linkable, and instantly readable in the browser. +Modern HTML, CSS, JavaScript, and webfonts are delivering reading experiences that are beginning to surpass the printed page. +Technotes are built for this modern communication era. Technotes are written in reStructuredText ----------------------------------------- @@ -65,11 +66,11 @@ For example, LaTeX, though common in astronomy, is tied too closely to PDF outpu `Markdown `_ is a hugely popular format among developers, in part because of its intuitive simplicity. However, this simplicity has prompted developers to `add non-standard extensions to the format `_ to their markup to add semantics and style to content. -Instead, reStructuredText is an ideal markup format for technotes. -reStructuredText's syntax and build process was designed to be extended, and in fact, those extensions are made in DM's primary language, Python. -Because the Python community has adopted reStructuredText as its official markup language, a our technote platform can take advantage of excellent open source tools, such as the Sphinx build system and the Read the Docs documentation hosting site. -For these same reasons, reStructuredText and the Sphinx build toolchain are also used by DM's new software documentation platform. -This allows both documentation platforms to benefit from the same bespoke infrastructure development, such as SQuaRE's `documenteer`_ package. +Instead, reStructuredText_ is an ideal markup format for technotes. +reStructuredText_\ 's syntax and build process was designed to be extended, and in fact, those extensions are made in DM's primary language, Python. +Because the Python community has adopted reStructuredText_ as its official markup language (including the scientific and astronomy Python community, led by `NumPy `_, `SciPy `_, `Matplotlib `_ and `AstroPy `_), a our technote platform can take advantage of excellent open source tools, such as the Sphinx_ build system and the `Read the Docs`_ documentation hosting site. +For these same reasons, reStructuredText_ and the Sphinx_ build toolchain are also used by DM's new software documentation platform. +This allows both documentation platforms to benefit from the same bespoke infrastructure development, such as SQuaRE's documenteer_ package. Technotes are single-page documents ----------------------------------- @@ -88,15 +89,19 @@ We use Zenodo_ as an archive and DOI provider, taking advantage of its GitHub in Technotes are versioned ----------------------- -Unlike scientific publications that are published once, technotes can and should be updated when appropriate. +Traditional scientific publications are published once, and updated only be authoring entirely new publications. +This necessitates a significant effort to update the literature and also makes it harder for the reader to understand what documents have been deprecated. +Unlike scientific publications that are published once, technotes can be updated when appropriate. +Technotes take a software development approach to publication by allowing technotes to be updated in-place when appropriate. The full version history is maintained by git and published on GitHub. +With GitHub's Zenodo_ integration, new releases are archived through Zenodo and given their own DOI (while also being linked to other versions). .. _implementation: Proof of Concept Implementation =============================== -We released a minimum-viable product for creating and publishing technotes. +We released a tool for creating and publishing technotes. Authors can create a technote by following the instructions at https://github.com/lsst-sqre/lsst-technote-bootstrap. Project automation @@ -112,36 +117,34 @@ By running cookiecutter https://github.com/lsst-sqre/lsst-technote-bootstrap.git the author is prompted to answer questions that configure the document. -When that is done, the author is left with a working Sphinx-based documentation project that can be immediately built with a ``make html`` command. +When that is done, the author is left with a working Sphinx_-based documentation project that can be immediately built with a ``make html`` command. This level of configuration automation is crucial to the adoption of tech notes, and :ref:`we intend to only increase this level of automation `. Document build configuration and metadata ----------------------------------------- -The Sphinx project prepared by `lsst-technote-bootstrap`_ appears conventional with the exception of how the Sphinx build is configured. -Most Sphinx projects have extensive ``conf.py`` files, which are ``execfile()``'d Python code that configure Sphinx and prepare the data available to document templates. -The Sphinx ``conf.py`` posed a maintenance threat to technotes: any infrastructural change to the Sphinx build system for technotes would require edits to the ``conf.py`` files of every technote and DM design document. -Our solution was to strip nearly all logic from the ``conf.py`` files, and centralize all configuration management in our documenteer_ Python package. +The Sphinx_ project prepared by `lsst-technote-bootstrap`_ appears conventional with the exception of how the Sphinx_ build is configured. +Most Sphinx_ projects have extensive :file:`conf.py` files, which are ``execfile()``'d Python code that configure Sphinx_ and prepare the data available to document templates. +The Sphinx_ :file:`conf.py` posed a maintenance threat to technotes: any infrastructural change to the Sphinx_ build system for technotes would require edits to the :file:`conf.py` files of every technote and DM design document. +Our solution was to strip nearly all logic from the :file:`conf.py` files, and centralize all configuration management in our documenteer_ Python package. Now, single commits to documenteer_ are effectively deployed instantly to all technotes. Of course, individual technotes need custom configuration, such as title and authorship information. -We keep this in a ``metadata.yaml`` file in each technote repository. -By effectively refactoring metadata out of both ``conf.py`` *and* the reStructuredText content, it is easy to develop a standardized schema for describing technotes. +We keep this in a :file:`metadata.yaml` file in each technote repository. +By effectively refactoring metadata out of both :file:`conf.py` *and* the reStructuredText_ content, it is easy to develop a standardized schema for describing technotes. See :ref:`metadata`. Such a schema opens opportunities for indexing DM's technote library. Deployment ---------- -GitHub is the central infrastructure for hosting technotes. -The ``master`` branch is considered a live publication, but 'releases' can be made as well using git tags or the GitHub release feature. +GitHub_ is the central infrastructure for hosting technotes. +The ``master`` branch is considered a live publication, but 'releases' can be made as well using git tags or the GitHub Release feature. -Technotes are published on `Read the Docs`_ a free and open-source platform for publishing Sphinx-based documentation, such as technotes. -`Read the Docs`_ integrates with GitHub to rebuild the technote's webpage whenever commits are pushed to the technote's ``master`` branch on GitHub. +Technotes are published on `Read the Docs`_ a free and open-source platform for publishing Sphinx_-based documentation, such as technotes. +`Read the Docs`_ integrates with GitHub_ to rebuild the technote's webpage whenever commits are pushed to the technote's ``master`` branch on GitHub_. We serve technotes as a subdomain of ``lsst.io``, e.g., http://sqr-000.lsst.io. -.. _`Read the docs`: http://readthedocs.org - Finally, major versions of the technote can be granted DOIs. The technote repository can be connected to Zenodo_. When a major version of a technote is completed, a GitHub Release can be made, and the contents of the technote repository are uploaded and archived on Zenodo_. @@ -158,12 +161,13 @@ Improved document creation and management automation Although lsst-technote-bootstrap_ automates report creation, there are still many facets of technote authorship that would benefit from automation: #. additional automation of technote configuration, beyond what cookiecutter_ provides (such as dynamic date suggestions) -#. creation of a `GitHub`_ repository +#. creation of a GitHub_ repository #. creation and configuration of a `Read The Docs`_ project #. provisioning of an ``lsst.io`` domain -#. reStructuredText and metadata linting (using `Travis CI `_ testing) +#. reStructuredText_ and metadata linting (using `Travis CI `_ testing) +#. synchronization of metadata version tags and revision dates with git history #. automatic local builds and browser updates (e.g., `Browsersync `_) -#. automation of releases and procurement of DOIs (leveraging ``metadata.yaml`` to automate the technote's deposition on Zenodo_) +#. automation of releases and procurement of DOIs (leveraging :file:`metadata.yaml` to automate the technote's deposition on Zenodo_) This likely demands a command line application to manage technotes, which would incorporate lsst-technote-bootstrap_. Likely the most challenging aspect will be automating the creation of a `Read the Docs`_ project, since project creation is not part of `RTD's API `_. @@ -171,27 +175,28 @@ Likely the most challenging aspect will be automating the creation of a `Read th Improved presentation --------------------- -Technotes are currently published with Read the Doc's default theme (including minor additions to incorporate metadata from ``metadata.yaml``). +Technotes are currently published with Read the Doc's default theme (including minor additions to incorporate metadata from :file:`metadata.yaml`). A new HTML/CSS theme is needed to - establish a visual identity for DM documents - provide allowances for navigation in long single page documents -- add facilities for styling elements created by an extended reStructuredText language (rather than retrofitting an existing theme) +- add facilities for styling elements created by an extended reStructuredText_ language (rather than retrofitting an existing theme) +- improve layout for print Extensions to reStructuredText ------------------------------ -DM authors need a richer reStructuredText language for technical writing. -One request is to implement citations and bibliographies of the same quality as are possible with LaTeX and natbib. -We can achieve this by developing Sphinx extensions within the documenteer_ package. +DM authors need a richer reStructuredText_ language for technical writing. +One need is to have citations and bibliographies of the same quality as are possible with LaTeX and `natbib `_. +We can achieve this by developing `Sphinx extensions `_ within the documenteer_ package. Development work done here will also benefit DM's software documentation. A document index ---------------- -From experience with Docushare and the Confluence wikis, we learnt that documentation can be easily buried if not indexed from a central, authoritative, reliable and highly visible place. +From experience with Docushare_ and the Confluence_ wikis, we learnt that documentation can be easily buried if not indexed from a central, authoritative, reliable and highly visible place. We need to provide a documentation index for DM, likely as part of http://dm.lsst.org. -The page could be automatically updated by leveraging the GitHub API and individual documents' ``metadata.yaml`` information. +The page could be automatically updated by leveraging the GitHub_ API and individual documents' :file:`metadata.yaml` information. Ideally, the index would provide facilities for filtering or searching. .. _metadata: @@ -199,7 +204,7 @@ Ideally, the index would provide facilities for filtering or searching. Metadata Standard ================= -Here we document the available keys in the ``metadata.yaml`` schema. +Here we document the available keys in the :file:`metadata.yaml` schema. series: A string identifying the technote series. @@ -214,7 +219,7 @@ series: series_number: Serial number of the document, as a string. - For the DMTN and SQR series we use three digit serial numbers (with leading zeros). + For the :abbr:`DMTN (Data Management Technote)` and :abbr:`SQR (SQuaRE Technote)` series we use three digit serial numbers (with leading zeros). Example: @@ -233,7 +238,7 @@ doc_id: authors: Author names, ordered as a list. - Each author name should be formatted as 'First Last' + Each author name should be formatted as 'First Last'. Example: @@ -298,7 +303,7 @@ copyright: Planned metadata extensions --------------------------- -We plan to add the following fields to the ``metadata.yaml`` schema. +We plan to add the following fields to the :file:`metadata.yaml` schema. description: A short 1-2 sentence description for document indices. @@ -335,7 +340,7 @@ docushare_url: docushare_url: 'https://docushare.lsstcorp.org/docushare/{{ path }}' github_url: - The document's URL on GitHub. + The document's URL on GitHub_. Example: @@ -375,4 +380,6 @@ ORCID_ iD integration would be used to improve the Zenodo_ submission process. .. _ORCID: http://orcid.org/ .. _reStructuredText: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html .. _Sphinx: http://sphinx-doc.org -.. _Read the Docs: http://readthedocs.org +.. _Read the Docs: https://readthedocs.org +.. _Docushare: https://docushare.lsstcorp.org/docushare/dsweb +.. _Confluence: https://confluence.lsstcorp.org/ From bade5d72a3020161096a08c592510cf62da3f5d6 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 23 Nov 2015 18:12:28 -0700 Subject: [PATCH 12/16] Update proposed metadata - Add deprecated field, and changes field - Note that version and last_revised *could* be machine-generated in the future For DM-4351. --- index.rst | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/index.rst b/index.rst index 55f3e52..42c2220 100644 --- a/index.rst +++ b/index.rst @@ -266,6 +266,8 @@ version: version: '0.1.dev' + This metadata may be replaced by tooling that uses git tags. + doi: Digital Object Identifier (DOI). Keep this DOI updated as new releases are pushed to Zenodo_. @@ -291,6 +293,8 @@ last_revised: '2015-11-18' + This metadata may be replaced by tooling that uses git history. + copyright: Copyright statement. @@ -304,6 +308,7 @@ Planned metadata extensions --------------------------- We plan to add the following fields to the :file:`metadata.yaml` schema. +These metadata fields are not currently in use, and are liable change prior to implementation. description: A short 1-2 sentence description for document indices. @@ -348,6 +353,31 @@ github_url: github_url: 'https://github.com/lsst-sqre/sqr-000' +deprecated: + This field can be added if the document has been superseded. + The deprecation notice may contain several fields, for example: + + .. code-block:: yaml + + deprecated: + date: 'YYYY-MM-DD' + superseded_by: 'http://{{url of new doc}}' + +changes: + A changelog. + The DM design documents currently embed change tables embedded in the content, but this would useful as independent metadata. + + .. code-block:: yaml + + changes: + - + tag: v1.0 + notes: 'First version' + - + tag: v2.0 + notes: 'Second version' + + Leveraging ORCID for Author Information --------------------------------------- From 3132de22a16d48b0ada68f2d9664196818c71ef7 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 23 Nov 2015 18:18:57 -0700 Subject: [PATCH 13/16] Add Acknowledgements section --- index.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/index.rst b/index.rst index 42c2220..5f43d7a 100644 --- a/index.rst +++ b/index.rst @@ -399,6 +399,12 @@ A possible revised syntax for declaring authorship metadata would be ORCID_ iD integration would be used to improve the Zenodo_ submission process. +Acknowlegements +=============== + +J. S. would like to thank Frossie Economou, Tim Jenness and Josh Hoblitt for authoring early technotes and providing feedback on the platform. +Tim Jenness and JMatt Peterson provided valuable editorial feedback on this technote. + .. _Zenodo: https://zenodo.org .. _GitHub: https://github.com .. _Community forum: https://community.lsst.org From 34deda9e0d53f72646d04e82b322805b71c7cfb1 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 23 Nov 2015 22:58:54 -0700 Subject: [PATCH 14/16] Use documenteer==0.1.0 --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index b458a12..4f91ed0 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,4 @@ PyYAML>=3.11 Sphinx>=1.3.1 +documenteer==0.1.0 -e git://github.com/lsst-sqre/lsst_dd_rtd_theme.git@master#egg=lsst_dd_rtd_theme --e git://github.com/lsst-sqre/documenteer.git@master#egg=documenteer From 8ef4c77c165593dd1d37de573bf645e537981577 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 23 Nov 2015 23:27:14 -0700 Subject: [PATCH 15/16] Pre-v1.0 proofreading pass For DM-4351. --- index.rst | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/index.rst b/index.rst index 5f43d7a..cae36ea 100644 --- a/index.rst +++ b/index.rst @@ -62,13 +62,13 @@ Technotes are written in reStructuredText A fundamental requirement of the technote platform is that documents are written in a plain text format, and are then compiled into a website. This way, content is written in common text editors and hosted on GitHub for collaboration (the same workflow we already use for source code). Many plain text formats have been invented for writing, however, not all formats are ideal for technotes. -For example, LaTeX, though common in astronomy, is tied too closely to PDF output and makes little sense for a web-native syntax. +For example, LaTeX, though common in astronomy, is tied too closely to PDF output and was never meant to be a web-native markup. `Markdown `_ is a hugely popular format among developers, in part because of its intuitive simplicity. -However, this simplicity has prompted developers to `add non-standard extensions to the format `_ to their markup to add semantics and style to content. +However, this simplicity has prompted developers to `add non-standard extensions to the format `_ to add semantics and style to content. Instead, reStructuredText_ is an ideal markup format for technotes. reStructuredText_\ 's syntax and build process was designed to be extended, and in fact, those extensions are made in DM's primary language, Python. -Because the Python community has adopted reStructuredText_ as its official markup language (including the scientific and astronomy Python community, led by `NumPy `_, `SciPy `_, `Matplotlib `_ and `AstroPy `_), a our technote platform can take advantage of excellent open source tools, such as the Sphinx_ build system and the `Read the Docs`_ documentation hosting site. +Because the Python community has adopted reStructuredText_ as its official markup language (including the scientific/astronomy Python community, led by `NumPy `_, `SciPy `_, `Matplotlib `_ and `AstroPy `_), our technote platform can take advantage of excellent open source tools, such as the Sphinx_ build system and the `Read the Docs`_ documentation hosting site. For these same reasons, reStructuredText_ and the Sphinx_ build toolchain are also used by DM's new software documentation platform. This allows both documentation platforms to benefit from the same bespoke infrastructure development, such as SQuaRE's documenteer_ package. @@ -89,10 +89,7 @@ We use Zenodo_ as an archive and DOI provider, taking advantage of its GitHub in Technotes are versioned ----------------------- -Traditional scientific publications are published once, and updated only be authoring entirely new publications. -This necessitates a significant effort to update the literature and also makes it harder for the reader to understand what documents have been deprecated. -Unlike scientific publications that are published once, technotes can be updated when appropriate. -Technotes take a software development approach to publication by allowing technotes to be updated in-place when appropriate. +Technotes take a software development-inspired approach to publishing by allowing technotes to be updated in-place when appropriate. The full version history is maintained by git and published on GitHub. With GitHub's Zenodo_ integration, new releases are archived through Zenodo and given their own DOI (while also being linked to other versions). @@ -141,7 +138,7 @@ Deployment GitHub_ is the central infrastructure for hosting technotes. The ``master`` branch is considered a live publication, but 'releases' can be made as well using git tags or the GitHub Release feature. -Technotes are published on `Read the Docs`_ a free and open-source platform for publishing Sphinx_-based documentation, such as technotes. +Technotes are published on `Read the Docs`_, a free and open-source platform for publishing Sphinx_-based documentation, such as technotes. `Read the Docs`_ integrates with GitHub_ to rebuild the technote's webpage whenever commits are pushed to the technote's ``master`` branch on GitHub_. We serve technotes as a subdomain of ``lsst.io``, e.g., http://sqr-000.lsst.io. @@ -365,7 +362,7 @@ deprecated: changes: A changelog. - The DM design documents currently embed change tables embedded in the content, but this would useful as independent metadata. + The DM design documents currently embed change tables in the content, but this would be more useful as independent metadata. .. code-block:: yaml From b50b0f35abe16a43a350711d829612d35d4192c4 Mon Sep 17 00:00:00 2001 From: Jonathan Sick Date: Mon, 23 Nov 2015 23:28:32 -0700 Subject: [PATCH 16/16] Mark v1.0 For DM-4351. --- metadata.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/metadata.yaml b/metadata.yaml index a70fc4b..f8e3fe6 100644 --- a/metadata.yaml +++ b/metadata.yaml @@ -20,11 +20,11 @@ authors: - 'Jonathan Sick' # Current document version -last_revised: '2015-11-18' +last_revised: '2015-11-23' # Version. Use Semvar, e.g., 1.0, including .dev, as necessary # This version string should correspond to the git tag when the document is published on Zenodo -version: '0.1.dev' +version: '1.0.0' # Digital Object Identifier (DOI). Uncomment if available. # Keep this DOI updated as new releases are pushed to Zenodo