Skip to content

Latest commit

 

History

History
403 lines (338 loc) · 18.7 KB

0004-tremor.md

File metadata and controls

403 lines (338 loc) · 18.7 KB
cSpell:ignore
Horgan onramps offramps LLDB Wayfair

Assessment: Project Tremor

Prepared by: Celeste Horgan Date: July 2021

Introduction

This document assesses the quality and completeness of a project's documentation and website (if present).

This document:

  • Measures existing documentation quality against the CNCF’s standards
  • Recommends specific and general improvements
  • Provides examples of great documentation as reference
  • Identifies key improvements with the largest return on investment

How this document works

The assessment is divided into three sections:

  • Project documentation: for end users of the project; aimed at people who intend to use it
  • Contributor documentation: for new and existing contributors to the project
  • Website: branding, website structure, and maintainability

Each section rates content based on different criteria.

Project documentation

Criteria 1 2 3 4 5
Information architecture ✅
New user content ✅
Content maintainability ✅
Content creation processes ✅

Scale:

  • 1 = (Is not present or requires significant work)
  • 3 = (Is present, but needs work)
  • 5 = (Is executed extremely well or no improvement required)

Comments

  • Information Architecture: Project Tremor's documentation makes a number of individually small IA choices that culminate in a confusing experience for the user.

    • The Getting Started section covers a number of seemingly random topics, with some appearing to be focused on selling the project to new users (i.e. Good User Experience), more traditional "getting started" content (Quick Install), and high-level conceptual documentation (Understanding Data).
    • Tremor has ample high-level conceptual overview information, but it's somewhat scattered.
      • The Architecture Overview - Goodness of fit section is probably the best overview for brand new users available, however the rest of the page gets very detailed very quickly, and feels like a collection of semi-random topics that someone felt the need to write down, with no real direction. This page probably needs to be broken up into many pages within a section, with some of the more esoteric topics currently under "Getting Started" included.
      • Low-level task content often gets buried in Recipes, generically titled Walkthrough pages, and the extremely helpful but poorly named Workshops section. This makes it hard for a user with a specific query ("How do I do....") to know, at a glance, if the documentation contains the answer to their question or not.
      • The Configuration walkthrough was the single most useful page in terms of understanding the overall 'picture' of what Tremor is and how a developer needs to move through the documentation. But it's buried in the Configuration section, and I wouldn't think to look there until I was much further in the process.
      • Given how valuable the Workshops section is, it needs a better name. "Recipes", "Tutorials", "How to" would all be clearer.
    • Some specific comments about the Tremor Query and Script sections:
      • The Tremor Query Overview and Tremor Script Overview would benefit from splitting the "Language" topic and subtopics into their own page in the section.
      • As someone unfamiliar with the project, I'm not sure how Tremor Query and Tremor Script are different based purely on their overviews. I'm also not sure if they work together or not, or if I as a user need to use one language versus another.
      • Given the complexity of Tremor's various query methods, I think focusing on producing more Recipes would be a helpful next step for the team.
    • History probably doesn't belong as a top-level section.
    • Development and Governance are targeted at contributor users rather than end users, and should be in a different section of the site if possible.

  • New user content: Project Tremor has a Getting Started section but it's a little disorganized as mentioned above. This makes it difficult for an actual new user to understand.

    • Starting Tremor for the first time is the best part of the getting started guide. However, it mentions that there are "many ways" to install Tremor - it would be good to point users to documentation on those other ways of installing.
    • Talking to other systems is definitely a foundational concept for Tremor (as an events processing system) but the style in which it's written isn't appropriate for a getting started guide. This goes for Understanding Data, Scripting and Specialize Tremor Pipelines.
      • These topics should be under Docs in their own section, perhaps titled "Overview".
    • Ensuring that all pages, like The Tremor CLI tool, have overviews goes a long way in ensuring the docs are user friendly for new users.
    • I like the FAQ as new user content, but FAQs often become a catch-all and get messy quickly. Proceed with caution.

  • Content maintainability:

    • Docs are mostly separated into their own repository, https://github.com/tremor-rs/tremor-www-docs, though as the maintainers note there are many other repositories with docs available in them. These need to be consolidated for easier maintenance.
    • The scripts which generate certain pieces of documentation introduce fragility into the system: if the location of files, names of repos, etc. change then the documentation pipeline breaks. This also makes it hard for non-"technical" users to make changes to these pages, and to reorganize these pages if needed.
      • The automation benefits might be worth it however.
    • Your release process is documented, which is great to see!
    • Your site is only partially searchable, with sections not within /docs inaccessible by search.

  • Content creation processes: There are no content creation processes documented and no CONTRIBUTING.md file for the main docs repository. Some things to think about:

    • If I write documentation, who verifies that it's technically accurate?
    • If I'm not a native English speaker, who can I ask for help with grammar and language review?
    • If I'm a trained technical writer and want to rearrange, create new, or split existing topics, who do I tag in issues/on Slack for comment?
    • As a developer user, how can I ensure that content is accurate and up to date? Are alpha and beta features flagged? What happens when content (+ functionality) gets deprecated or changed in a major way?
    • Who reviews documentation PRs?

Recommendations

  • Get specific about your Getting Started: Getting Started Guides contain a very predictable set of content:

    • In one paragraph or less, what is this piece of software
    • What prerequisite software (i.e. Kubernetes) and knowledge (i.e., Go) do I need on my system to launch this?
      • What special features does this piece of software have that other similar software might not have?
    • What are the installation instructions?
    • Is there a sample application I can set up and execute basic commands on?

  • Reorganize the Operations section and retitle it "Using Tremor": Additionally, move some topics from the existing Getting Started section:

  • Reorganize Overview with user-centeredness in mind: I suggest creating a new section with the following content:

  • Audit for colloquial/casual language, abbreviations, and other "basic" English:

    • This recommendation has less to do with casual language or abbreviations being bad, and more to do with the perception of professionalism for the project.
    • Review all documentation for the following:
      • Unnecessary abbreviations (for example This page).
      • Casual language or colloquialisms, for example: A short "canned" synopsis, referring to processes either as painless (on the same page: "let's be real"), etc.
      • Spelling, consistent capitalization of titles, consistent verbiage in titles:

  • Rename Workshops section to "Recipes" or "Examples"

  • Create a CONTRIBUTING.md file for the docs repository

Contributor documentation

Criteria 1 2 3 4 5
Communication methods documented ✅
Beginner friendly issue backlog ✅
“New contributor” getting started content ✅
Project governance documentation

Scale:

  • 1 = (Is not present or requires significant work)
  • 3 = (Is present, but needs work)
  • 5 = (Is executed extremely well or no improvement required)

Comments

  • Communication methods documented: Contributing documents these, but it would be useful to pull these up to the Community page

  • Beginner friendly issue backlog: You document your GitHub best practices and issues for new users are tagged. Great work!

    • One thing to note is when tagging an issue as a good first issue, assume the reader knows nothing and must be hand-held through the entire process. This issue is a great example of the granularity needed by a first time contributor.

  • New contributor getting started content: The Community page exists but doesn't provide much guidance for new users. How do I get involved? When do community meetings happen? Where do I find the team on Discord or Slack?

    • As mentioned above, Development and Governance are targeted at contributor users and should be under the community section.

  • Governance documentation: Your Project Governance is linked front and center, which I love! The only critique I have is that as a new user I am unlikely to click through to all of the links available. It would be lovely to distill these documents into something Tremor maintains at some point in the future.

Recommendations

  • Turn the Community page into a subsection and house Governance and Development documentation under it:

  • Pull meeting and contact information out of contributing and onto the community page: people cannot join a meeting or chat that they can't find a link to.

Website

Criteria 1 2 3 4 5
Branding and design ✅
Case studies/social proof ✅
Maintenance planning ✅

Scale:

  • 1 = (Is not present or requires significant work)
  • 3 = (Is present, but needs work)
  • 5 = (Is executed extremely well or no improvement required)

Comments

  • Branding and design:

    • On the whole, the Tremor docs site looks professional, if a little plain.
    • Because of the way the site deploys (from different repositories via GitHub pages), navigation is disjointed: if you're in the RFCs, you don't have the full site's navigation available to you and clicking the logo takes you back to https://rfcs.tremor.rs, with similar behavior occurring in the docs. This is a bad user experience.
    • Tremor could do a bit more selling on its homepage:
      • What are the project's key features that make it different from other event processing systems?
      • Are there any known use cases you can talk about or use the logos of?
      • Have you given any talks on Tremor, or is there other (particularly video) content you can showcase about the project?

  • Case studies/social proof: None available.

    • Case studies/talks/generally showing that other projects use Tremor is one of the most powerful ways to gain new users. People love real-world examples of a codebase in action.
    • Because Tremor was developed by Wayfair, we suggest documenting Wayfair's use as a "use case" or case study, and then adding a section on the website which allows people to submit their own use cases!

  • Maintenance planning: In general, we recommend that projects keep their documentation all in one repository. Tremor's documentation currently deploys from https://github.com/tremor-rs/tremor-www-main, https://github.com/tremor-rs/tremor-rfcs, and https://github.com/tremor-rs/tremor-www-docs.

    • It's confusing for contributors who don't know where to file issues, open pull requests, or whom to tag for reviews (and where).
    • The easiest fix for this pain point is to deploy everything using and Hugo.
    • In general, we prefer sites to deploy via Netlify.

Recommendations

  • Develop a case study or other content around Wayfair's use of Tremor, to give users a tangible example of what they can do with the project.

  • Fix navigation and improve site maintainability by putting all your documentation in one repository.

Recommendations

Reorganize your documentation's information architecture: In addition to the suggestions above, I recommend an overall reorganization of the documentation, along the lines of:

  • Docs
    • Overview (renamed from Architecture Overview)
    • Using Tremor
      • subpages as described previously
    • Artifacts
    • Tremor Query
    • Tremor Script
    • Examples (renamed from Workshops)
    • API Reference
    • Glossary
    • FAQ (moved from top level navigation)

Reorganize the top-level navigation of the site: once again, as mentioned previously:

  • Getting Started
  • Docs
    • subsections as described above
  • Community
    • RFCs
    • Contributing (moved from Docs section)
    • Governance (moved from Docs section)
  • Community Chat
  • Blog

Consolidate documentation into one repository, and document contributing guidelines for users: I recommend collapsing everything into https://github.com/tremor-rs/tremor-www-main, barring any technical difficulties doing so, and switching deploys over to Netlify to access Deploy previews and other useful features.