Tremor is a cloud-native event-processing engine designed for operator and systems reliability engineer convenience in the design, development, and support of continuous streaming systems deployed and operated at scale.
Tremor was initially designed for the needs of Wayfair’s observability platform engineering teams within our infrastructure organization and was first deployed to production in October 2018 for load-shedding and traffic-shaping of massive logging and metrics workloads. It has since been adopted for transaction orchestration, and our development platforms teams for dynamic code analysis on production PHP deployments to minimize the observable overhead.
Tremor was open-sourced in February 2020, and adopted into the CNCF sandbox in September 2020.
The Tremor devstats page can be found here.
Tremor is a runtime environment that supports pluggable connectors to consume and contribute events to the outside world. Tremor supports a growing number of connectors to systems such as Kafka, InfluxDB via Telegraf, ElasticSearch through its REST API, the Open Telemetry suite and a number of Google Cloud services.
Tremor's language is a holistic aproach to configuring the system starting from defining deployments, connecting components using a query-like syntax and an ETL focus that allows for the creation of pipelines that can be used to transform and enrich events.
Tremor has seven primary maintainers from three organisations:
- Darach Ennis and Heinz Gies founded the project and continue to contribute.
- Matthias Wahl has been with the project since September 2020.
- Sharon Koech is a graduated LFX Mentee who redesigned the Tremor www and is now our lead documentation maintainer and technical writer.
- Ramona Luczkiewicz has been with the project since April 2022.
- Natali Vlatko of Wayfair’s OSPO and Ana Hobden of Hoverbear Consulting are also maintainers of Tremor and assist with mentee selection and with community moderation to ensure that the code of conduct is adhered to by all members of the Tremor community's Discord Server.
This is a community-focused release. Feedback from our LFX mentorships was that writing connectors was hard - this promoted the connectors refactoring in the 0.12 release. The release is a major rewrite of the connector internals and related runtime facilities to deliver better consistency to users and a better programming model for connector authors.
The deployment language work was feedback from production users that favored dropping YAML-based deployment configuration for unifying deployment configuration with the Tremor language. The connectors refactoring allowed for the introduction of the deployment language, associated runtime and API changes to be udertaken holistically.
The work in progress 0.13 release concentrates on Quality of Life improvements over 0.12 that introduce minor backwards compatibility breaks for application writers based on feedback from early adopters of the new deployment model that have migrated applications to the new model.
RAFT-based consensus mechanism for maintaining cluster state such as registered applications and pipelines.
This provides the basic capabilities for nodes to join and leave the cluster enabling auto-scaling and a shared cluster-wide configuration.
The plugin development kit is a huge contribution to the Tremor ecosystem. It allows for the creation of Tremor connectors that are not part of the core binary. This makes contributing to Tremor a significantly easier task and allows for the creation of connectors that are separately maintained. The PDK will be productized into Tremor in a future release once performance overhead is brought into line with the current release.
Sandbox proposal PR #492, content and inclusion references cited for completeness.
Launch of the Tremor communityon Discord - this has grown to 235 members, 30 of which are online on a daily basis.
Most of the day to day coding, design work, refactoring, bug hunting sessions and planning are now conducted in the open on Discord in the General channel or the Summer or Winter voice/live presentation channels - and we are continuously evolving our governance so that our processes, practices and decision-making are by the community, for the community.
In addition, we hold bi-weekly office hours for the community.
For the second year in a row, we held TremorCon- a one-day annual Tremor conference showcasing the work of our 2nd and 3rd LXF mentorship cohorts and community members. This was hosted on Twitch, and the videos are archived on YouTube.
The initial launch of Tremor use-case studies for Wayfair’s ongoing adoption and sponsoring of Tremor.
Tremor hosts open office hours for end-users and contributors bi-weekly as part of the Tremor CNCF Community. We have begun canvassing for adoption case studies in this and other forums with our starting set of adoption case studies for community reference and guidance. A list can be found in the Incubation Stage Requirements section.
In addition, we had some adopters talk at TremorCon 2022.
- Sasha Pourcelot (
@scrabsha) - Database Connectors. - Prashant Mishra (
@PrimalPimmy) - CI and Release Process Improvements.
- Rebecca Abli (
@znwar) - Pluggable Logging. - Carol Geng (
@carolgeng) - Hygenic Error Handling and Validation for Pipelines.
Document That is Being Used Successfully in Production by at Least Three Independent End-users Which, in the TOC’s Judgment, are of Adequate Quality and Scope
We have documented a number of use-cases, beginning with our originating sponsor and founding adopter - Wayfair.
In addition to that, we have a number of organic adoptions from the community:
Tremor's core codebase has contributions from at least 17 different organizations and more than 42 different individuals. This does not count contributions to the web presence, documentation or supporting libraries.
Out of the last 100 merged pull requests to the main repository, 22 were made from non-core contributors (this excludes Dependabot or other automated PRs). The community contributions are significant and valuable in their own right and add a lot to the core value of Tremor as a solution for prospective adopters.
This puts Tremor close to over 20% ratio of team to community PRs.
The project's versioning scheme conforms to the semantic versioning scheme and is implemented as follows:
- Major: currently 0 as the core implementation is active and in progress.
- Minor: is incremented for new major features or breaking changes.
- Patch: is incremented for bug fixes and minor new features.
The project has a 3-6 month release cadence for minor increments since version v0.7.0. Patch releases are produced continuously as production needs demand.
For our releases, maintainers follow the release process now fully run by an automated release process contributed by a CNCF mentee.
For security patch releases, we follow a separate disclosure process and announce CVEs accordingly. For example, we have published one CVE against Tremor for a language behavior that could have been exploited and was fixed in a security patch release.
We do not currently have a roadmap available as a publicly accessible GitHub project but are prototyping the changes to governance, documentation, and processes that would support a roadmap for the current release, a tentative roadmap for the next release, and a horizon looking 2 releases into the future.
These changes are forward-looking.
In broad terms, and subject to community feedback and changes, there are:
RAFT-based consensus mechanism for maintaining cluster state such as registered applications and pipelines.
This provides the basic capabilities for nodes to join and leave the cluster enabling auto-scaling and a shared cluster-wide configuration.
Introduction of a micro / macro architecture using a ring-based topology for the macro ring in the spirit of Riak's V-Nodes for ease of scalabiltiy
As a production-focused project, Tremor maintainers favor essential production-impacting features, enhancements and fixes over roadmap completeness.
This set of practical planning and delivery constraints of roadmap definition and release strategy against a planned and visible roadmap is not thoroughly documented at this time. This is acknowledged as a flaw in our governance model and is being reviewed.
The V0.12 release will be the final release in the current governance model that was introduced with the open-sourcing of the project. Subsequent releases will use an evolution of the governance model driven by community feedback and supported by community decisions in regularly held Tremor Open Office Hours where changes are discussed, feedback gathered and decisions voted upon.
We have referenced the known governance model issues in this review document.