Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Processes stats v2 and new Hermes API concept #3180

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

jeanleonov
Copy link
Contributor

@jeanleonov jeanleonov commented Sep 30, 2019

This branch was inspired by ELK and how processes stats can be analyzed there (and other types of stats actually). It doesn't modify existing functionality, but extends it with a new v2 API which currently supports only processes stats.

I'd highlight two aspects of changes:

  1. Processes stats v2 structure provides more easy-to-use properties comparing to v1 (which is not removed btw).
  2. New API v2 concept.

1. Processes stats v2 vs v1

Instead of 3 categorization fields monit_name, unified_service_name and application_id, in version two there are two repeated fields own_tags and all_tags, which suppose to get values like [appscale, solr] or [appscale, searchservice] or [<rabbitmq-childprocess-name>, appscale, rabbitmq] etc.
own_tags holds only explicitly assigned tags like [appscale, datastore] or a single [<process_name>]. all_tags contains own + ancestors' tags.
Such tagging allows flexible and powerful processes filtering in ELK.

Additionally to cumulative CPU and IO counters, v2 brings *_1h_diff fields which provides estimated hourly diff since previous measurement. So it suppose that a single client will request processes stats regularly. Otherwise estimated diffs might be inaccurate.

2. API v2 concept

Objects provided by API v2 suppose to be flat which matches analytic needs (like ELK) much better.
It doesn't support include lists. It should simplify caching on proxy side and simplify collecting stats from multiple nodes without need to parse and serialize JSON (just joining responses).

@jeanleonov

This comment has been minimized.

@jeanleonov
Copy link
Contributor Author

Recommended to go after #3165 and it may have a conflict with it in solr management command related to sudo removal.

@jeanleonov jeanleonov marked this pull request as ready for review October 4, 2019 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant