PMM-6579 Add monitoring for current and recent slow queries

[stefano-pogliani]

Hello 👋

We would like to monitor queries recorded in the system.profile collection and running queries for queries slower then a configurable threshold.
This change would add two optional metrics, disabled by default, for slow queries in system.profile and for slow queries currently running.

I'd be happy to iterate over this if it is of interest.

[it-percona]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ stefano-pogliani
❌ Anton Kucherov

---

Anton Kucherov seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

Merging #157 (ef222f3) into master (07caa60) will increase coverage by 2.35%.
The diff coverage is 48.08%.

@@            Coverage Diff             @@
##           master     #157      +/-   ##
==========================================
+ Coverage    9.82%   12.18%   +2.35%     
==========================================
  Files          46       47       +1     
  Lines        2452     2609     +157     
==========================================
+ Hits          241      318      +77     
- Misses       2178     2250      +72     
- Partials       33       41       +8     

Impacted Files	Coverage Δ
mongodb_exporter.go	32.72% <0.00%> (+0.58%)	⬆️
collector/mongodb_collector.go	55.21% <6.66%> (-4.33%)	⬇️
collector/mongod/database_profiler.go	55.12% <55.12%> (ø)
shared/connection.go	45.91% <100.00%> (-1.26%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07caa60...ef222f3. Read the comment docs.

[idexter]

Hi @stefano-pogliani . We also use dep, for backward compatibility. I see new dependencies in go.mod file. Could you update those dependencies with dep, please?

Also, it would be great if you could add a few tests. At least for GetDatabaseProfilerStats() and GetDatabaseCurrentOpStats() methods, but it's up to you.

[alitoufighi]

I have a question. Isn't it possible to have something to show what queries are taking too much to complete? For example, by exporting execution time of queries. I know there is same thing for this PostgreSQL exporter which also exports calls per min.

[stefano-pogliani]

Hello @dexterHD,

We also use dep, for backward compatibility. I see new dependencies in go.mod file. Could you update those dependencies with dep, please?

I am very new to go so apologies in advance for the silly question: I have run dep ensure --update and I now see many changes in the vendor directory, is this what you were looking for?

Also, it would be great if you could add a few tests.

I will look into adding some tests.

[stefano-pogliani]

Isn't it possible to have something to show what queries

@alitoufighi you mean the actual query document?

I know there is same thing for this PostgreSQL exporter

I had a quick look and I'm not sure which bit of the exporter you are referring to.
Do you know what code or metric exports this information?

[alitoufighi]

@alitoufighi you mean the actual query document?

Yes. Take a look at this.

I had a quick look and I'm not sure which bit of the exporter you are referring to.
Do you know what code or metric exports this information?

Sorry to waste your time, my bad. My colleagues had used its adding new metrics via config file option to use above utility provided by PostgreSQL. This is the yaml we're using for PSQL:

pg_query_stats:
  query: "SELECT d.datname, LEFT(s.query, 200) as query, sum(s.calls) as calls, sum(s.total_time) as total_time, min(s.min_time) as min_time, max(s.max_time) as max_time FROM pg_stat_statements s inner join pg_database d on s.dbid = d.oid group by d.datname, LEFT(query, 200) order by sum(calls) desc"
  metrics:
    - datname:
        usage: "LABEL"
        description: "Database name"
    - query:
        usage: "LABEL"
        description: "SQL query"
    - calls:
        usage: "COUNTER"
        description: "Number of calls"
    - total_time:
        usage: "DURATION"
        description: "Execution time of query"
    - min_time:
        usage: "DURATION"
        description: "Min execution time of query"
    - max_time:
        usage: "DURATION"
        description: "Max execution time of query"

I wonder if there is such utility in MongoDB. I just read about explain.execusionStats(), but it doesn't seem to be able to be used as an automated way for use in monitoring.

EDIT: I saw something similar in query fileld of returned data of db.system.profile.find() in the profiler you're working on.

[stefano-pogliani]

Hi @alitoufighi,

The query field is needed the full operation that was executed.
This is stored as a structured document with all the query values set in it so it can have an extremely high carnality: if you have a slow query that includes an account_id you would get a different label value for each account that runs the query.

It would be a lot of work (potentially adding significant complexity) to map queries to query shapes.
It seems in MongoDB 4.2, released a couple of weeks ago at most, they are introducing more on this (with the queryHash being part of the profiler data) but I'm not sure how convenient it would be to use.
I also don't want to build something that outputs different metrics (labels only?) for different MongoDB versions.

[stefano-pogliani]

Hello @dexterHD 👋

Sorry it took a while but I was finally able to sync dep and the Gopkg.lock.
Let me know if, and if so how, you want me to squash commits or anything.

[Zhang21]

Same demand, I also want this feature to get mongodb slow query.

Can add this feature to mongodb_exporter?

[Zhang21]

I found this feature in PR Add monitoring for current and recent slow queries ， it's open.

Can merge this code to master?

[alitoufighi]

Is anybody here to complete this PR? Performance analysis would be a great step in monitoring.

[AlekSi]

Hi. We are currently hard at work on a new, entirely rewritten version of the exporter. Please send PRs to exporter_v2 branch, that will help us integrate it faster.

https://jira.percona.com/browse/PMM-6579

[denisok]

Hello PR owner,

we are cleaning up the mess with the branches. New mongodb_exporter implementation would be default main branch.
Old master branch is moved to the release-0.1x branch.

Please resubmit and resolve any conflicts against the main and we would start to do the better job to get them merged to the main and produce regular releases.

We also moved to the Github Actions from Travis CI, so that would check your code going forward.

Thanks,
Denys from Percona

[denisok]

master branch was renamed to release-0.1x.

If you have a local clone, you can update it by running:

git branch -m master release-0.1x
git fetch origin
git branch -u origin/release-0.1x release-0.1x

[percona-csalguero]

Do we really want this in the exporter?
We have pt-mongodb-query-digest to analyze slow queries.
This PR is outdated, has merge conflicts and it is for the old version. Not sure if we can/want to implement it.
It this is still needed, can we create a Jira ticket to track the discussion with the Percona experts?

[askomorokhov]

Thank you for your PR. Unfortunately, it does not align with how we collect and use slow queries in PMM.

[dfredell]

Thank you for your PR. Unfortunately, it does not align with how we collect and use slow queries in PMM.

@askomorokhov How does PMM collect and use slow queries? Is it pt-mongodb-query-digest?

[dfredell]

Exporting of the system profile collection just got added to the main branch.
#710