iot edge workbook device details for module resource usage is wrong

[tervoju]

when checking resource usage in edge device (e.g. memory usage) locally I see different memory usage to what is shown in IoT edge workbook device details

Expected Behavior

memory usage figures should be the same

Current Behavior

big difference in memory usage

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

1. install edge and IoTEdgeMetricsCollector to edge and other relevant containers
2. use docker stats in remote edge device
3. check figures what is available in iot hub workbook

Context (Environment)

linux ubuntu 20.04
iotedge 1.4

Output of iotedge check

Click here

<Paste here between the triple backticks>

Device Information

• Host OS [e.g. Ubuntu 18.04, Windows Server IoT 2019]:
• Architecture [e.g. amd64, arm32, arm64]:
• Container OS [e.g. Linux containers, Windows containers]:

Runtime Versions

• aziot-edged 1.4.3:
• Edge Agent [image tag 1.4:
• Edge Hub [image tag 1.4:
• Docker/Moby [run docker version]:
  Client: Docker Engine - Community
  Version: 20.10.21

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

aziot-edged logs

<Paste here between the triple backticks>

edge-agent logs

<Paste here between the triple backticks>

edge-hub logs

<Paste here between the triple backticks>

Additional Information

Please provide any additional information that may be helpful in understanding the issue.

[ancaantochi]

@tervoju
Can you please provide logs for metrics collector module?

[tervoju]

current logs

$ docker logs IoTEdgeMetricsCollector --tail 200
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92
[2023-02-07 06:06:45.632 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070
[2023-02-07 06:06:45.638 INF] Scraping finished, received 375 metrics from endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:06:45.638 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics
[2023-02-07 06:06:46.341 DBG] 200 OK
[2023-02-07 06:06:46.344 INF] Successfully sent 400 metrics to fixed set table
[2023-02-07 06:06:46.344 INF] Successfully completed periodic operation Scrape and Upload Metrics
[2023-02-07 06:11:46.342 INF] Starting periodic operation Scrape and Upload Metrics...
[2023-02-07 06:11:46.342 INF] Scraping endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:11:46.342 INF] Scraping endpoint http://edgeHub:9600/metrics
[2023-02-07 06:11:46.342 INF] Scraping endpoint http://cadvisor:7070
[2023-02-07 06:11:46.344 ERR] Error scraping endpoint http://cadvisor:7070 - System.Net.Http.HttpRequestException: Connection refused (cadvisor:7070)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92 [2023-02-07 06:11:46.344 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070 [2023-02-07 06:11:46.348 INF] Scraping finished, received 375 metrics from endpoint http://edgeAgent:9600/metrics [2023-02-07 06:11:46.350 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics [2023-02-07 06:11:46.919 DBG] 200 OK [2023-02-07 06:11:46.922 INF] Successfully sent 400 metrics to fixed set table [2023-02-07 06:11:46.922 INF] Successfully completed periodic operation Scrape and Upload Metrics [2023-02-07 06:16:46.926 INF] Starting periodic operation Scrape and Upload Metrics... [2023-02-07 06:16:46.926 INF] Scraping endpoint http://edgeAgent:9600/metrics [2023-02-07 06:16:46.927 INF] Scraping endpoint http://edgeHub:9600/metrics [2023-02-07 06:16:46.927 INF] Scraping endpoint http://cadvisor:7070 [2023-02-07 06:16:46.928 ERR] Error scraping endpoint http://cadvisor:7070 - System.Net.Http.HttpRequestException: Connection refused (cadvisor:7070) ---> System.Net.Sockets.SocketException (111): Connection refused at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92
[2023-02-07 06:16:46.928 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070
[2023-02-07 06:16:46.932 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics
[2023-02-07 06:16:46.934 INF] Scraping finished, received 375 metrics from endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:16:47.558 DBG] 200 OK
[2023-02-07 06:16:47.561 INF] Successfully sent 400 metrics to fixed set table
[2023-02-07 06:16:47.561 INF] Successfully completed periodic operation Scrape and Upload Metrics
[2023-02-07 06:21:47.562 INF] Starting periodic operation Scrape and Upload Metrics...
[2023-02-07 06:21:47.562 INF] Scraping endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:21:47.562 INF] Scraping endpoint http://edgeHub:9600/metrics
[2023-02-07 06:21:47.562 INF] Scraping endpoint http://cadvisor:7070
[2023-02-07 06:21:47.564 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics
[2023-02-07 06:21:47.564 ERR] Error scraping endpoint http://cadvisor:7070 - System.Net.Http.HttpRequestException: Connection refused (cadvisor:7070)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92 [2023-02-07 06:21:47.564 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070 [2023-02-07 06:21:47.571 INF] Scraping finished, received 375 metrics from endpoint http://edgeAgent:9600/metrics [2023-02-07 06:21:48.175 DBG] 200 OK [2023-02-07 06:21:48.179 INF] Successfully sent 400 metrics to fixed set table [2023-02-07 06:21:48.179 INF] Successfully completed periodic operation Scrape and Upload Metrics [2023-02-07 06:26:48.182 INF] Starting periodic operation Scrape and Upload Metrics... [2023-02-07 06:26:48.182 INF] Scraping endpoint http://edgeAgent:9600/metrics [2023-02-07 06:26:48.182 INF] Scraping endpoint http://edgeHub:9600/metrics [2023-02-07 06:26:48.182 INF] Scraping endpoint http://cadvisor:7070 [2023-02-07 06:26:48.184 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics [2023-02-07 06:26:48.184 ERR] Error scraping endpoint http://cadvisor:7070 - System.Net.Http.HttpRequestException: Connection refused (cadvisor:7070) ---> System.Net.Sockets.SocketException (111): Connection refused at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92
[2023-02-07 06:26:48.184 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070
[2023-02-07 06:26:48.191 INF] Scraping finished, received 377 metrics from endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:26:48.852 DBG] 200 OK
[2023-02-07 06:26:48.855 INF] Successfully sent 402 metrics to fixed set table
[2023-02-07 06:26:48.855 INF] Successfully completed periodic operation Scrape and Upload Metrics
[2023-02-07 06:31:48.854 INF] Starting periodic operation Scrape and Upload Metrics...
[2023-02-07 06:31:48.854 INF] Scraping endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:31:48.854 INF] Scraping endpoint http://edgeHub:9600/metrics
[2023-02-07 06:31:48.866 INF] Scraping endpoint http://cadvisor:7070
[2023-02-07 06:31:48.881 ERR] Error scraping endpoint http://cadvisor:7070 - System.Net.Http.HttpRequestException: Connection refused (cadvisor:7070)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92 [2023-02-07 06:31:48.881 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070 [2023-02-07 06:31:48.911 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics [2023-02-07 06:31:48.929 INF] Scraping finished, received 377 metrics from endpoint http://edgeAgent:9600/metrics [2023-02-07 06:31:49.658 DBG] 200 OK [2023-02-07 06:31:49.663 INF] Successfully sent 402 metrics to fixed set table [2023-02-07 06:31:49.663 INF] Successfully completed periodic operation Scrape and Upload Metrics [2023-02-07 06:36:49.666 INF] Starting periodic operation Scrape and Upload Metrics... [2023-02-07 06:36:49.666 INF] Scraping endpoint http://edgeAgent:9600/metrics [2023-02-07 06:36:49.666 INF] Scraping endpoint http://edgeHub:9600/metrics [2023-02-07 06:36:49.666 INF] Scraping endpoint http://cadvisor:7070 [2023-02-07 06:36:49.668 ERR] Error scraping endpoint http://cadvisor:7070 - System.Net.Http.HttpRequestException: Connection refused (cadvisor:7070) ---> System.Net.Sockets.SocketException (111): Connection refused at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken) at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92
[2023-02-07 06:36:49.668 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070
[2023-02-07 06:36:49.672 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics
[2023-02-07 06:36:49.674 INF] Scraping finished, received 377 metrics from endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:36:50.276 DBG] 200 OK
[2023-02-07 06:36:50.280 INF] Successfully sent 402 metrics to fixed set table
[2023-02-07 06:36:50.280 INF] Successfully completed periodic operation Scrape and Upload Metrics
[2023-02-07 06:41:50.279 INF] Starting periodic operation Scrape and Upload Metrics...
[2023-02-07 06:41:50.279 INF] Scraping endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:41:50.279 INF] Scraping endpoint http://edgeHub:9600/metrics
[2023-02-07 06:41:50.281 INF] Scraping endpoint http://cadvisor:7070
[2023-02-07 06:41:50.283 ERR] Error scraping endpoint http://cadvisor:7070 - System.Net.Http.HttpRequestException: Connection refused (cadvisor:7070)
---> System.Net.Sockets.SocketException (111): Connection refused
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
at System.Net.Sockets.Socket.g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Azure.Devices.Edge.Azure.Monitor.MetricsScraper.ScrapeEndpoint(String endpoint, CancellationToken cancellationToken) in /mnt/vss/_work/1/s/edge-modules/metrics-collector/src/MetricsScraper.cs:line 92
[2023-02-07 06:41:50.283 INF] Scraping finished, received 0 metrics from endpoint http://cadvisor:7070
[2023-02-07 06:41:50.290 INF] Scraping finished, received 377 metrics from endpoint http://edgeAgent:9600/metrics
[2023-02-07 06:41:50.291 INF] Scraping finished, received 25 metrics from endpoint http://edgeHub:9600/metrics
[2023-02-07 06:41:50.945 DBG] 200 OK
[2023-02-07 06:41:50.948 INF] Successfully sent 402 metrics to fixed set table
[2023-02-07 06:41:50.948 INF] Successfully completed periodic operation Scrape and Upload Metrics

[ancaantochi]

What is the time period you used to check the memory usage in portal? Does the time period in IotHub matches with the time you check on device with docker stats?
The metrics that are displayed in portal are sent from edge using docker stats command, it should have the same value and from collector logs it looks the metrics are sent upstream. It possible at the time you checked there was an error in collector and metrics were not sent. Can you please check collector logs and immediately after check docker stats and portal?

[tervoju]

it is about same time. looking this again and it looks that the difference is with data-acquisition container and the difference seems to be pretty static

[ancaantochi]

@tervoju Can you please run a query on your IotHub so you can see if the values that are sent from edge are correct?
To run the query go to Iothub to Monitoring -> Logs and run query on InsightsMetrics table:

let startTime = now(-2h);
let endTime = now();
InsightsMetrics
| where TimeGenerated between (startTime .. endTime)
| where Name == "edgeAgent_used_memory_bytes"
| extend dimensions=parse_json(Tags)
| extend module = tostring(dimensions.module_name)
| order by TimeGenerated, module

You can also adjust the startTime and endTime form the top dropdown next to Run.

Please let me know if the values match with docker stats or not.

[tervoju]

no, the data is the same as previously:

2/9/2023, 10:28:55.000 AM edgeAgent_used_memory_bytes 1,275,760,640

and in docker stats ~ 4 GB

[ancaantochi]

Thanks! That is a discrepancy between the docker docs on how to calculate the memory usage by subtracting cache value and I think the metric migth not be using the right field. Can you please run one more command to check what docker cli is returning:

first get the container id for data-acquisition module
sudo docker ps

then use docker api to get stats:

sudo curl --unix-socket /run/docker.sock --url http://localhost/containers/<container_id>/stats

I would like to take a look at "memory_stats", mostly at "usage", "cache" and "total_inactive_file" values

[tervoju]

some values here.
acq_metrics.txt

[ancaantochi]

Thanks for the info! it is a bug on how the memory usage is calculated on edge side, I will look into having it fixed.

[tervoju]

ok, great.

[github-actions]

This issue is being marked as stale because it has been open for 30 days with no activity.