Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating the Readme file with perf optimizations and monitoring commands #240

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

Updating the Readme file with perf optimizations and monitoring commands #240

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c885cd9
Update README.md
astrostakers Feb 19, 2024
ccd6da9
Merge pull request #1 from astrostakers/astrostakers-patch-1
astrostakers Feb 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,34 @@ fetching and saving them to local storage can take time (depends on EL RPC host)
For avoiding `heap out of memory` error, you can pass `NODE_OPTIONS` env var with `--max-old-space-size=8192` value
and when the application completes its first cycle, you can restart your instance without this env variable.

## Performance optimizations and monitoring the monitor

To tail the logs of your instantiation you can use `docker compose logs -f`. Watch out for connection errors and other issues especially regarding `ethereum-validators-monitoring` process. You can choose to ignore alerting related errors if you haven't configured this feature yet.
To see a real-time, dynamic view of these containers performance use `docker stats`.
After the initial launch you can improve the performance, by using some or all of the following options:
- consider running a local CL+EL.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a few words about:

  • Increasing CPU on the host on which the application runs
  • If running local nodes is not an option for you - reduce possible obstacles between remote nodes and application (firewalls, etc.) - anything that can slow down the connection

- remove `--max-old-space-size=8192` from `start:prod` section of `ethereum-validators-monitoring/package.json` file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- remove `--max-old-space-size=8192` from `start:prod` section of `ethereum-validators-monitoring/package.json` file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't bring new resources and even take them away. Removing this option can lead to a heap overflow in JS

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I ran the app locally I faced some quirks with this parameter. If it was presented, the app didn't consume the whole memory available for the container for some reason. When I removed this param, the issue was gone.

You are correct that without this key the app can face heap overflow and we faced this issue in other environments, so, it is mandatory. I think we should remove the max-old-space-size key from the start:prod script and add the ability to pass this value to the app via the env variable. It will allow users to do more fine-grained configuration and also can resolve conflicts with limits in containers when the app is run via the docker-compose. But it should be done in a separate PR. As far as your note regarding the current PR, I agree, it's better not to include this suggestion in Readme as a performance optimization.

- edit `./docker-compose.yml` file and assign more cpus and memory to `app` and `clickhouse` like so:
```
clickhouse:
<< text omitted >>
deploy:
resources:
limits:
cpus: '8'
memory: 16g
app:
<< text omitted >>
deploy:
resources:
limits:
cpus: '8'
memory: 16g
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
memory: 16g

Even at peak loads you don't need more than 3g, so I would suggest keep the current value (as in the current docker-compose) - 8g

```

Do not forget to stop and re-launch docker-compose after performing a modification.


## Run via docker-compose

1. Use `.env.example` file content to create your own `.env` file
Expand Down