Incorrect Timestamps in Gno Chains #1950
Comments
|
I've just setup ntpd on test3's node: root@test3:~# date
Sun 21 Apr 2024 07:52:02 PM UTC
root@test3:~# ntpd
root@test3:~# date
Sun 21 Apr 2024 07:53:28 PM UTC
root@test3:~#Can you check again? |
|
@moul Seems like the gap's been reduced to 10~20 seconds. https://www.loom.com/share/507f77eb80de48738545bc3ce3b75ea0?sid=0b3c282d-cc8b-4d2d-a78f-6c752330cf9d Was Also, would it make sense to include a section in the WIP Multinode Docs that recommends the validator nodes to install |
|
The real issue is that I'm not sure what happens if a validator provides an incorrect timestamp. I think we should implement some checks in the first round of consensus to verify that the suggested timestamp is correct. It should at least be "after the previous block" and not too far in the past or future. The question is wether we should care about fixing it before the launch or not. The essential aspect of this issue is to assess the potential impact of postponing it after the launch. |
Is this even possible?? AFAIK, gno's returns timestamp from block(object that gets finalized), not running node it-self. @zivkovicmilos @thehowl Could you please check if it is manipulatable?? |
|
We investigated this issue with @zivkovicmilos for a while and didn’t find anything conclusive. The average delta is currently about 2 seconds on test5, which is not alarming. This issue being resolved for now, I'm closing it. |
github-project-automation
bot
moved this from In Progress
to Done
in 🧙♂️gno.land core team
|
Not sure if this is related but tendermint1 chains had a similar problem, the fix implemented in cometbft is described here https://docs.cometbft.com/v1.0/explanation/core/proposer-based-timestamps |
|
We read several documents and issues on the topic, then we discussed the idea of replacing the calculation of the median time with a mix of:
However, given the complexity of the subject (even just to test and make a decision) and the ongoing issues for the mainnet launch, we decided to keep the implementation as it is. |
This PR adds a tool to the contribs dedicated to adding subcommands for health checks. The first available subcommand simply detects timestamp drift on a given chain. This test was useful in the context of this issue: #1950. However, it could later run on a dedicated server or on a GitHub Actions cron job to alert us in case significant drift occurs again. Results on test5 : ``` > gnohealth timestamp -ws -remote 'wss://rpc.test5.gno.land:443/websocket' -verbose block 411344 drifted of 3.094940942s (max 10s): OK block 411345 drifted of 2.368750176s (max 10s): OK block 411346 drifted of 2.310184977s (max 10s): OK block 411347 drifted of 2.158713327s (max 10s): OK block 411348 drifted of 2.203484957s (max 10s): OK block 411349 drifted of 2.156479203s (max 10s): OK block 411350 drifted of 2.155613458s (max 10s): OK block 411351 drifted of 2.296832155s (max 10s): OK block 411352 drifted of 2.132230389s (max 10s): OK block 411353 drifted of 2.181071735s (max 10s): OK block 411354 drifted of 2.575055701s (max 10s): OK block 411355 drifted of 2.034728695s (max 10s): OK block 411356 drifted of 2.285932658s (max 10s): OK block 411357 drifted of 2.330991247s (max 10s): OK block 411358 drifted of 2.365136593s (max 10s): OK block 411359 drifted of 2.035198868s (max 10s): OK block 411360 drifted of 2.128274141s (max 10s): OK block 411361 drifted of 2.48608003s (max 10s): OK block 411362 drifted of 2.072144703s (max 10s): OK block 411363 drifted of 2.297280076s (max 10s): OK block 411364 drifted of 2.224310386s (max 10s): OK no timestamp drifted beyond the maximum delta (average 2.280639734s) ``` Results on test4 : ``` > gnohealth timestamp -ws -remote 'wss://rpc.test4.gno.land:443/websocket' -verbose block 3618022 drifted of 3.765561468s (max 10s): OK block 3618023 drifted of 3.697091353s (max 10s): OK block 3618024 drifted of 3.576602477s (max 10s): OK block 3618025 drifted of 3.542585771s (max 10s): OK block 3618026 drifted of 3.72231133s (max 10s): OK block 3618027 drifted of 3.751154575s (max 10s): OK block 3618028 drifted of 8.312827308s (max 10s): OK block 3618029 drifted of 3.712806121s (max 10s): OK block 3618030 drifted of 3.65324572s (max 10s): OK no timestamp drifted beyond the maximum delta (average 4.192687347s) ``` <details><summary>Contributors' checklist...</summary> - [x] Added new tests, or not needed, or not feasible - [x] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [x] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details>
This PR adds a tool to the contribs dedicated to adding subcommands for health checks. The first available subcommand simply detects timestamp drift on a given chain. This test was useful in the context of this issue: gnolang#1950. However, it could later run on a dedicated server or on a GitHub Actions cron job to alert us in case significant drift occurs again. Results on test5 : ``` > gnohealth timestamp -ws -remote 'wss://rpc.test5.gno.land:443/websocket' -verbose block 411344 drifted of 3.094940942s (max 10s): OK block 411345 drifted of 2.368750176s (max 10s): OK block 411346 drifted of 2.310184977s (max 10s): OK block 411347 drifted of 2.158713327s (max 10s): OK block 411348 drifted of 2.203484957s (max 10s): OK block 411349 drifted of 2.156479203s (max 10s): OK block 411350 drifted of 2.155613458s (max 10s): OK block 411351 drifted of 2.296832155s (max 10s): OK block 411352 drifted of 2.132230389s (max 10s): OK block 411353 drifted of 2.181071735s (max 10s): OK block 411354 drifted of 2.575055701s (max 10s): OK block 411355 drifted of 2.034728695s (max 10s): OK block 411356 drifted of 2.285932658s (max 10s): OK block 411357 drifted of 2.330991247s (max 10s): OK block 411358 drifted of 2.365136593s (max 10s): OK block 411359 drifted of 2.035198868s (max 10s): OK block 411360 drifted of 2.128274141s (max 10s): OK block 411361 drifted of 2.48608003s (max 10s): OK block 411362 drifted of 2.072144703s (max 10s): OK block 411363 drifted of 2.297280076s (max 10s): OK block 411364 drifted of 2.224310386s (max 10s): OK no timestamp drifted beyond the maximum delta (average 2.280639734s) ``` Results on test4 : ``` > gnohealth timestamp -ws -remote 'wss://rpc.test4.gno.land:443/websocket' -verbose block 3618022 drifted of 3.765561468s (max 10s): OK block 3618023 drifted of 3.697091353s (max 10s): OK block 3618024 drifted of 3.576602477s (max 10s): OK block 3618025 drifted of 3.542585771s (max 10s): OK block 3618026 drifted of 3.72231133s (max 10s): OK block 3618027 drifted of 3.751154575s (max 10s): OK block 3618028 drifted of 8.312827308s (max 10s): OK block 3618029 drifted of 3.712806121s (max 10s): OK block 3618030 drifted of 3.65324572s (max 10s): OK no timestamp drifted beyond the maximum delta (average 4.192687347s) ``` <details><summary>Contributors' checklist...</summary> - [x] Added new tests, or not needed, or not feasible - [x] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [x] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details>
This PR adds a tool to the contribs dedicated to adding subcommands for health checks. The first available subcommand simply detects timestamp drift on a given chain. This test was useful in the context of this issue: #1950. However, it could later run on a dedicated server or on a GitHub Actions cron job to alert us in case significant drift occurs again. Results on test5 : ``` > gnohealth timestamp -ws -remote 'wss://rpc.test5.gno.land:443/websocket' -verbose block 411344 drifted of 3.094940942s (max 10s): OK block 411345 drifted of 2.368750176s (max 10s): OK block 411346 drifted of 2.310184977s (max 10s): OK block 411347 drifted of 2.158713327s (max 10s): OK block 411348 drifted of 2.203484957s (max 10s): OK block 411349 drifted of 2.156479203s (max 10s): OK block 411350 drifted of 2.155613458s (max 10s): OK block 411351 drifted of 2.296832155s (max 10s): OK block 411352 drifted of 2.132230389s (max 10s): OK block 411353 drifted of 2.181071735s (max 10s): OK block 411354 drifted of 2.575055701s (max 10s): OK block 411355 drifted of 2.034728695s (max 10s): OK block 411356 drifted of 2.285932658s (max 10s): OK block 411357 drifted of 2.330991247s (max 10s): OK block 411358 drifted of 2.365136593s (max 10s): OK block 411359 drifted of 2.035198868s (max 10s): OK block 411360 drifted of 2.128274141s (max 10s): OK block 411361 drifted of 2.48608003s (max 10s): OK block 411362 drifted of 2.072144703s (max 10s): OK block 411363 drifted of 2.297280076s (max 10s): OK block 411364 drifted of 2.224310386s (max 10s): OK no timestamp drifted beyond the maximum delta (average 2.280639734s) ``` Results on test4 : ``` > gnohealth timestamp -ws -remote 'wss://rpc.test4.gno.land:443/websocket' -verbose block 3618022 drifted of 3.765561468s (max 10s): OK block 3618023 drifted of 3.697091353s (max 10s): OK block 3618024 drifted of 3.576602477s (max 10s): OK block 3618025 drifted of 3.542585771s (max 10s): OK block 3618026 drifted of 3.72231133s (max 10s): OK block 3618027 drifted of 3.751154575s (max 10s): OK block 3618028 drifted of 8.312827308s (max 10s): OK block 3618029 drifted of 3.712806121s (max 10s): OK block 3618030 drifted of 3.65324572s (max 10s): OK no timestamp drifted beyond the maximum delta (average 4.192687347s) ``` <details><summary>Contributors' checklist...</summary> - [x] Added new tests, or not needed, or not feasible - [x] Provided an example (e.g. screenshot) to aid review or the PR is self-explanatory - [x] Updated the official documentation or not needed - [x] No breaking changes were made, or a `BREAKING CHANGE: xxx` message was included in the description - [x] Added references to related issues and PRs - [ ] Provided any useful hints for running manual tests </details>
The timestamps in blocks are 2~3 minutes behind the actual time.
This bug affects apps that rely on timestamps to display the transaction/block time.
(e.g. Once you send a transaction and immediately check the history in the app, it will display "2 mins ago" instead of "n seconds ago")
In the video below, I send a transaction on
Test3in Adena to stimulate a new block and compare its timestamp with the actual time.09:00:30height 353208):08:58:45https://www.loom.com/share/7b1034678a794bedafa72f2d54b93229?sid=3e223cec-b021-4c66-8ac9-6ec584fd693e
Not sure what's causing the discrepancy here, but it's also happening to a custom Gno Chain that we've spun up to develop GnoSwap.
cc @zivkovicmilos @r3v4s
The text was updated successfully, but these errors were encountered: