Providing trusted-hash produces daser errors

[Bidon15]

Celestia Node version

a7c49d2

OS

linux/amd64 and macos/arm64

Install tools

docker or vanilla install

Others

No response

Steps to reproduce it

1. Initialise a new da-light node
2. Execute this command

celestia light start --p2p.network mocha --core.ip http://consensus-full-mocha-4.celestia-mocha.com --headers.trusted-hash B1930EF8E01DE5684D854ACE8730C16F4498CE3BB2E2EE1C2000CF7FA3F7F875

Expected result

node starts at a non-genesis height without errors

Actual result

daser errors with failed to get header from header store

Relevant log output

2024-02-05T13:27:03.622+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 116924, "to": 116924, "errors": 1, "finished (s)": 0.000490375}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 101613, "finished (s)": 0.000004917}
2024-02-05T13:27:03.621+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 66038, "finished (s)": 0.000006417}
2024-02-05T13:27:03.622+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 86816, "finished (s)": 0.000011208}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 86816, "to": 86816, "errors": 1, "finished (s)": 0.000419916}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 101613, "to": 101613, "errors": 1, "finished (s)": 0.0000535}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 66038, "to": 66038, "errors": 1, "finished (s)": 0.001750667}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 31754, "finished (s)": 0.00000625}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 31754, "to": 31754, "errors": 1, "finished (s)": 0.000036833}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 46092, "finished (s)": 0.00000475}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 47513, "finished (s)": 0.000006}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 47513, "to": 47513, "errors": 1, "finished (s)": 0.000013}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 46092, "to": 46092, "errors": 1, "finished (s)": 0.000037667}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 24177, "finished (s)": 0.000003875}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 24177, "to": 24177, "errors": 1, "finished (s)": 0.00002225}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 107287, "finished (s)": 0.000007833}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 107287, "to": 107287, "errors": 1, "finished (s)": 0.000381417}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 51372, "finished (s)": 0.000005125}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 51372, "to": 51372, "errors": 1, "finished (s)": 0.000033208}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 71673, "to": 71673, "errors": 1, "finished (s)": 0.002123125}
2024-02-05T13:27:03.622+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 16583, "finished (s)": 0.00000475}
2024-02-05T13:27:03.623+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 39254, "finished (s)": 0.000006125}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 16583, "to": 16583, "errors": 1, "finished (s)": 0.000983458}
2024-02-05T13:27:03.623+0100	INFO	das	das/worker.go:88	finished sampling headers	{"type": "retry", "from": 39254, "to": 39254, "errors": 1, "finished (s)": 0.000012208}

Notes

I also see that the daser can fail the whole 100 batches instead of just 1 like in the shared log

[Bidon15]

If we don't provide the trusted hash flag - no errors

[Bidon15]

Same error with latest commit 2ebca8f @Wondertan
I changed the trusted hash from the flag to the config.toml

celestia light start --p2p.network mocha --core.ip http://consensus-full-mocha-4.celestia-mocha.com
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121267, "finished (s)": 0.000005041}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121268, "finished (s)": 0.000000791}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121269, "finished (s)": 0.000000833}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121270, "finished (s)": 0.000000792}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121271, "finished (s)": 0.00000175}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121272, "finished (s)": 0.000016125}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121273, "finished (s)": 0.000003042}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121274, "finished (s)": 0.000001083}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121275, "finished (s)": 0.000000666}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121276, "finished (s)": 0.000000708}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121277, "finished (s)": 0.000001666}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121278, "finished (s)": 0.000000667}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121279, "finished (s)": 0.000001125}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121280, "finished (s)": 0.000004958}
2024-02-27T14:35:29.927+0100	ERROR	das	das/worker.go:170	failed to get header from header store	{"height": 121281, "finished (s)": 0.000000959}

[Wondertan]

A little guide to running a celestia node on a public network without waiting for it to sync:

• Init a fresh node
• Change the two following config fields for the recent header(the example is from mocha):
  • Header.TrustedHash = "0E005E02A1EE6F9350E2B74EF388925F65B8EC1E6D11E3DD92DDD20C82A860F8"
  • DASer.SampleFrom = 1213740
• Start the node

[Bidon15]

A little guide to running a celestia node on a public network without waiting for it to sync:

• Init a fresh node

• Change the two following config fields for the recent header(the example is from mocha):

  • Header.TrustedHash = "0E005E02A1EE6F9350E2B74EF388925F65B8EC1E6D11E3DD92DDD20C82A860F8"
  • DASer.SampleFrom = 1213740

• Start the node

Works! Thanks for the review.
for broader context - should we treat this as a bug or CR (Change Request) @ramin ? As there is no vanilla way through cli flags to provide daser.SampleFrom

The cli flag can help immensely devRel and external teams devX wise