Skip to content

Commit

Permalink
tweak the in-mem buffer for more stable throughput
Browse files Browse the repository at this point in the history
msf committed Jul 25, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
1 parent edc19bf commit febd30f
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions ingester/mainloop.go
Original file line number Diff line number Diff line change
@@ -46,8 +46,10 @@ func (i *ingester) Run(ctx context.Context, startBlockNumber int64, maxCount int
defer close(blockNumbers)

// We buffer the block channel so that RPC requests can be made concurrently with sending blocks to Dune.
// We limit the buffer size to the same number of concurrent requests, so we exert some backpressure.
blocks := make(chan models.RPCBlock, maxBatchSize)
// We limit the buffer size to the k * maxBatchSize,
// so we exert some backpressure.
// but having enough to feed full batches to DuneAPI
blocks := make(chan models.RPCBlock, 2*maxBatchSize)
defer close(blocks)

// Start MaxBatchSize goroutines to consume blocks concurrently

0 comments on commit febd30f

Please sign in to comment.