feat: extends stamps capacity

[luissanchez0305]

Closes #311

Basically, instead of buying a new postage stamp every time the usage threshold is reached with this feature, it would be possible to extend the capacity (depth) of the postage stamp using the dilute endpoint.

[AuHau]

Few general things we need to address before moving forward.

Firstly, you are treating this feature as another exclusive mode of operation of gateway-proxy's stamp management. I don't think that is correct though. The whole gateway-proxy layout of features is getting quite complicated and once Attila is back we really should think of some better architecture for this. But in meanwhile...

What I see is that there is sort of two modes of the postage management. Either it would make sense to run it with autobuy enabled, which basically allows uploading content to Swarm that has a limited life-span on the network and the extending modes then does not really make sense (at least to me, feel free to come up with use-case proving me wrong). Or, you want to make sure that the uploaded stays in some form on the network, for which there are the extending stamp's features.

So, basically, I can see the possibility that when stamps management is not running in hardcoded or autobuy it can run with both extendsTTL and extendsCapacity modes... I think we should support this use case.

Secondly, we have quite neglected testing in your previous PRs. We should fix that later on, but lets start with this PR first. Vojtech created quite a nice tooling for testing of the StampsManager (eq. you have stamps DB, you can set their values etc.), so please use it in order to test thoroughly the (edge) cases of the extendsCapacity mode. Also, there should be maybe some tests that cover the mentioned interoperability of the extendsCapacity and extendsTtl modes. Lastly, please structure the tests so that each mode has its own describe section.

[Cafe137]

Hmmm, maybe @agazso or @AuHau can confirm it, but IIRC topup is additive instead of specifying the target amount. So the * 2 is unnecessary here.

[Cafe137]

Also as a general comment (maybe concerns @agazso the most), we would need an atomic topup + dilute operation on blockchain level in order to make this feature solid, otherwise one of these operations can fail (http network issue, gas issue, non-graceful shutdown, running out of funds) making stamps grow asymmetrically.

[Cafe137]

Oops, meant these comments for src/stamps.ts, not the test 🙂

[luissanchez0305]

I'll wait for @AuHau @agazso 's insight on this

[luissanchez0305]

@Cafe137 I already test the additive property of topup call, and yes, no need to double up the amount

[AuHau]

Oh, good catch @Cafe137 about the additive side!

And yes, agree about the atomicity problem here, I am afraid though that with the current Bee API this is not possible for us to handle :-( ethersphere/bee#3359 would help us with this.

[Cafe137]

I am running with this env:

BEE_DEBUG_API_URL: 'http://localhost:1635',
POSTAGE_EXTENDS_CAPACITY: 'true',

Have a clean Bee, no stamps.

And it tries to buy a new stamp:

time="2022-10-14T11:43:02.100Z" level="info" msg="buying new stamp"

Which fails due to missing envs

time="2022-10-14T11:43:02.107Z" level="error" msg="failed to refresh on extends postage stamps Value has to be bigger then zero" value="0"
at ./src/stamps.ts:150:34
at ./src/stamps.ts:8:71"

But I think it should not buy stamps at all

[Cafe137]

I want to run only EXTENDSTTL, but EXTENDS_CAPACITY also runs.

I have this env

  POSTAGE_DEPTH: '22',
  POSTAGE_AMOUNT: '200000',
  POSTAGE_TTL_MIN: '60',
  BEE_DEBUG_API_URL: 'http://localhost:1635',
  POSTAGE_EXTENDSTTL: 'true',

Calls made to Bee:

By EXTENDSTTL:

PATCH /stamps/topup/310d42a6301771736bb495fbe07ecba841c41169055291d8adf663b237528e04/200000

By EXTENDS_CAPACITY which I have not enabled:

PATCH  /stamps/topup/310d42a6301771736bb495fbe07ecba841c41169055291d8adf663b237528e04/400000
PATCH  /stamps/dilute/310d42a6301771736bb495fbe07ecba841c41169055291d8adf663b237528e04/23

[Cafe137]

Also another observation when POSTAGE_REFRESH_PERIOD is low

EXTENDSTTL does not run topup on a stamp again and again, instead it waits for the existing operation to complete.

EXTENDS_CAPACITY, however runs topup and dilute in every cycle, even if there is already an operation in progress.

[AuHau]

I am sorry, but the code gets too messy beyond what I can understand and reason about. But the fact that we have cleared out how the Stamps Manager can operate (eq. that autobuy and extends functionality are exclusive), gives me an idea of how to improve the situation. Let's discuss it today on the call later on.

There are several other smaller issues I have identified. One of the major ones is regarding the isNaN check and generally input validation, which we should tackle (as per #393 issue).

Then there is the testing. I am sorry but the tests you have are very non-sufficient, we will have to improve that to ensure that the higher complexity that this project grows is working correctly. Lets discuss it on the call as well so we can get aligned with what I expect here.

[AuHau]

Oh, good catch @Cafe137 about the additive side!

And yes, agree about the atomicity problem here, I am afraid though that with the current Bee API this is not possible for us to handle :-( ethersphere/bee#3359 would help us with this.

[AuHau]

Ad @Cafe137 comment:

But I think it should not buy stamps at all

I don't agree with that. IMHO it should buy one stamp and then keep it extending...