Flaky test failure: Test_single_restart: failed to create SCRawDB: resource temporarily unavailable

[mark-rushakoff]

I haven't made an objective measurement, but it feels like this test fails around 10-20% of the time, with this panic:

panic: failed to create SCRawDB: resource temporarily unavailable

goroutine 2613 [running]:
cosmossdk.io/simapp/v2.NewSimApp[...]({0x53bd[65](https://github.com/gordian-engine/gcosmos/actions/runs/11528399347/job/32095489248#step:5:66)0, 0xc00440ca70}, 0xc0044e4c40)
	/home/runner/work/gcosmos/gcosmos/_cosmosvendor/cosmos-sdk/simapp/v2/app_di.go:180 +0x4e5
cosmossdk.io/simapp/v2/simdv2/cmd.newApp[...]({0x53bd650, 0xc00440ca70}, 0xc0044e4c40)
	/home/runner/work/gcosmos/gcosmos/_cosmosvendor/cosmos-sdk/simapp/v2/simdv2/cmd/commands.go:40 +0x7b
cosmossdk.io/server/v2.createStartCommand[...].func1({0xc001bc5600, 0x0?, 0x8?})
	/home/runner/work/gcosmos/gcosmos/_cosmosvendor/cosmos-sdk/server/v2/commands.go:117 +0xc2
github.com/spf13/cobra.(*Command).execute(0xc00[67](https://github.com/gordian-engine/gcosmos/actions/runs/11528399347/job/32095489248#step:5:68)7c308, {0xc001bc5580, 0x8, 0x8})
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:985 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc00127d508)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1117 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1034
github.com/gordian-engine/gcosmos_test.CmdEnv.RunWithInputC({0xc0012a60c0?, {0xc0012b4060?, 0x10?}}, {0x53b0b80?, 0xc000b8fea0?}, {0x0, 0x0}, {0xc0034fe000, 0x7, 0xa})
	/home/runner/work/gcosmos/gcosmos/helpers_for_test.go:495 +0x3d4
github.com/gordian-engine/gcosmos_test.CmdEnv.RunC(...)
	/home/runner/work/gcosmos/gcosmos/helpers_for_test.go:454
github.com/gordian-engine/gcosmos_test.Chain.Start.func3(0x0)
	/home/runner/work/gcosmos/gcosmos/helpers_for_test.go:319 +0x44d
created by github.com/gordian-engine/gcosmos_test.Chain.Start in goroutine 11
	/home/runner/work/gcosmos/gcosmos/helpers_for_test.go:299 +0x5b5
FAIL	github.com/gordian-engine/gcosmos	5.014s

(This just happened on a PR.)

I believe cosmos/cosmos-sdk#22240 was the change that went from this failure mode happening every time we did a validator restart, to only sometimes. (It was intended to eliminate the failure, to be clear.)

Right now I don't have enough information to judge if this issue is in the gcosmos side or the SDK side.