chore: make CI green

[ramin]

Building off @vgonkivs's work to skip tests, this attempts to get a baseline "green" ci for us. There are still a couple of tests that VERY intermittently flake in CI, but this way they will stand out WHEN they happen and we can track down vs entire thing being red always.

This does quite a few things

• introduces build tags on swamp tests in each named file to allow us to run parts of the swamp/integration tests independently (ie: go test ./... -tags=blob
• adds an integration tag to allow running all still
• splits the integration tests into it's own workflow file (integration-tests.yml) which is now triggered from go-ci.yml
• splits each swamp/integration tests to run as its own job so we can see which are failing/flakey more explicitly
• utilizes go's -short test flag to Skip a few swamp/integration tests that are consistently failing. Current we are skipping TestFullReconstructFromFulls, TestFullReconstructFromLights,
  TestSyncStartStopLightWithBridge which is less than we were originally skipping in refactor(swamp/tests)skip tests #2802
• plugs in our verbose/debug stuff to integration tests in addition to unit which we had before

Unit tests

• splits some of the unit tests that were "race flakey" into *_no_race_test.go and adds !race tag to some others to get to pass consistently when running unit tests with -race flag
• macos-latest unit tests still fail on race test ONLY in GitHub actions CI 🤷‍♂️

Next Steps

• create issues for each short run / skipped
• create issues for any tests that fail race that are NOT the race issue on upstream cosmos-sdk
• create issue for macos-latest in GitHub race fail
• create issue for macos-latest intermittent fail

I think we let this run for a while, then once we see it be consistently green for a bit and isolate any more flakes that pop up, we can toggle on more branch requirements, then in fixing the above issues, we can be green. Being green is not easy.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2294805) 51.41% compared to head (ef672e2) 51.10%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3114      +/-   ##
==========================================
- Coverage   51.41%   51.10%   -0.31%     
==========================================
  Files         177      177              
  Lines       11157    11157              
==========================================
- Hits         5736     5702      -34     
- Misses       4925     4953      +28     
- Partials      496      502       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[distractedm1nd]

[ramin]

ok, there are still a few flakes from time to time, but this turns green more times than there is an intermittent flakey red, so IMO lets proceed then nab those flakes if/when they appear and keep moving forward

[distractedm1nd]

TIL about -short and that you can use || in build flags like this. Very nice!

[vgonkivs]

Looks good 🚀 . I assume the most common case for flakiness is the race inside cosmos-sdk(that's why a lot of test are build with no race flag)

[vgonkivs]

Let's pay attention to our swamp tests after this PR is merged. In case it's a known flakiness, let's leave a comment in further PRs