Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add ASAN CI "platform" #2347

Merged
merged 1 commit into from
Mar 28, 2024
Merged

add ASAN CI "platform" #2347

merged 1 commit into from
Mar 28, 2024

Conversation

spoonincode
Copy link
Member

Adds a new "platform" that builds & tests the software with ASAN.

It's possible to enable a single build with UBSAN+ASAN simultaneously, but this further increases both the compile time and run time overhead. So do UBSAN & ASAN separately for now.

And speaking of ASAN build overhead, I was unable to get NP or LR tests to complete in CI without either running out of memory or having some sort of timing problem. I even tried increasing the resources for the run but that still wasn't enough. I think it's likely worthwhile to go ahead with what we've got though as at least we'll still get unit test coverage, and reevaluate whatever the performance problem is later (I was able to complete them successfully locally).

Example run of ASAN failures:
https://github.com/AntelopeIO/leap/actions/runs/8458172328/job/23172568986

@spoonincode spoonincode added the CICD Anything dealing with the CI workflow behavior label Mar 27, 2024
set(CMAKE_CXX_FLAGS "-fsanitize=address -fno-omit-frame-pointer" CACHE STRING "")
EOF

ENV ASAN_OPTIONS=detect_leaks=0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are some WAVM memory "leaks" that have been known about for 5+ years. It'd be nice to resolve them one day to reenable the leak test.

Base automatically changed from chainbase_mlock_asan_war to main March 28, 2024 02:05
Copy link
Member

@heifner heifner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was unable to get NP or LR tests to complete in CI without either running out of memory or having some sort of timing problem. I even tried increasing the resources for the run but that still wasn't enough. I think it's likely worthwhile to go ahead with what we've got though as at least we'll still get unit test coverage, and reevaluate whatever the performance problem is later (I was able to complete them successfully locally).

Is this still true, looks like NP and LR tests are running.

@linh2931
Copy link
Member

I was unable to get NP or LR tests to complete in CI without either running out of memory or having some sort of timing problem. I even tried increasing the resources for the run but that still wasn't enough. I think it's likely worthwhile to go ahead with what we've got though as at least we'll still get unit test coverage, and reevaluate whatever the performance problem is later (I was able to complete them successfully locally).

Is this still true, looks like NP and LR tests are running.

I got a failure in Ubsan CI #2350

@spoonincode
Copy link
Member Author

Is this still true, looks like NP and LR tests are running.

Not for asan platform.

I had tried in a different branch to give NP/LR asan more resources, for example by doing for NP,

- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20', machine: 'enf-x86-midtier'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22', machine: 'enf-x86-midtier'}
- cfg: {name: 'asserton', base: 'asserton', builddir: 'asserton', machine: 'enf-x86-midtier'}
- cfg: {name: 'ubsan', base: 'ubsan', builddir: 'ubsan', machine: 'enf-x86-midtier'}
- cfg: {name: 'asan', base: 'asan', builddir: 'asan', machine: 'enf-x86-hightier'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible', machine: 'enf-x86-midtier'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible', machine: 'enf-x86-midtier'}

but plugin_http_api_test failed (doesn't locally) and it looks like a timing problem,
https://github.com/AntelopeIO/leap/actions/runs/8430283230/job/23086076054

and for LR doing something similar such as

- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20', machine: 'enf-x86-lowtier'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22', machine: 'enf-x86-lowtier'}
- cfg: {name: 'asserton', base: 'asserton', builddir: 'asserton', machine: 'enf-x86-lowtier'}
- cfg: {name: 'ubsan', base: 'ubsan', builddir: 'ubsan', machine: 'enf-x86-lowtier'}
- cfg: {name: 'asan', base: 'asan', builddir: 'asan', machine: 'enf-x86-midtier'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible', machine: 'enf-x86-lowtier'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible', machine: 'enf-x86-lowtier'}

but the box fell over running out of memory
https://github.com/AntelopeIO/leap/actions/runs/8430283230/job/23086075824

maybe giving them even more resources would work.

@spoonincode spoonincode merged commit e29e3ea into main Mar 28, 2024
36 checks passed
@spoonincode spoonincode deleted the asan_ci branch March 28, 2024 16:20
@ericpassmore
Copy link
Contributor

Note:start
group: STABILITY
category: CHORE
summary: Adds test pass with AddressSanitizer.
Note:end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD Anything dealing with the CI workflow behavior
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants