infra: bump fuzz introspector #10481

DavidKorczynski · 2023-06-06T23:08:43Z

Contains bug fixes for failed Python and Java projects

Contains bug fixes for failed Python and Java projects Signed-off-by: David Korczynski <[email protected]>

oliverchang · 2023-06-08T05:29:15Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

Signed-off-by: David Korczynski <[email protected]>

oliverchang · 2023-06-13T04:52:19Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

jonathanmetzman · 2023-06-13T16:17:08Z

The second bump broke almost everywhere because of a race condition where introspector needs coverage to be run earlier that day

jonathanmetzman · 2023-06-13T16:17:14Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

jonathanmetzman · 2023-06-14T00:35:52Z

Lots of build failures.

DavidKorczynski · 2023-06-14T13:13:32Z

@jonathanmetzman is it possible to see the projects that fail?

jonathanmetzman · 2023-06-14T13:24:19Z

zxing
zip4j
xz-java
xmlpull
xerces
unit
stringtemplate4
spring-retry
snappy-java
snakeyaml
slf4j-api
python-email-validator
protobuf-python
numexpr
lark-parser
kryo
jupyter-nbconvert
junrar
jul-to-slf4j
jsoup
json-smart-v2
json-sanitizer
json-java
jsign
jline3
jettison
jdom
javassist
jakarta-mail-api
gson
greenmail
cron-utils
cbor-java
apache-felix-dev
apache-commons-logging
apache-commons-lang
apache-commons-io
apache-commons-imaging
apache-commons-geometry
apache-commons-csv
apache-commons-compress
apache-commons-codec
apache-commons-cli
antlr4-java
antlr3-java

Signed-off-by: David Korczynski <[email protected]>

DavidKorczynski · 2023-06-14T22:34:18Z

Contains bug fixes for failed Python and Java projects

Thanks! We should have addressed those now -- in particular the ones that succeeded already.

In addition to this we also added return types for functions in the summary.json which can be digested by the web app APIs.

oliverchang · 2023-06-15T00:18:38Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

DavidKorczynski · 2023-06-15T14:47:44Z

Did the projects succeed now? If it's just timeouts that causes the red cross then this has happened before: #10007 (comment)

jonathanmetzman · 2023-06-15T15:12:51Z

This failed again due to the race condition I mentioned before.

jonathanmetzman · 2023-06-15T15:13:11Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

oliverchang · 2023-06-19T07:39:35Z

Latest trial build timed out on:

Step #1: INFO:root:Polling: defaultdict(<class 'list'>, {'hsqldb': ['f8811ae3-2c86-4dbb-ad28-a446e0c4b482'], 'openssl': ['c1cd969f-fce2-4b40-9fb6-31f6b439df6d'], 'spring-retry': ['2a766711-a0d9-4cb1-94f0-c2442be69529'], 'toolbelt': ['5b25c962-5486-4649-825e-68b031cf6075'], 'tor': ['377d981e-29b5-4848-accc-95f9493a636e']})

Which seems small enough.

@jonathanmetzman @hogo6002 I think we need a smarter way to handle these timeouts as part of the new trial build output improvements. e.g. if a build is known to be fairly slow we can skip marking those as blockign failures.

oliverchang · 2023-06-19T07:50:32Z

Taking a closer look, some of these do look like regressions in that they didn't time out before.

e.g. hsqldb, toolbelt all used to complete within minutes but timed out/exceptioned out after 20hrs.

hsqldb error:

Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 172, in deepcopy
Step #22 - "compile-libfuzzer-introspector-x86_64":     y = _reconstruct(x, memo, *rv)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 270, in _reconstruct
Step #22 - "compile-libfuzzer-introspector-x86_64":     state = deepcopy(state, memo)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy
Step #22 - "compile-libfuzzer-introspector-x86_64":     y = copier(x, memo)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Step #22 - "compile-libfuzzer-introspector-x86_64":     y[deepcopy(key, memo)] = deepcopy(value, memo)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy
Step #22 - "compile-libfuzzer-introspector-x86_64":     y = copier(x, memo)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 205, in _deepcopy_list
Step #22 - "compile-libfuzzer-introspector-x86_64":     append(deepcopy(a, memo))
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 172, in deepcopy
Step #22 - "compile-libfuzzer-introspector-x86_64":     y = _reconstruct(x, memo, *rv)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 270, in _reconstruct
Step #22 - "compile-libfuzzer-introspector-x86_64":     state = deepcopy(state, memo)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 146, in deepcopy
Step #22 - "compile-libfuzzer-introspector-x86_64":     y = copier(x, memo)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Step #22 - "compile-libfuzzer-introspector-x86_64":     y[deepcopy(key, memo)] = deepcopy(value, memo)
Step #22 - "compile-libfuzzer-introspector-x86_64":   File "/usr/local/lib/python3.8/copy.py", line 137, in deepcopy
Step #22 - "compile-libfuzzer-introspector-x86_64":     d = id(x)
Step #22 - "compile-libfuzzer-introspector-x86_64": RecursionError: maximum recursion depth exceeded while calling a Python object

oliverchang · 2023-06-19T07:52:20Z

(sorry for the slow turnaround on these btw! @hogo6002 is working on some UX fixes to make these failures surfaced to the public).

oliverchang · 2023-06-26T00:00:52Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

oliverchang · 2023-06-26T04:45:20Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

jonathanmetzman · 2023-06-26T14:06:37Z

/gcbrun trial_build.py all --sanitizer introspector --fuzzing-engine libfuzzer

oliverchang · 2023-06-27T02:57:34Z

@DavidKorczynski can you please take a look at the failure / timeout logs? The GitHub logs should be much improved now and provide public links to the failed project builds.

oliverchang · 2023-06-29T03:13:30Z

@DavidKorczynski friendly ping!

DavidKorczynski · 2023-06-29T20:30:46Z

Apologies for the delay! The new info looks nice, thank you! But, just noticed at the moment all java projects are currently failing Fuzz Introspector following #10538

Step #6 - "compile-libfuzzer-introspector-x86_64": ---------------------------------------------------------------
Step #6 - "compile-libfuzzer-introspector-x86_64": chmod: cannot access '/workspace/out/libfuzzer-introspector-x86_64/jazzer_driver_with_sanitizer': No such file or directory
Step #6 - "compile-libfuzzer-introspector-x86_64": ********************************************************************************

There is a missing "introspector" conditional in

oss-fuzz/infra/base-images/base-builder/compile

Lines 152 to 171 in 6922c8c

    
           if [ "$FUZZING_LANGUAGE" = "jvm" ]; then 
        
             cp $(which jazzer_agent_deploy.jar) $(which jazzer_driver) $OUT/ 
        
             jazzer_driver_with_sanitizer=$OUT/jazzer_driver_with_sanitizer 
        
             if [ "$SANITIZER" = "address" ]; then 
        
               cat > $jazzer_driver_with_sanitizer << 'EOF' 
        
           #!/bin/bash 
        
           this_dir=$(dirname "$0") 
        
           "$this_dir/jazzer_driver" --asan "$@" 
        
           EOF 
        
             elif [ "$SANITIZER" = "undefined" ]; then 
        
               cat > $jazzer_driver_with_sanitizer << 'EOF' 
        
           #!/bin/bash 
        
           this_dir=$(dirname "$0") 
        
           "$this_dir/jazzer_driver" --ubsan "$@" 
        
           EOF 
        
             elif [ "$SANITIZER" = "coverage" ]; then 
        
               # Coverage builds require no instrumentation. 
        
               cp $(which jazzer_driver) $jazzer_driver_with_sanitizer 
        
             fi 
        
             chmod +x $jazzer_driver_with_sanitizer

This causes line 171 to fail.

You can search for "jvm" in the table here https://introspector.oss-fuzz.com/indexing-overview which will show the introspector build status is red for all.

Ref: #10481 (comment)

DavidKorczynski · 2023-06-29T21:30:18Z

Went over the logs, and the tldr is:

I would mark apache-commons-compress, hsqldb, spring-retry as true bugs, in that they run into stack-overflow exceptions due to some recursion depth.

The remainder projects, I don't think are true failures, in short due to:

guava: timeout, but not sure if this is something on this PR. I would expect it to have long processing before as well?
wget2: timeout, due to unrelated network stuff.
vlc: timeout, but unrelated network stuff.
openssl: looks to me it actually succeeds
tor: looks to me it actually succeeds
uwebsockets: fails, but already fails.
numexpr: fails, but already fails
protobuf-java: timeout, but I don't think it's necessarily this PR. I would expect it to have long processing before as well?

I'd be okay with merging this PR and following up with ways to handle the stack-overflow in a follow-up PR.

More in-depth analysis, as I'm slightly confused about some of the specifics in the trial output, so will write a bit
verbose in case I get something wrong. The following parts at the end of the log show the projects that fail:

...
Step #1: Remaining builds: 7/572, defaultdict(<class 'list'>, 
{
	'guava': ['bde78259-41cc-471a-88b5-ac0a70771e8e'], 
	'hsqldb': ['dc91a0cb-914c-4fa2-81c4-38323b70a711'], 
	'numexpr': ['e00f959d-1ed1-497d-b487-d49aeab67e3c'], 
	'openssl': ['aaaa977c-2755-4b7d-8f1e-c7016badee79'], 
	'protobuf-java': ['b41ff2df-f6b0-4482-a018-d730e49769bd'], 
	'spring-retry': ['e9a24aaf-40e7-4c86-b2a4-28c4c7444874'], 
	'tor': ['ee27dba0-b904-4b9f-a087-afd2fb4d41a8']}).
...
Step #1: Failed builds: 5/572, {
	'augeas': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e90c826c-3c8e-4b23-a435-c5f3ccac8b37.txt', 
	'uwebsockets': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-52843379-7211-482e-9068-5196966ca6d5.txt', 
	'vlc': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f3d17eb3-4fa1-4594-8c11-655667e485de.txt', 
	'wget2': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1652e573-3494-477a-b097-23c40eeecd87.txt', 
	'apache-commons-compress': 'https://oss-fuzz-gcb-logs.storage.googleapis.com/log-94fd456e-2d15-49de-907d-0e08a0352ce8.txt'
}

So, if I understand correctly these are all the ones that should be inspected? If so:

guava failed due to a timeout (https://oss-fuzz-gcb-logs.storage.googleapis.com/log-bde78259-41cc-471a-88b5-ac0a70771e8e.txt):

...
Step #22 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_introspector.analysis:[+] Accummulating profiles
Step #22 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_introspector.datatypes.fuzzer_profile:Loading coverage of type jvm
Step #22 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_introspector.code_coverage:FOUND XML COVERAGE FILES: ['/src/inspector/jacoco.xml']
TIMEOUT
ERROR: context deadline exceeded

I assume this processing time would have been there before as well?

wget2:

Step #20: Step 10/14 : RUN git clone --depth=1 https://gitlab.com/gnutls/gnutls.git
Step #20:  ---> Running in 4bc33024b257
Step #20: �[91mCloning into 'gnutls'...
Step #20: �[0m�[91mfatal: unable to access 'https://gitlab.com/gnutls/gnutls.git/': gnutls_handshake() failed: Handshake failed
Step #20: The command '/bin/sh -c git clone --depth=1 https://gitlab.com/gnutls/gnutls.git' returned a non-zero code: 128
Finished Step #20
ERROR

Which I assume is unrelated?

vlc:

Step #12: e88268cbf419: Pull complete
Step #16: e88268cbf419: Pull complete
Step #10: Error response from daemon: Head "https://gcr.io/v2/oss-fuzz-base/base-builder-jvm-testing-bump-to-latest-fi-100/manifests/latest": Get "https://gcr.io/v2/token?account=oauth2accesstoken&scope=repository%3Aoss-fuzz-base%2Fbase-builder-jvm-testing-bump-to-latest-fi-100%3Apull&service=gcr.io": dial tcp 142.250.103.82:443: i/o timeout
Step #14: c2f0992125df: Pull complete
Step #6: c2f0992125df: Pull complete
Step #8: c2f0992125df: Pull complete
Step #4: c2f0992125df: Pull complete

Which I assume is unrelated?

openssl: The log for openssl shows that it succeeded.

tor The log shows it succeeds?

uwebsockets: it shows:

Step #22 - "compile-libfuzzer-introspector-x86_64": [Log level 1] : 14:27:36 : Wrapping all functions
Step #22 - "compile-libfuzzer-introspector-x86_64": [Log level 1] : 14:27:36 : Ended wrapping all functions
Step #22 - "compile-libfuzzer-introspector-x86_64": [Log level 1] : 14:27:36 : Finished introspector module
Step #22 - "compile-libfuzzer-introspector-x86_64": clang-15: �[0;1;31merror: �[0m�[1munable to execute command: Segmentation fault (core dumped)�[0m
Step #22 - "compile-libfuzzer-introspector-x86_64": clang-15: �[0;1;31merror: �[0m�[1mlinker command failed due to signal (use -v to see invocation)�[0m
Step #22 - "compile-libfuzzer-introspector-x86_64": make: *** [Makefile:29: oss-fuzz] Error 254
Step #22 - "compile-libfuzzer-introspector-x86_64": ********************************************************************************

However, uwebsockets already fails. This is not introduced by this PR.

numexpr is already failing, but this time it's timing out. In this context it's like the case of some of the improvements caused the project to not fail as it originally did, but now it ran into a timeout because the processing get's further but there's a lot to process.

protobuf-java is failing due to a timeout, but it looks like the processing of this project already took a long time.

oliverchang · 2023-06-30T00:18:53Z

Thanks for the analysis and feedback! CC @jonathanmetzman @hogo6002 for the feedback on the trial build logs. At the very least perhaps we need to clarify that the timeout for the trial build is shorter than the timeout for the actual underlying builds, which is why some builds end up succeeding even if the trial build times out.

Ref: #10481 (comment) Java projects are currently failing Fuzz Introspector following #10538 ```bash Step #6 - "compile-libfuzzer-introspector-x86_64": --------------------------------------------------------------- Step #6 - "compile-libfuzzer-introspector-x86_64": chmod: cannot access '/workspace/out/libfuzzer-introspector-x86_64/jazzer_driver_with_sanitizer': No such file or directory Step #6 - "compile-libfuzzer-introspector-x86_64": ******************************************************************************** ``` There is a missing "introspector" conditional in https://github.com/google/oss-fuzz/blob/6922c8c1ac27c9a459ce41201991e296a6bace54/infra/base-images/base-builder/compile#L152-L171 This causes line 171 to fail. It may be convenient to search for "jvm" in the table here https://introspector.oss-fuzz.com/indexing-overview which will show the introspector build status is red for all.

infra: bump fuzz introspector

1d97ece

Contains bug fixes for failed Python and Java projects Signed-off-by: David Korczynski <[email protected]>

bump to newer version

1d8028a

Signed-off-by: David Korczynski <[email protected]>

arthurscchan mentioned this pull request Jun 14, 2023

Java Frontend: Fix typo ossf/fuzz-introspector#1069

Merged

bump again

8d9e1d8

Signed-off-by: David Korczynski <[email protected]>

Merge branch 'master' into bump-to-latest-fi-100

34a481a

DavidKorczynski added a commit that referenced this pull request Jun 29, 2023

compile: fix jvm introspector builds

b65c824

Ref: #10481 (comment)

DavidKorczynski mentioned this pull request Jun 29, 2023

infra: compile: fix jvm introspector builds #10623

Merged

oliverchang merged commit 2b2008a into master Jun 30, 2023

oliverchang deleted the bump-to-latest-fi-100 branch June 30, 2023 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

infra: bump fuzz introspector #10481

infra: bump fuzz introspector #10481

DavidKorczynski commented Jun 6, 2023

oliverchang commented Jun 8, 2023

oliverchang commented Jun 13, 2023

jonathanmetzman commented Jun 13, 2023

jonathanmetzman commented Jun 13, 2023

jonathanmetzman commented Jun 14, 2023

DavidKorczynski commented Jun 14, 2023

jonathanmetzman commented Jun 14, 2023

DavidKorczynski commented Jun 14, 2023

oliverchang commented Jun 15, 2023

DavidKorczynski commented Jun 15, 2023

jonathanmetzman commented Jun 15, 2023

jonathanmetzman commented Jun 15, 2023

oliverchang commented Jun 19, 2023

oliverchang commented Jun 19, 2023

oliverchang commented Jun 19, 2023

oliverchang commented Jun 26, 2023

oliverchang commented Jun 26, 2023

jonathanmetzman commented Jun 26, 2023

oliverchang commented Jun 27, 2023

oliverchang commented Jun 29, 2023

DavidKorczynski commented Jun 29, 2023 •

edited

Loading

DavidKorczynski commented Jun 29, 2023

oliverchang commented Jun 30, 2023

infra: bump fuzz introspector #10481

infra: bump fuzz introspector #10481

Conversation

DavidKorczynski commented Jun 6, 2023

oliverchang commented Jun 8, 2023

oliverchang commented Jun 13, 2023

jonathanmetzman commented Jun 13, 2023

jonathanmetzman commented Jun 13, 2023

jonathanmetzman commented Jun 14, 2023

DavidKorczynski commented Jun 14, 2023

jonathanmetzman commented Jun 14, 2023

DavidKorczynski commented Jun 14, 2023

oliverchang commented Jun 15, 2023

DavidKorczynski commented Jun 15, 2023

jonathanmetzman commented Jun 15, 2023

jonathanmetzman commented Jun 15, 2023

oliverchang commented Jun 19, 2023

oliverchang commented Jun 19, 2023

oliverchang commented Jun 19, 2023

oliverchang commented Jun 26, 2023

oliverchang commented Jun 26, 2023

jonathanmetzman commented Jun 26, 2023

oliverchang commented Jun 27, 2023

oliverchang commented Jun 29, 2023

DavidKorczynski commented Jun 29, 2023 • edited Loading

DavidKorczynski commented Jun 29, 2023

oliverchang commented Jun 30, 2023

DavidKorczynski commented Jun 29, 2023 •

edited

Loading