Crash on benchmark

[kernelogic]

Not entirely clear on what happened but here is the logs:

2020-07-02T00:33:59.547 INFO filcrypto::proofs::api > generate_data_commitment: start
2020-07-02T00:33:59.547 INFO filecoin_proofs::api::seal > compute_comm_d:start
2020-07-02T00:33:59.547 INFO filecoin_proofs::pieces > verifying 8192 pieces
2020-07-02T00:33:59.549 INFO filecoin_proofs::api::seal > compute_comm_d:finish
2020-07-02T00:33:59.549 INFO filcrypto::proofs::api > generate_data_commitment: finish
2020-07-02T00:33:59.549-0700 INFO lotus-bench lotus-bench/main.go:518 [1] Running replication(1)...
2020-07-02T00:33:59.550 INFO filcrypto::proofs::api > seal_pre_commit_phase1: start
2020-07-02T00:33:59.551 INFO filecoin_proofs::api::seal > seal_pre_commit_phase1:start
2020-07-02T00:36:12.310 INFO filecoin_proofs::api::seal > building merkle tree for the original data
2020-07-02T00:39:03.164 INFO filecoin_proofs::api::seal > verifying pieces
2020-07-02T00:39:03.164 INFO filecoin_proofs::pieces > verifying 1 pieces
2020-07-02T00:39:03.164 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase1
2020-07-02T00:39:03.164 INFO storage_proofs_porep::stacked::vanilla::proof > generate labels
2020-07-02T00:39:03.164 INFO storage_proofs_porep::stacked::vanilla::graph > using parent_cache[2048 / 1073741824]
2020-07-02T00:39:03.164 INFO storage_proofs_porep::stacked::vanilla::cache > parent cache: opening /var/tmp/filecoin-parents/v27-sdr-parent-e1fa5d5b811ddbd118be3412c4a8c329156b8b8acc72632bca459455b5a05a13.cache
2020-07-02T00:39:03.164 INFO storage_proofs_porep::stacked::vanilla::cache > parent cache: opened
2020-07-02T00:39:03.164 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 1
2020-07-02T00:51:32.036 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T00:51:39.631 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T00:52:51.238 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 1 store with id layer-1
2020-07-02T00:52:51.239 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 2
2020-07-02T01:05:55.325 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T01:05:57.138 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T01:07:06.838 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 2 store with id layer-2
2020-07-02T01:07:06.838 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 3
2020-07-02T01:20:04.755 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T01:20:06.574 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T01:21:15.663 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 3 store with id layer-3
2020-07-02T01:21:15.663 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 4
2020-07-02T01:34:12.389 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T01:34:14.214 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T01:35:23.566 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 4 store with id layer-4
2020-07-02T01:35:23.566 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 5
2020-07-02T01:48:21.367 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T01:48:23.176 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T01:49:31.846 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 5 store with id layer-5
2020-07-02T01:49:31.846 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 6
2020-07-02T02:02:37.811 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T02:02:39.617 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T02:03:47.710 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 6 store with id layer-6
2020-07-02T02:03:47.710 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 7
2020-07-02T02:16:42.553 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T02:16:44.360 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T02:17:53.403 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 7 store with id layer-7
2020-07-02T02:17:53.403 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 8
2020-07-02T02:30:53.580 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T02:30:55.384 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T02:32:03.857 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 8 store with id layer-8
2020-07-02T02:32:03.857 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 9
2020-07-02T02:44:58.622 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T02:45:00.423 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T02:46:09.814 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 9 store with id layer-9
2020-07-02T02:46:09.814 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 10
2020-07-02T02:59:04.227 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T02:59:06.047 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T03:00:15.875 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 10 store with id layer-10
2020-07-02T03:00:15.875 INFO storage_proofs_porep::stacked::vanilla::proof > generating layer: 11
2020-07-02T03:13:10.038 INFO storage_proofs_porep::stacked::vanilla::proof > setting exp parents
2020-07-02T03:13:11.838 INFO storage_proofs_porep::stacked::vanilla::proof > storing labels on disk
2020-07-02T03:14:21.921 INFO storage_proofs_porep::stacked::vanilla::proof > generated layer 11 store with id layer-11
2020-07-02T03:14:23.375 INFO filecoin_proofs::api::seal > seal_pre_commit_phase1:finish
2020-07-02T03:14:23.377 INFO filcrypto::proofs::api > seal_pre_commit_phase1: finish
2020-07-02T03:14:23.377-0700 INFO lotus-bench lotus-bench/main.go:529 [1] Running replication(2)...
2020-07-02T03:14:23.378 INFO filcrypto::proofs::api > seal_pre_commit_phase2: start
2020-07-02T03:14:23.378 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:start
2020-07-02T03:14:23.379 INFO filecoin_proofs::api > validate_cache_for_precommit_phase2:finish
2020-07-02T03:14:23.379 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:start
2020-07-02T03:14:23.417 INFO storage_proofs_porep::stacked::vanilla::proof > replicate_phase2
2020-07-02T03:14:23.418 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree c using the CPU
2020-07-02T03:14:23.418 INFO storage_proofs_porep::stacked::vanilla::proof > Building column hashes
2020-07-02T03:47:28.390 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 1/8
2020-07-02T04:27:34.531 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 2/8
2020-07-02T05:12:28.205 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 3/8
2020-07-02T05:52:13.021 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 4/8
2020-07-02T06:38:13.577 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 5/8
2020-07-02T07:21:58.448 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 6/8
2020-07-02T08:03:22.447 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 7/8
2020-07-02T08:47:27.456 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_c 8/8
2020-07-02T08:49:34.136 INFO storage_proofs_porep::stacked::vanilla::proof > tree_c done
2020-07-02T08:49:34.136 INFO storage_proofs_porep::stacked::vanilla::proof > building tree_r_last
2020-07-02T08:49:34.137 INFO storage_proofs_porep::stacked::vanilla::proof > generating tree r last using the CPU
2020-07-02T08:49:47.043 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 1/8
2020-07-02T08:52:49.964 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 2/8
2020-07-02T08:56:01.866 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 3/8
2020-07-02T08:59:05.353 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 4/8
2020-07-02T09:02:06.970 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 5/8
2020-07-02T09:05:16.870 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 6/8
2020-07-02T09:08:23.334 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 7/8
2020-07-02T09:11:31.426 INFO storage_proofs_porep::stacked::vanilla::proof > building base tree_r_last with CPU 8/8
2020-07-02T09:14:13.130 INFO storage_proofs_porep::stacked::vanilla::proof > tree_r_last done
2020-07-02T09:14:13.131 INFO storage_proofs_core::data > dropping data /mnt/ipfs/bench/bench204480528/sealed/s-t01000-1
2020-07-02T09:14:13.634 INFO filecoin_proofs::api::seal > seal_pre_commit_phase2:finish
2020-07-02T09:14:13.634 INFO filcrypto::proofs::api > seal_pre_commit_phase2: finish
2020-07-02T09:14:13.639-0700 INFO lotus-bench lotus-bench/main.go:551 [1] Generating PoRep for sector (1)
2020-07-02T09:14:13.656 INFO filcrypto::proofs::api > seal_commit_phase1: start
2020-07-02T09:14:13.658 INFO filecoin_proofs::api > validate_cache_for_precommit:start
2020-07-02T09:14:13.658 INFO filecoin_proofs::api > validate_cache_for_precommit:finish
2020-07-02T09:14:13.659 INFO filecoin_proofs::api::seal > seal_commit_phase1:start
2020-07-02T09:14:13.659 INFO filecoin_proofs::pieces > verifying 1 pieces
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
thread '' panicked at 'Invalid encoding proof generated at layer 1', <::std::macros::panic macros>:2:4
2020-07-02T09:14:41.755-0700 WARN ffiwrapper ffiwrapper/sealer_cgo.go:491 StandaloneSealCommit error: Rust panic: no unwind information
2020-07-02T09:14:41.756-0700 WARN ffiwrapper ffiwrapper/sealer_cgo.go:492 num:1 tkt:[14 87 81 192 38 229 67 178 232 171 46 176 96 153 218 161 209 229 223 71 119 143 119 135 250 171 69 205 241 47 227 168] seed:[1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 255], pi:[{34359738368 bafk4chzamdx4bfrlkgql2uy6uso2wvljaexpii3suuby6asoeoszcehj3myq}] sealedCID:bafk4ehza5scn4ikbpmyiga2h2er4p5ong2fsr4snxl5urasd43ncij77oqzq, unsealedCID:bafk4chzamdx4bfrlkgql2uy6uso2wvljaexpii3suuby6asoeoszcehj3myq
2020-07-02T09:14:56.229-0700 WARN lotus-bench lotus-bench/main.go:91 failed to run seals:
main.glob..func3
/home/dev/lotus/cmd/lotus-bench/main.go:249

• StandaloneSealCommit:
  github.com/filecoin-project/sector-storage/ffiwrapper.(*Sealer).SealCommit1
  /home/dev/go/pkg/mod/github.com/filecoin-project/[email protected]/ffiwrapper/sealer_cgo.go:494
• Rust panic: no unwind information
  github.com/filecoin-project/filecoin-ffi.SealCommitPhase1
  /home/dev/lotus/extern/filecoin-ffi/proofs.go:359
  github.com/filecoin-project/sector-storage/ffiwrapper.(*Sealer).SealCommit1
  /home/dev/go/pkg/mod/github.com/filecoin-project/[email protected]/ffiwrapper/sealer_cgo.go:478
  main.runSeals.func1.1
  /home/dev/lotus/cmd/lotus-bench/main.go:552
  main.runSeals.func1
  /home/dev/lotus/cmd/lotus-bench/main.go:641
  runtime.goexit
  /usr/local/go/src/runtime/asm_amd64.s:1373