Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

caffe-classification-opencl mispredicts and segfaults on 32-bit Android #76

Open
psyhtest opened this issue Mar 6, 2017 · 11 comments
Open

caffe-classification-opencl mispredicts and segfaults on 32-bit Android #76

psyhtest opened this issue Mar 6, 2017 · 11 comments

Comments

@psyhtest
Copy link
Member

psyhtest commented Mar 6, 2017 •
edited
Loading

When running on the T-Firefly board (Android 5.1.1, 32-bit, Mali-T760 w/ driver v6.0):

      I0306 23:35:31.354465  6560 caffe.cpp:678] Start with command: device_query
      I0306 23:35:31.356835  6560 common.cpp:379] Total devices: 1
      I0306 23:35:31.356926  6560 common.cpp:380] CUDA devices: 0
      I0306 23:35:31.356967  6560 common.cpp:381] OpenCL devices: 1
      I0306 23:35:31.357000  6560 common.cpp:405] Device id:                     0
      I0306 23:35:31.357033  6560 common.cpp:407] Device backend:                OpenCL
      I0306 23:35:31.357064  6560 common.cpp:409] Backend details:               ARM: OpenCL 1.1 v1.r6p0-02rel0.62fc6567775af6cc198164823101c352
      I0306 23:35:31.357133  6560 common.cpp:411] Device vendor:                 ARM
      I0306 23:35:31.357172  6560 common.cpp:413] Name:                          Mali-T760
      I0306 23:35:31.357213  6560 common.cpp:415] Total global memory:           2110341120

I observe different behaviour of the caffe-classification-opencl program:

$ ck install package:lib-caffe-bvlc-opencl-clblast-universal \
  --target_os=android21-arm-v7a \
  --env.DISABLE_DEVICE_HOST_UNIFIED_MEMORY=ON \
  --env.DISABLE_DOUBLE_SUPPORT=ON
$ ck compile program:caffe-classification-opencl --target_os=android21-arm-v7a
$ ck run program:caffe-classification-opencl --target_os=android21-arm-v7a
  • SqueezeNet 1.1:
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a168
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.1813 - "n03788365 mosquito net"
      0.0735 - "n01930112 nematode, nematode worm, roundworm"
      0.0704 - "n03958227 plastic bag"
      0.0632 - "n03920288 Petri dish"
      0.0575 - "n01498041 stingray"
      Segmentation fault

(the prediction is wrong; moreover, similar mispredictions are produced for other images)

  • SqueezeNet 1.0:
      executing code ...
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a168
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.9928 - "n02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca"
      0.0015 - "n02509815 lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens"
      0.0011 - "n02096437 Dandie Dinmont, Dandie Dinmont terrier"
      0.0011 - "n02113624 toy poodle"
      0.0004 - "n02085620 Chihuahua"
      Segmentation fault

(the prediction is correct despite the segfault)

  • AlexNet:
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a168
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.9797 - "n02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca"
      0.0051 - "n02509815 lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens"
      0.0033 - "n02111889 Samoyed, Samoyede"
      0.0033 - "n02104029 kuvasz"
      0.0016 - "n02111500 Great Pyrenees"
      Segmentation fault
  • GoogleNet:
    * tmp-output1.tmp

      executing code ...
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a168
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.9953 - "n02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca"
      0.0015 - "n02395406 hog, pig, grunter, squealer, Sus scrofa"
      0.0006 - "n02110341 dalmatian, coach dog, carriage dog"
      0.0004 - "n02104029 kuvasz"
      0.0004 - "n02134084 ice bear, polar bear, Ursus Maritimus, Thalarctos maritimus"
      Segmentation fault
@psyhtest
Copy link
Member Author

psyhtest commented Mar 6, 2017 •
edited
Loading

When setting --target_os=android21-arm-v7a-hard-neon, things are worse, which suggests we shouldn't use this setting for this board:

$ ck install package:lib-caffe-bvlc-opencl-clblast-universal \
  --target_os=android21-arm-v7a-hard-neon \
  --env.DISABLE_DEVICE_HOST_UNIFIED_MEMORY=ON \
  --env.DISABLE_DOUBLE_SUPPORT=ON
$ ck compile program:caffe-classification-opencl --target_os=android21-arm-v7a-hard-neon
$ ck run program:caffe-classification-opencl --target_os=android21-arm-v7a-hard-neon
  • SqueezeNet 1.1:
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a188
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.1367 - "n01930112 nematode, nematode worm, roundworm"
      0.0643 - "n04592741 wing"
      0.0575 - "n04286575 spotlight, spot"
      0.0406 - "n03196217 digital clock"
      0.0279 - "n03729826 matchstick"
      Segmentation fault
  • SqueezeNet 1.0:
      executing code ...
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a188
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.1367 - "n01930112 nematode, nematode worm, roundworm"
      0.0643 - "n04592741 wing"
      0.0575 - "n04286575 spotlight, spot"
      0.0406 - "n03196217 digital clock"
      0.0279 - "n03729826 matchstick"
      Segmentation fault
  • AlexNet:
      executing code ...
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a188
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.0061 - "n01697457 African crocodile, Nile crocodile, Crocodylus niloticus"
      0.0056 - "n09246464 cliff, drop, drop-off"
      0.0056 - "n01774750 tarantula"
      0.0054 - "n01698640 American alligator, Alligator mississipiensis"
      0.0050 - "n04606251 wreck"
  • GoogleNet:
    * tmp-output1.tmp

      executing code ...
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a188
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6ffffffe arg 0x928
      WARNING: linker: librtlxopenme.so: unused DT entry: type 0x6fffffff arg 0x1
      Use GPU with device ID 0
      F0306 17:32:38.638336  6083 filler.hpp:161] Check failed: blob->count()
      *** Check failure stack trace: ***
      Aborted

@psyhtest
Copy link
Member Author

psyhtest commented Mar 6, 2017

It's ironic but on the Galaxy A6 tablet:

      I0306 23:33:33.764765  4304 common.cpp:379] Total devices: 1
      I0306 23:33:33.764847  4304 common.cpp:380] CUDA devices: 0
      I0306 23:33:33.764874  4304 common.cpp:381] OpenCL devices: 1
      I0306 23:33:33.764899  4304 common.cpp:405] Device id:                     0
      I0306 23:33:33.764930  4304 common.cpp:407] Device backend:                OpenCL
      I0306 23:33:33.764955  4304 common.cpp:409] Backend details:               ARM: OpenCL 1.1 v1.r7p0-03rel0.dec36ba15c3f1bc697cb53aefb95b03e
      I0306 23:33:33.765012  4304 common.cpp:411] Device vendor:                 ARM
      I0306 23:33:33.765043  4304 common.cpp:413] Name:                          Mali-T830
      I0306 23:33:33.765076  4304 common.cpp:415] Total global memory:           1909428224

I get the opposite failures with --target_os=android21-arm-v7a:

  • SqueezeNet 1.1
      Use GPU with device ID 0
      ---------- Prediction for cropped_panda.jpg ----------
      0.9949 - "n02510455 giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca"
      0.0011 - "n02113624 toy poodle"
      0.0006 - "n02500267 indri, indris, Indri indri, Indri brevicaudatus"
      0.0006 - "n02085620 Chihuahua"
      0.0003 - "n02509815 lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens"
      Segmentation fault
  • SqueezeNet 1.0, AlexNet, GoogleNet:
      ---------- Prediction for cropped_panda.jpg ----------
      CLBlast: OpenCL error: clEnqueueNDRangeKernel: -5
      F0306 23:41:00.439342  4655 greentea_math_functions.cpp:253] Check failed: static_cast<int>(status) == static_cast<int>(clblast::StatusCode::kSuccess) (-5 vs. 0) GREENTEA ERROR: CLBlast error
      *** Check failure stack trace: ***

@DVEfremov
Copy link
Contributor

I've reproduced at my Galaxy Note 4 but with a little bit another error :

    * tmp-output1.tmp

      executing code ...
      Use GPU with device ID 0
      ---------- Prediction for data.jpg ----------
      F0307 13:08:47.679939 20736 syncedmem.cpp:278] Check failed: mapped_ptr == cpu_ptr_ (0 vs. 0xae87b000) Device claims it support zero copy but failed to create correct user ptr buffer
      *** Check failure stack trace: ***
      Aborted

@psyhtest
Copy link
Member Author

psyhtest commented Mar 7, 2017 •
edited
Loading

That's a different issue. Remember it's a Mali device - you need to build with DISABLE_DEVICE_HOST_UNIFIED_MEMORY=ON e.g.

$ ck install package:lib-caffe-bvlc-opencl-clblast-universal \
  --env.DISABLE_DEVICE_HOST_UNIFIED_MEMORY=ON \
  --env.DISABLE_DOUBLE_SUPPORT=ON \
  --target_os=android21-arm-v7a

@DVEfremov DVEfremov mentioned this issue Mar 7, 2017
Closed
@DVEfremov
Copy link
Contributor

ok, thanks
for my 32 bit Galaxy Note 4
after correct package installation I've got
same prediction for any image
using SqueezeNet 1.1

Use GPU with device ID 0
      ---------- Prediction for photo.jpg ----------
      0.1813 - "n03788365 mosquito net"
      0.0735 - "n01930112 nematode, nematode worm, roundworm"
      0.0704 - "n03958227 plastic bag"
      0.0632 - "n03920288 Petri dish"
      0.0575 - "n01498041 stingray"

without any segmentation fault.

And the same for android app using scenario
Caffe OpenCL SqueezeNet 1.1

@DVEfremov
Copy link
Contributor

For 64 Dima's Xiaomi I've got another error:

      executing code ...
      WARNING: linker: ./classification: unused DT entry: type 0x6ffffffe arg 0xb274
      WARNING: linker: ./classification: unused DT entry: type 0x6fffffff arg 0x1
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6ffffffe arg 0x20a168
      WARNING: linker: libcaffe.so: unused DT entry: type 0x6fffffff arg 0x2

@dsavenko
Copy link
Contributor

dsavenko commented Mar 9, 2017 •
edited
Loading

It's Meizu actually (m3 note). Stop calling it Xiaomi! :)

@gfursin
Copy link
Contributor

gfursin commented Mar 9, 2017 •
edited
Loading

BTW, we also position our Android app now as an engine to crowdsource OpenCL bug detection in DNN ;) !

@psyhtest
Copy link
Member Author

psyhtest commented Mar 9, 2017

I also see linking warnings (see above) - they seem harmless...

BTW, @DVEfremov, when I use different images, I get slightly different results (if I look at the probabilities, not the labels). Is it the same for you?

I'm beginning to think this issue may be related to: naibaf7/caffe#60

@DVEfremov
Copy link
Contributor

I've tested at Galaxy note different images fro same OpeCL SqueezeNet 1.1
but get exactly the same result

@psyhtest
Copy link
Member Author

psyhtest commented Apr 4, 2017

So this issue is still a mystery...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dsavenko @psyhtest @gfursin @DVEfremov