The demo apk does not output correct results

[tlhchutang]

I directly run the apk build by this project without any change on my own android device.
However, it seems that the outputs(top5 labels) do not make any sense, I also tried to use a better model, that is the GoogleNet converted from the Caffe model zoo using the tool provided by Caffe2, the result is still very strange, none of the top 5 labels is correct at all time.

[bwasti]

This may be an issue related to the YUV decoding (and improper input being fed into the predictor) based on your particular device. Please pull down the latest version with a fix, which hopefully resolves the issue for you. If that does not work, may I ask what device you are using?

[tlhchutang]

@bwasti
Thanks for your reply, in fact, there are two different issues in my case, one is the caffe2 translator bug https://github.com/caffe2/caffe2/issues/595 which I have worked around;
the other is just as you mentioned, the yuv to rgb decoding issue, this still exists even if I use the latest version, however, I am able to get correct results using OpenCV's convert function, btw., I am not sure about the device type I am using, because it is an customized device for experiment :X .
Still very appreciate for your kind reply!

[xshwen]

I don't know if I am right or not. Maybe (max[_j - 1] = max[_j];) should be changed to (max[_j] = max[_j - 1];) in the native-lib.cpp.

[northeastsquare]

@tlhchutang Hello, I met the same problem like you.How do you use Opencv's convert function, Can you give it a detailed description?

[northeastsquare]

@tlhchutang , @bwasti maybe from yuv to rgb conversion code is not correct, I make some change , result seems better https://github.com/northeastsquare/AICamera

[zvadaszi]

Tested @northeastsquare's change and results seems to be good. Tested on a Motorola G5 Plus (~2.9fps)

[xshipeng]

I think the error comes from yuv to rgb conversion and I have clarified it in #14.
Moreover, if using PyTorch to train the model, the input_data in JNI should be divided by 255 to be in the scale of 0 to 1 just like what it is in PyTorch and the channel should be RGB rather than BGR. In my case, when I train the model in PyTorch, I use

trans = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])

so the code in native_cpp should be

float r_mean = 0.485f;
float g_mean = 0.456f;
float b_mean = 0.406f;

float r_std=0.229f;
float g_std=0.224f;
float b_std=0.225f;

 auto r_i = 0 * IMG_H * IMG_W + j * IMG_W + i;
 auto g_i = 1 * IMG_H * IMG_W + j * IMG_W + i;
 auto b_i = 2 * IMG_H * IMG_W + j * IMG_W + i;

if (infer_HWC) {
                r_i = (j * IMG_W + i) * IMG_C;
                g_i = (j * IMG_W + i) * IMG_C + 1;
                b_i = (j * IMG_W + i) * IMG_C + 2;
            }

input_data[r_i] =  (((float) ((float) min(255., max(0., (float) (y + 1.402 * (v - 128))))))/255.0f-r_mean)/r_std;
input_data[g_i] = (((float) ((float) min(255., max(0., (float) (y - 0.34414 * (u -128) - 0.71414 * (v - 128))))))/255.0f-g_mean)/g_std;
input_data[b_i] = ((float) ((float) min(255., max(0., (float) (y + 1.772 * (u - v)))))/255.0f-b_mean)/b_std;            

And I also change the IMG_W and IMG_H to 224

[vlomonaco]

Hi all!

Has this bug been fixed? My results are pretty poor with the standard SqueezeNet provided on an Asus ZenFone3. Do you know the exact pre-processing made during training for this model?

Thanks!