Is it possible to support Wechat miniprogram?

[qucc]

I found comment about wechat in the file jeelizFaceFilter.moduleNoDOM.js

Is there any demos or tutorial about this topic?

[xavierjs]

Yes it is possible.
jeelizFaceFilter.moduleNoDOM.js is made at this purpose.
The few DOM function are emulated in this module (since in WeChat miniprogram there are no DOM objects). So it can run out of the DOM.
Also the video is different with WeChat, it is not an HTML video element but an array of pixel colors. You need to init FaceFilter with:

JEEFACEFILTERAPI.init({
   NNC: NEURALNETWORKMODEL,
   videoSettings: {
      videoElement: FAKEVIDEOELEMENT
    },
    // ... otherSettings
});

Where you init FAKEVIDEOELEMENT by:

const arrayBuffer = new Uint8Array(vw*vh*4); // vw and vh are video width and height in pixels
  FAKEVIDEOELEMENT = {
    isFakeVideo: true, //always true
    arrayBuffer: arrayBuffer,// the WeChat video arrayBuffer
    videoHeight: vh, // height in pixels
    videoWidth:  vw, //width in pixels
    needsUpdate: true // boolean
  };

and NEURALNETWORKMODEL should be the parsed JSON content of the neural network model (not the URL or the string content)

Once you update FAKEVIDEOELEMENT.arrayBuffer by the content of a new video frame, you need to set FAKEVIDEOELEMENT.needsUpdate=true so that facefilter knows that the frame has changed and can internally update the texture.

[wulixu]

Great!

[qucc]

Thanks for your reply ,
I don't know if i done it right. Today i try the solution above and stuck with this errors.
ERROR in ContextFF: Cannot filter float textures

Here is demo
https://github.com/qucc/minicodear

[wulixu]

I think canvas has different methods or properties in HTML and WeChat, need to modify the dist files for fixing that.

another WeChat mini-program project integrated with tracking.js for the same purpose, please refer to https://github.com/sanyuered/WeChat-MiniProgram-WebAR

[xavierjs]

Hi,
Thank you for your feedback.
Indeed, even if WebGL is implemented for Wechat miniprograms, it is the version1 without extensions (or few of them). However we can make it. I have updated a new version of jeelizFaceFilter.moduleNoDOM.js which should fix the issue (the 3.0.5)

[qucc]

I just test the 3.0.5 this morning , now it show this error.

I also change these lines blow . but still get a error .
https://github.com/qucc/minicodear/blob/39f18559ec8feaacc990a6378c1ef976a3d818dc/index/index.js#L59
https://github.com/qucc/minicodear/blob/39f18559ec8feaacc990a6378c1ef976a3d818dc/libs/jeelizFaceFilter.moduleNoDOM.js#L67

[xavierjs]

Thank you for your feedback. Indeed it seems that in WeChat Miniprogram the createImageData method is carried by the canvas itself instead of the context (https://developers.weixin.qq.com/miniprogram/en/dev/api/canvas/Canvas.createImageData.html) I will push a fix.

[xavierjs]

Hi,
I have pushed a potential fix (version 3.0.6). I hope it will help

[wulixu]

Thanks for your help，we want to share the first demo about wechat app to your demos if done!

[wulixu]

Just buy a beer for you, thanks again!

[wulixu]

Hi, xavierjs

I have tested the latest version you uploaded, then It can run normally in the monitor of Weixin Devtool. unfortunately, It cannot be shown the cube within face in real phone, but the background video mesh is show normally. I did different experiments , It doesn't work. actually, I am not an professional in webgl area, ask for your help.
the latest code is here: https://github.com/qucc/minicodear/tree/threecube
the office Three.js + WeApp tutorial is here : https://github.com/wechat-miniprogram/threejs-miniprogram/tree/master/example

[wulixu]

and, your lib need to fix these errors. in my local version, I just remove theses.

[xavierjs]

Hi @wulixu I have just pushed a fix to this bug (It seems that in WeChat the canvas has no addEventListener method). I hope it will help. The new version number is 3.0.7. I will have more time to work on WeChat issue and test it for real on Wednesday and Thursday this week.

[wulixu]

If you need any help for setting up mini-program env, pls contact with me anytime.

[xavierjs]

Hi @wulixu thank you for your help testing FaceFilter on Wechat is in my work pipeline, I just have some bugs to solve on another project before, I will come back to you once it is done. If you encounter new issues feel free to post them here.

[xavierjs]

Hi @wulixu I have sent you an email to your Github email ( [email protected] ) with my phone number to activate my WeChat registration.

[wulixu]

The first thing is to setup the env in your local machine. so you do some tests for yourself.
and you can refer some projects based tensorflow.js with facelandermark model in WeChat app, here: https://github.com/sanyuered/WeChat-MiniProgram-AR-TFJS
It can be run fine. just like below.
tips: when you use WeChat dev tools, there are some few bugs when to debug mode, just go to preview your result with scan qrcode.

[wulixu]

Thanks for coming back. currently, we just use directly tensorflow.js, while JeelizFaceFilter needs to support miniprograms with webgl, Three.js or something else.

So, If you need any help, pls feel free to contact with via email [email protected]. or we can meet online at your local time 9:00~10:00AM today.

[wulixu]

If done, we can create an chat group on WeChat for the further help!

[xavierjs]

Hi @wulixu I have just sent you an email with my phone number to activate so that I can get my account validated on WeChat. I have redo the subscription process just now so it should not be expired.

[wulixu]

same error, if you are online now, we can try it again.

[xavierjs]

@wulixu I am online now, the confirmation page is opened right now, I don't touch my phone this time !

[xavierjs]

It should work on https://github.com/xavierjs/minicodear, master branch (it displays a 3D cube above user's head).
I have fixed a bug and I have updated the main FaceFilter script. I also use 2 separate canvases (1 for facefilter and video display, and 1 for THREE)

[xavierjs]

Hi @wulixu Whats up on this issue?

[wulixu]

Sorry for late, It cannot work again.
I will push my co-workers to deep into your library working with WeChat mini-program.
if any update, I will contact with you!
Thanks again!

[xavierjs]

Hi @wulixu
Is it because of an update of Jeeliz FaceFilter?
Does https://github.com/xavierjs/minicodear work?