Added connect to Pi button #103

sidntrivedi012 · 2019-07-16T19:25:39Z

Adding the static button as of now. In this thread will work on #75.
/cc : @jywarren @starkblaze01 @Dhiraj240 @harshithpabbati

jywarren · 2019-07-16T20:15:44Z

Awesome!!!

sidntrivedi012 · 2019-07-17T20:36:49Z

Hey, I have added the following code from Infragram to the example file of capture. It can be found here - https://github.com/sidntrivedi012/spectral-workbench.js/blob/pi-button/examples/capture/index.html#L363-L380 . Have also included the infragram.js lib here.

But, when I run index.html file of capture, it shows the following error -

Any suggestions how to resolve it? Thanks :)

/cc: @jywarren @starkblaze01 @harshithpabbati @Dhiraj240

sidntrivedi012 · 2019-07-18T11:12:42Z

I have added the following code to index.html in the examples/capture directory. But the error of event being null still persists. Any suggestion how to resolve this?

    var infragram, boost, openInSequencer;
    document.getElementById("pibutton").onclick=function connect_to_pi(){
    //infragram code to connect to raspberry-pi
      infragram = Infragram({
        uploader: false,
        processor: 'webgl',
        shaderVertPath: "../../node_modules/dist/shader.vert",
        shaderFragPath: "../../node_modules/dist/shader.frag"
      });
      var piImage = new Image();
      piImage.onload = function () {
        infragram.options.processor.updateImage(piImage);
      };
      var fetchImageInterval = setInterval(function fetchImage() {
        // piImage.src = "http://pi.local/cam/cam_pic.php?time=" + new Date().getTime();
        piImage.src = "../logo.png";
      }, 50);
    };

jywarren · 2019-07-19T22:44:34Z

examples/capture/index.html

+  if(el){
+    //if pi_div button does not return null
+document.getElementById("pi-button").addEventListener("click", function connect_to_pi() {
+  infragram = Infragram({


Ah, actually i don't think it's necessary to include all of Infragram here, but we can simply borrow the appropriate sections of code! Actually all we want is the function that fetches an image from the Pi. Let me highlight.

Ok. Removing infragram initialisation from here.

jywarren · 2019-07-19T22:45:15Z

examples/capture/index.html

+    };
+    var fetchImageInterval = setInterval(function fetchImage() {
+      // piImage.src = "http://pi.local/cam/cam_pic.php?time=" + new Date().getTime();
+      piImage.src = "../logo.png";


:-) you've got this part right!

Thanks 🎉

jywarren · 2019-07-19T23:02:13Z

examples/capture/index.html

+  });
+    var piImage = new Image();
+    piImage.onload = function () {
+      infragram.options.processor.updateImage(piImage);


So here, instead of instantiating Infragram, we should add the image to the SWB canvas, so it gets grabbed and sent to the graph.

$W.ctx.drawImage(piImage, 0, 0, $W.width, 1); // this will draw it to the waterfall canvas

The only problem here is that it's actually drawing the WHOLE camera's output, squishing it to 1 pixel tall. So, see https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/drawImage

Instead let's crop the source image dimensions too!

Suggested change

infragram.options.processor.updateImage(piImage);

var sourceX = 0, sourceY = 0, sourceHeight = 300, sourceWidth = 800; // although we may modify these later to match the Pi output image dimensions

$W.ctx.drawImage(piImage, sourceX, sourceY, sourceHeight, sourceWidth, 0, 0, $W.width, 1); // this will draw it to the waterfall canvas

However, it seems we may also need to stop this from getting images from the webcam:

spectral-workbench.js/examples/capture/capture.js

Line 164 in 07014a8

onSave: function (data) {

So, we ought to add a conditional there that checks for a new property $W.loadingFromPi, and when you click that button we'll set that to true -- that way it no longer pulls from the webcam, make sense?

@jywarren I have added a condition to onSave:

onSave: function(data) { if ($W.loadingfromPi == false) { <onSave code snippet> } },

And assigned default value to loadingfromPi to false. I am setting the value of it to true when the button is being clicked.

Can you just push up your latest code when you have a chance? Thanks!

jywarren · 2019-07-19T23:02:31Z

package.json

    "getusermedia-js": "~1.0.0",
-    "d3":              "~3.4.0"
+    "infragram": "^0.2.1",


So we can remove this too!

Sure, removing it. Thanks.

jywarren · 2019-07-21T20:49:00Z

Hi @sidntrivedi012, any updates here? Thank you!!!

jywarren · 2019-07-21T20:54:24Z

Oh sorry, i see you responded yesterday. Thanks!

sidntrivedi012 · 2019-07-23T21:26:48Z

@jywarren , Apologies for the delay. Have tried to fix the loopholes as pointed by you. Also, bumped some dependencies except flot since the demo was showing ERR:file not found errors with the newer version of flot. PTAL. Thanks. 🙂

/cc: @starkblaze01 @Dhiraj240 @harshithpabbati

jywarren · 2019-07-24T17:58:06Z

Were you able to try putting this up on your own gh-pages so we can try it out live? That would be great!

jywarren · 2019-07-24T17:58:46Z

you'll have to add the node modules to the gh-pages branch and push them up too, without adding them to your main branch. It's a little finicky, but give it a try!

sidntrivedi012 · 2019-07-29T14:15:36Z

@jywarren , Have pushed the code to fork's gh-pages branch. Can be seen here - https://sidntrivedi012.github.io/spectral-workbench.js/examples/capture .

sidntrivedi012 · 2019-07-29T14:31:27Z

What href should I add here - https://github.com/sidntrivedi012/spectral-workbench.js/blob/876d657fd88e15b5e01b7b8373b69d068446a197/examples/capture/index.html#L127 ?

Should I add #capture ?

jywarren

OK, I traced this a bit and found that there's a second place the image is being copied, this time to the preview window. It may not be the only issue, because i see that the graph is also not sourcing from the pi image... let me keep looking, but this definitely will help!

jywarren · 2019-08-02T09:34:12Z

So, the broader issue here is that there are references to $("video")[0] throughout -- and uses of video or window.video both of which reference that same <video> element.

What we need, I think, is to make a new function called $W.getVideoEl(), which contains the conditional, so:

function getVideoEl() {
  if ($W.loadingfromPi) return $W.piImage;
  else return $("video")[0];
}

Note I'm using the property $W.piImage instead of the locally defined piImage, as in my earlier comment, so that change will have to happen too.

Does this make sense, @sidntrivedi012 ? It's a change throughout the file, but then you'll be able to change any reference to the original video element to instead be to $W.getVideoEl() which will route correctly.

jywarren · 2019-08-14T19:19:14Z

Hi, can you update us on progress here? Thank you!

…

On Wed, Jul 31, 2019 at 10:59 AM Siddhant N Trivedi < ***@***.***> wrote: Ok @starkblaze01 <https://github.com/starkblaze01> . I will be testing it tonight on my local fork repo gh-pages. If it works, will make a commit. Thanks. 🙂 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#103?email_source=notifications&email_token=AAAF6J4OP3GCJERYQGEZMTLQCGSCRA5CNFSM4IEGNJUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3HRDGQ#issuecomment-516886938>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J7SW2SIFPLTSLYW6XLQCGSCRANCNFSM4IEGNJUA> .

sidntrivedi012 · 2019-08-21T17:37:26Z

@jywarren Can you please review it now? I have made the changes as suggested by you.

jywarren · 2019-08-21T18:28:00Z

Hi, Siddhant, i'm not seeing the static image load when I press Connect. Can you look and try to figure out why? Thanks!

sidntrivedi012 · 2019-08-21T18:34:31Z

Sure, looking into it.

sidntrivedi012 · 2019-08-22T13:18:51Z

So, the broader issue here is that there are references to $("video")[0] throughout -- and uses of video or window.video both of which reference that same <video> element.

What we need, I think, is to make a new function called $W.getVideoEl(), which contains the conditional, so:
function getVideoEl() {
  if ($W.loadingfromPi) return $W.piImage;
  else return $("video")[0];
}
Note I'm using the property $W.piImage instead of the locally defined piImage, as in my earlier comment, so that change will have to happen too.

Does this make sense, @sidntrivedi012 ? It's a change throughout the file, but then you'll be able to change any reference to the original video element to instead be to $W.getVideoEl() which will route correctly.

Hey @jywarren , can you please guide me how and where to use this function? I mean where will this function be called?

jywarren · 2019-08-22T16:34:46Z

Basically all the instances of $("video")[0], video or window.video reference the same <video> element, but it's quite messy how it's set up. So you should replace all instances of that throughout the file with a standard method getVideoEl(). That way, you can switch between the Pi sample image and the true webcam all in one place, since all references to the video element will be routed through that function. Make sense?

sidntrivedi012 · 2019-08-22T18:18:58Z

Basically all the instances of $("video")[0], video or window.video reference the same <video> element, but it's quite messy how it's set up. So you should replace all instances of that throughout the file with a standard method getVideoEl(). That way, you can switch between the Pi sample image and the true webcam all in one place, since all references to the video element will be routed through that function. Make sense?

@jywarren window.video is being used only once in capture.js here -

spectral-workbench.js/examples/capture/capture.js

Line 102 in eaee043

window.video = video

I was thinking to hard code a conditional here. Just to ask, what should we insert here if the image is being loaded from the Pi?

if(getVideoEL()==$("video")[0]) window.video=video;
else <<what to use here?>>

Regarding the rest, I think we can easily use a function as suggested by you.

jywarren · 2019-08-22T22:09:10Z

You can do window.video - getVideoEl() at that spot.

The function should be:

function getVideoEl() {
  if ($W.loadingfromPi) return $W.piImage;
  else return $("video")[0];
}

That way it returns an image or video element no matter where you call it from. Make sense?

sidntrivedi012 · 2019-08-23T12:23:09Z

@jywarren Its somewhat working. Also have made some changes in the code. Will push in a new commit. PTAL. Thanks.

The problem now is that the page is distorting when one clicks connect to Pi.

jywarren · 2019-08-23T20:38:59Z

I think the distortion is OK; it's mapping the aspect ratio of the source image to the area available for display. This won't affect the spectrum. Is this ready then? Let's merge it!

sidntrivedi012 · 2019-08-23T21:46:14Z

@jywarren But there's a problem that unfortunately the live plot has shifted downwards.

jywarren · 2019-08-23T22:33:46Z

I think you may have updated to Bootstrap 4 somehow? And that is affecting the styling?

sidntrivedi012 · 2019-08-24T13:45:13Z

@jywarren Nope, I didn't upgrade to Bootstrap 4. Maybe there's something else. Will debug it. Thanks.

sidntrivedi012 · 2019-08-24T18:16:50Z

@jywarren Also, there's one more problem that even when the image is being loaded from the Pi, the plot still gets made of the image from the camera.

So, as of now there are two problems :

Adjustment of the canvas plot.
Plotting of the data from Pi when connected to Pi instead of the device camera

sidntrivedi012 · 2019-08-28T13:19:49Z

@jywarren Since this thread has become very long, should we tackle the two issues(as mentioned in the above comment) emerging due to the connect to Pi feature in another PR?

sidntrivedi012 · 2019-08-28T13:32:28Z

@jywarren You were right. I had upgraded bootstrap from 3.4.0 (currently being used) version to 4.3.0 . And thus, maybe the plot was being formed on the lower end. But when I installed the 3.4.0 version of bootstrap again. The video capturing stopped.

sidntrivedi012 · 2019-08-28T15:56:46Z

@jywarren This is happening from when I downgraded the version of bootstrap. Don't know why. Searching it out. Do you have any idea why this may be occuring?

jywarren · 2019-08-28T21:08:27Z

Hm, how odd. Are you seeing any errors in the console?

…

On Wed, Aug 28, 2019 at 11:57 AM Siddhant N Trivedi < ***@***.***> wrote: [image: videonotworking] <https://user-images.githubusercontent.com/31186013/63871912-69683000-c9da-11e9-94b6-c2e9d325b95e.gif> @jywarren <https://github.com/jywarren> This is happening from when I downgraded the version of bootstrap. Don't know why. Searching it out. Do you have any idea why this may be occuring? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#103?email_source=notifications&email_token=AAAF6J2JMKA6BGJ7POGLLCDQG2N35A5CNFSM4IEGNJUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5LTMHA#issuecomment-525809180>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAF6J6K3YEOBPVB6BW7WD3QG2N35ANCNFSM4IEGNJUA> .

sidntrivedi012 · 2019-08-28T21:16:20Z

@jywarren Nah, no errors in the console.

jywarren · 2019-08-30T21:41:20Z

I would use the Element inspector to see if the proper canvas or video element is being displayed under the area in question. Perhaps some CSS has gone wrong? Or, can you confirm that all libraries are installed with npm install ?

sidntrivedi012 · 2019-08-31T03:45:41Z

@jywarren Have checked all the libraries and installed them with npm install. Don't know what's going wrong. Also, no error is being displayed in the console. Will debug it today and send a patch by evening surely.

sidntrivedi012 · 2019-08-31T04:10:27Z

@jywarren Its working now. PTAL.

sidntrivedi012 added 3 commits July 13, 2019 16:52

<demo-fix>added width in style.css file

87ec896

added connect to pi button

f7bff2e

Merge branch 'main' into pi-button

b0b7d5d

sidntrivedi012 mentioned this pull request Jul 16, 2019

Add "capture" variant code for Raspberry Pi camera streaming #75

Open

added infragram npm module

63043fc

sidntrivedi012 added 3 commits July 18, 2019 01:48

added infragram code of piImage

0ff7ee7

removed requirejs npm module

5f89593

changed test source for image from static file to cam

90d1f57

fixed the src in newly added function

ca81510

added the buggy code

a44169f

sidntrivedi012 mentioned this pull request Jul 19, 2019

PLANNING ISSUE : Spectral Workbench Capture #92

Closed

5 tasks

jywarren reviewed Jul 19, 2019

View reviewed changes

sidntrivedi012 added 3 commits July 24, 2019 02:46

bumped dependencies. Kept flot at same version

a34234d

added code to connect to pi

15feed6

removed infragram from package.json

44279d1

added grunt-cli

0f95bb5

sidntrivedi012 requested a review from starkblaze01 July 29, 2019 17:45

jywarren requested changes Aug 2, 2019

View reviewed changes

made changes

43b216a

replaced piImage with .piImage in html file

56cc268

added function and checked

6a765d5

sidntrivedi012 force-pushed the pi-button branch from d8a0343 to acbe4ca Compare August 23, 2019 12:34

sidntrivedi012 force-pushed the pi-button branch from acbe4ca to 6a765d5 Compare August 31, 2019 04:02

jywarren mentioned this pull request Jul 27, 2021

Spectral Workbench goals #159

Open

30 tasks

	infragram.options.processor.updateImage(piImage);
	var sourceX = 0, sourceY = 0, sourceHeight = 300, sourceWidth = 800; // although we may modify these later to match the Pi output image dimensions
	$W.ctx.drawImage(piImage, sourceX, sourceY, sourceHeight, sourceWidth, 0, 0, $W.width, 1); // this will draw it to the waterfall canvas

Added connect to Pi button #103

Are you sure you want to change the base?

Added connect to Pi button #103

Conversation

sidntrivedi012 commented Jul 16, 2019

jywarren commented Jul 16, 2019

sidntrivedi012 commented Jul 17, 2019

sidntrivedi012 commented Jul 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jywarren commented Jul 21, 2019

jywarren commented Jul 21, 2019

sidntrivedi012 commented Jul 23, 2019 • edited Loading

jywarren commented Jul 24, 2019

jywarren commented Jul 24, 2019

sidntrivedi012 commented Jul 29, 2019 • edited Loading

sidntrivedi012 commented Jul 29, 2019

jywarren left a comment

Choose a reason for hiding this comment

jywarren commented Aug 2, 2019

jywarren commented Aug 14, 2019 via email

sidntrivedi012 commented Aug 21, 2019

jywarren commented Aug 21, 2019

sidntrivedi012 commented Aug 21, 2019

sidntrivedi012 commented Aug 22, 2019

jywarren commented Aug 22, 2019

sidntrivedi012 commented Aug 22, 2019 • edited Loading

jywarren commented Aug 22, 2019

sidntrivedi012 commented Aug 23, 2019

jywarren commented Aug 23, 2019

sidntrivedi012 commented Aug 23, 2019 • edited Loading

jywarren commented Aug 23, 2019

sidntrivedi012 commented Aug 24, 2019

sidntrivedi012 commented Aug 24, 2019

sidntrivedi012 commented Aug 28, 2019

sidntrivedi012 commented Aug 28, 2019 • edited Loading

sidntrivedi012 commented Aug 28, 2019

jywarren commented Aug 28, 2019 via email

sidntrivedi012 commented Aug 28, 2019

jywarren commented Aug 30, 2019

sidntrivedi012 commented Aug 31, 2019

sidntrivedi012 commented Aug 31, 2019

sidntrivedi012 commented Jul 23, 2019 •

edited

Loading

sidntrivedi012 commented Jul 29, 2019 •

edited

Loading

sidntrivedi012 commented Aug 22, 2019 •

edited

Loading

sidntrivedi012 commented Aug 23, 2019 •

edited

Loading

sidntrivedi012 commented Aug 28, 2019 •

edited

Loading