Support output dimensions with local dev

[deBhal]

This PR combines #18 & #22. Each of those PRs "works" as a standalone PR, and we can merge either of them, but in this case each PR makes the other easier to test, so in this case it's arguably easier to review the one big diff rather than the two little ones independantly.

As a reviewer please use the PR(s) that you feel most comfortable with, including commenting/approving whatever you feel comfortable with. I'll keep the three branches in sync and the other two branches operating independantly until one of them gets merged and everything gets simple again, so you can approve just one of the bases if you like.

There were a number of little fixes required after combining these branches together, and cleaning up the git history doesn't seem like a good use of time unless someone cares. If you'd like to review/approve one of the source branches separately let me know and happy to cherry-pick all the changes into the two source branches.

Testing Instructions

• npm install
• npm start
• Navigate to something like http://127.0.0.1:8000/mshots/v1/https://public-api.wordpress.com/rest/v1/template/demo/rockfield/rockfield%3Ffont_headings%3DPlayfair%20Display%26font_base%3DFira%20Sans%26site_title%3DRockfield%26viewport_height%3D600%26language%3Dfr?vph=1000&screen_height=3000
• You'll be redirected to Wait a few seconds while the screenshot is generated (you can use docker-compose logs to watch for the output if you prefer)
• navigate to the url again

You can find examples of the URLs we care about on the https://wordpress.com/new/style page.

• go to https://wordpress.com/new/style
• open devtools network requests
• choose a design

This will get you a url like:
https://s0.wp.com/mshots/v1/https%3A%2F%2Fpublic-api.wordpress.com%2Frest%2Fv1%2Ftemplate%2Fdemo%2Fmaywood%2Fmaywood%3Ffont_headings%3DPlayfair%2520Display%26font_base%3DFira%2520Sans%26site_title%3DMaywood%26viewport_height%3D700%26language%3Dde?vpw=1600&vph=1600&w=600&h=600&count=1

Replace https://s0.wp.com/ with 127.0.0.1:8000/ to point it at your dev server and add the new ?screen_height=3600 param to mshots itself, giving us a url like:

127.0.0.1:8000/mshots/v1/https%3A%2F%2Fpublic-api.wordpress.com%2Frest%2Fv1%2Ftemplate%2Fdemo%2Fmaywood%2Fmaywood%3Ffont_headings%3DPlayfair%2520Display%26font_base%3DFira%2520Sans%26site_title%3DMaywood%26viewport_height%3D700%26language%3Dde?vpw=1600&vph=1600&w=600&h=600&count=1&screen_height=3600

You can also point calypso at it by editing wp-calypso/client/landing/gutenboarding/components/mshots-image/index.tsx:
`const mshotsUrl = 'http://127.0.0.1:8000/mshots/v1/';

-    const mshotsUrl = 'https://s0.wp.com/mshots/v1/';
+    const mshotsUrl = 'http://127.0.0.1:8000/mshots/v1/';

The problem with this approach is that you'll most likely swamp your little dev server and get a bunch of timeouts resulting in 404s. Then mshots will cache those 404 images and return them until you tell it to try again by adding &requeue=true to the query, e.g.: http://127.0.0.1:8000/mshots/v1/https%3A%2F%2Fpublic-api.wordpress.com%2Frest%2Fv1%2Ftemplate%2Fdemo%2Fmaywood%2Fmaywood%3Ffont_headings%3DPlayfair%2520Display%26font_base%3DFira%2520Sans%26site_title%3DMaywood%26viewport_height%3D700%26language%3Dde?vpw=1600&vph=1600&w=600&h=600&count=1&screen_height=3600&requeue=true

Once you requeue the individual images in a separate tab you can refresh the design page to see them there.

[autumnfjeld]

Local mShots is working! 🥳

I installed this in /opt/mshots/ (had to create that folder).

One hickup when I ran npm install:

Error response from daemon: Mounts denied:
The path /opt/mshots is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing. 

Problem solved after following those instructions there to add /opt/mshots directory in Docker.

I was able to successfully generate screenshots for :

http://127.0.0.1:8000/mshots/v1/https%3A%2F%2Fpublic-api.wordpress.com%2Frest%2Fv1%2Ftemplate%2Fdemo%2Fbalasana%2Fbalasana%3Ffont_headings%3DOpen%2520Sans%26font_base%3DChivo%26site_title%3DBalasana%26viewport_height%3D700%26language%3Den?vpw=1600&vph=1600&w=600&h=600&count=1

and

http://127.0.0.1:8000/mshots/v1/https://public-api.wordpress.com/rest/v1/template/demo/rockfield/rockfield%3Ffont_headings%3DPlayfair%20Display%26font_base%3DFira%20Sans%26site_title%3DRockfield%26viewport_height%3D600%26language%3Dfr?vph=1000&screen_height=3000

and

http://127.0.0.1:8000/mshots/v1/public-api.wordpress.com/rest/v1/template/demo/coutoire/coutoire/?language=ar&site_title=Brave%252C%2520Not%2520Perfect%2520with%2520Reshma%2520Saujani&font_headings=Playfair%2520Display&font_base=Fira%2520Sans?screen_height=3600

As we discussed in our call here were my newbie points of confusion:

To get theme url and generate a screenshot from the running local mShots:

1. Go to https://wordpress.com/new/style
2. Click on the theme of interest
3. In dev tools run '127.0.0.1:8000/mshots/v1/' + location.hostname + location.pathname + encodeURI( location.search ) + '?screen_height=3600'
4. Copy the resulting encoded url into new browser tab
5. If this is a new screenshot being generated you'll be redirected to https://s0.wp.com/mshots/v1/default and you will see loading spinner
6. Wait a couple minutes and then repaste the url from step 4 in the browser and see the screenshot! Voila!

I looked for the generated screenshots in /opt/mshots/public_html/thumbnails but that folder was empty.

Great job @deBhal

cc: @ramonjd Hope this helps with handholding ✋

[deBhal]

I looked for the generated screenshots in /opt/mshots/public_html/thumbnails but that folder was empty.

The folder will be relative to your mshots checkout - ls $(git rev-parse --show-toplevel)/public_html/thumbnails

[p-jackson]

I seem to be having trouble testing this, although I've managed to get it going in the past :(

All I can generate is these 404 screenshots

This is even for simple requests like http://localhost:8000/mshots/v1/example.com

I do see the spinner initially. It's like the container doesn't have access to the internet or something.

[p-jackson]

Ignore that picture, example.com is supposed to be a 404 😆

This is what I see

I'm not seeing any errors in docker-compose logs 🤔

I've also tried bashing into the running container and starting the service manually just to make sure it's actually started. It is. So I'm a bit confused.

Edit: I also ran wget google.com inside the container to make sure I actually had internet connection

[deBhal]

@p-jackson and I looked into those issues and they're related to sandboxing the public-api endpoint.

For now, please just unsandbox any sites you want to point mshots at. It would be really nice to get the dev setup working with sandboxed sites, but we'll punt that feature to a subsequent PR

[ramonjd]

This is amazing.

Install and boot worked first time for me.

I decided to run Calypso locally and set the mshots URL to run through http://127.0.0.1:8000/

Looks fantastic.

The only curiosity was that I had to remove URL encoding (encodeURIComponent) to get it to work locally.

So it renders this

http://127.0.0.1:8000/mshots/v1/https://public-api.wordpress.com/rest/v1/template/demo/rockfield/rockfield?font_headings=Playfair%20Display&font_base=Fira%20Sans&site_title=Rockfield&viewport_height=700&language=en&vpw=1600&vph=1600&w=600&h=600&count=1

But not this

http://127.0.0.1:8000/mshots/v1/https%3A%2F%2Fpublic-api.wordpress.com%2Frest%2Fv1%2Ftemplate%2Fdemo%2Frockfield%2Frockfield%3Ffont_headings%3DPlayfair%2520Display%26font_base%3DFira%2520Sans%26site_title%3DRockfield%26viewport_height%3D700%26language%3Den?vpw=1600&vph=1600&w=600&h=600&count=1

I'm not sure why. The rawurldecode is there 🦐

[andrewserong]

This is testing nicely for me too, @deBhal, but I can also recreate the issue @ramonjd mentioned above. I wonder if it's a difference between how Apache behaves in our local dev containers, and nginx in prod? I suppose in that second URL (http://127.0.0.1:8000/mshots/v1/https%3A%2F%2Fpublic-api.wordpress.com%2Frest%2Fv1%2Ftemplate%2Fdemo%2Frockfield%2Frockfield%3Ffont_headings%3DPlayfair%2520Display%26font_base%3DFira%2520Sans%26site_title%3DRockfield%26viewport_height%3D700%26language%3Den?vpw=1600&vph=1600&w=600&h=600&count=1) we're encoding too much of it? We might only want to encode the query string of the site we're taking a screenshot of, and not the entire url (which appears to be what's happening here in Gutenboarding) and in the page layout picker here). Instead of URI encoding the target url in those places, maybe it'd be better to sanitize?

Or, alternately, figure out how to get Apache to behave as we'd expect with the additional encoding?

[deBhal]

Or, alternately, figure out how to get Apache to behave as we'd expect with the additional encoding?

Yep, you were totally right. I turned on all the logging and found:

AH00026: found %2f (encoded '/') in URI (decoded='/mshots/v1/http://example.com'), returning 404

After that it was just a matter of learning a new sed trick to get around an old apache bug ;)

[ramonjd]

After that it was just a matter of learning a new sed trick to get around an old apache bug ;)

Brilliant!

[deBhal]

I've created Automattic/wp-calypso#51261 to test this PR, and everything is looking great, so I think I'm ready to merge now and close the three branches this one swallowed (#18, #20 and #22)

I'll just hold off in case systems would like an opportunity to review it in a nice convenient PR format.

[autumnfjeld]

Congrats @deBhal ! This was an epic work and getting this submitted to systems is a huge milestone! Many props to you and to everyone who helped review and provide feedback. 🎉

[deBhal]

Update: Barry explicitly said "merge the changes", so I've done that, and I'll just hold off on deleting the branch until the changes are merged instead.