Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Image cdn support checks #718

Closed
wants to merge 32 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f242cc9
feat: image cdn support
kathmbeck Nov 14, 2023
42765a9
fix: package-lock
kathmbeck Nov 14, 2023
fb24aff
Merge remote-tracking branch 'origin/main' into image-cdn-support
kathmbeck Nov 14, 2023
3d7adbd
fix: lint
kathmbeck Nov 15, 2023
5139347
fix: dependencies
kathmbeck Nov 15, 2023
538dd98
feat: testing
kathmbeck Nov 15, 2023
c48feda
fix: update function
kathmbeck Nov 15, 2023
d275cb2
fix: temp remove function
kathmbeck Nov 15, 2023
1c1b10c
fix: reorder redirects
kathmbeck Nov 15, 2023
693ed8f
feat: base url encoded paths
kathmbeck Nov 16, 2023
f220bbe
fix: eslint
kathmbeck Nov 16, 2023
f008de8
Merge branch 'main' into image-cdn-support
kathmbeck Nov 16, 2023
f565c7a
fix: update redirect
kathmbeck Nov 16, 2023
8a89b21
Merge branch 'image-cdn-support' of https://github.com/netlify/netlif…
kathmbeck Nov 16, 2023
940f490
fix: error handling
kathmbeck Nov 16, 2023
3fa9d05
fix: debug log
kathmbeck Nov 16, 2023
5b9e0de
fix: path match
kathmbeck Nov 16, 2023
a7f95ac
fix: more tweaks
kathmbeck Nov 16, 2023
6ad89ad
fix: split
kathmbeck Nov 16, 2023
01a1d3f
fix: url param
kathmbeck Nov 16, 2023
03b2d66
fix: /functions
kathmbeck Nov 20, 2023
0055290
fix: param pattern
kathmbeck Nov 20, 2023
5688b39
fix: properly unencode args
kathmbeck Nov 20, 2023
7c0ff3b
fix: use force for redirects
kathmbeck Nov 21, 2023
bfec59a
fix: generate __image lambda only if NETLIFY_IMAGE_CDN is set
pieh Nov 23, 2023
4041e0d
chore: drop dev/debug logs, skip unnecesary awaits
pieh Nov 23, 2023
4f6a19e
feat: apply caching headers
pieh Nov 24, 2023
3ad83e2
fix: add regular cache-control too
pieh Nov 24, 2023
dda27f6
Merge remote-tracking branch 'origin/main' into image-cdn-support
pieh Nov 24, 2023
aed8e72
chore: use Netlify Image CDN in v5 demo as well
pieh Nov 24, 2023
b799256
chore: update image-cdn docs
pieh Nov 24, 2023
96e27f0
chore: is it netlify-cli?
pieh Nov 24, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
key:
ubuntu-build-${{ env.cache-name }}-${{
hashFiles('plugin/test/fixtures/**/package.json') }}-node-modules
- run: npm install -g netlify-cli
- run: npm install -g netlify-cli@17.5.1
- run: npm ci
- run: cd plugin && npm ci && npm run build
- run: npm test
Expand Down
5 changes: 4 additions & 1 deletion demo-v5/netlify.toml
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
[build]
command = "npm run build"
publish = "public/"
environment = { GATSBY_CLOUD_IMAGE_CDN = "true" }
environment = { NETLIFY_IMAGE_CDN = "true" }
ignore = "if [ $CACHED_COMMIT_REF == $COMMIT_REF ]; then (exit 1); else git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF ..; fi;"

[[plugins]]
package = "../plugin/src/index.ts"

[[plugins]]
package = "@netlify/plugin-local-install-core"

[images]
remote_images = ['https://images.unsplash.com/*']
5 changes: 4 additions & 1 deletion demo/netlify.toml
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
[build]
command = "npm run build"
publish = "public/"
environment = { GATSBY_CLOUD_IMAGE_CDN = "true" }
environment = { NETLIFY_IMAGE_CDN = "true" }
ignore = "if [ $CACHED_COMMIT_REF == $COMMIT_REF ]; then (exit 1); else git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF ..; fi;"

[[plugins]]
package = "../plugin/src/index.ts"

[[plugins]]
package = "@netlify/plugin-local-install-core"

[images]
remote_images = ['https://images.unsplash.com/*']
91 changes: 54 additions & 37 deletions docs/image-cdn.md
Original file line number Diff line number Diff line change
@@ -1,51 +1,68 @@
# Gatsby Image CDN on Netlify

Gatsby Image CDN is a new feature available in the prerelease version of Gatsby.
Instead of downloading and processing images at build time, it defers processing
until request time. This can greatly improve build times for sites with remote
images, such as those that use a CMS. Netlify includes full support for Image
CDN, on all plans.

When using the image CDN, Gatsby generates URLs of the form
`/_gatsby/image/...`. On Netlify, these are served by a
[builder function](https://docs.netlify.com/configure-builds/on-demand-builders/),
powered by [sharp](https://sharp.pixelplumbing.com/) and Nuxt's
[ipx image server](https://github.com/unjs/ipx/). It supports all image formats
supported by Gatsby, including AVIF and WebP.

On first load there will be a one-time delay while the image is resized, but
subsequent requests will be super-fast as they are served from the edge cache.
Gatsby Image CDN is a feature available since Gatsby v4.10.0. Instead of
downloading and processing images at build time, it defers processing until
request time. This can greatly improve build times for sites with remote images,
such as those that use a CMS. Netlify includes full support for Image CDN, on
all plans.

## Enabling the Image CDN

To enable the Image CDN during the beta period, you should set the environment
variable `GATSBY_CLOUD_IMAGE_CDN` to `true`.
To enable the Image CDN, you should set the environment variable
`NETLIFY_IMAGE_CDN` to `true`. You will also need to declare allowed image URL
patterns in `netlify.toml`:

Image CDN currently requires the beta version of Gatsby. This can be installed
using the `next` tag:

```shell
npm install gatsby@next gatsby-plugin-image@next gatsby-plugin-sharp@next gatsby-transformer-sharp@next
```toml
[images]
remote_images = [
'https://example1.com/*',
'https://example2.com/*'
]
```

Currently Image CDN supports Contentful and WordPress, and these source plugins
should also be installed using the `next` tag:
Exact URL patterns to use will depend on CMS you use and possibly your
configuration of it.

```shell
npm install gatsby-source-wordpress@next
```
- `gatsby-source-contentful`:

or
```toml
[images]
remote_images = [
# <your-contentful-space-id> will be specified via `spaceId`
# gatsby-source-contentful option in gatsby-config
"https://images.ctfassets.net/<your-contentful-space-id>/*"
]
```

```shell
npm install gatsby-source-contentful@next
```
- `gatsby-source-drupal`:

```toml
[images]
remote_images = [
# <your-drupal-base-url> will be speciafied via `baseUrl`
# gatsby-source-drupal plugin option in gatsby-config
"<your-drupal-base-url>/*"
]
```

- `gatsby-source-wordpress`:

```toml
[images]
remote_images = [
# <your-wordpress-url> will be specified via `url`
# gatsby-source-wordpress plugin option in gatsby-config
# with "/graphql" endpoint part being omitted
"<your-wordpress-url>/wp-content/uploads/*"
]
```

Gatsby will be adding support to more source plugins during the beta period.
These should work automatically as soon as they are added.
Above examples are the most likely ones to be needed. However if you configure
your CMS to host assets on different domain or path, you might need to adjust
the patterns accordingly.

## Using the Image CDN
## How it works

Your GraphQL queries will need updating to use the image CDN. The details vary
depending on the source plugin. For more details see
[the Gatsby docs](https://support.gatsbyjs.com/hc/en-us/articles/4522338898579)
When using the Image CDN, Gatsby generates URLs of the form
`/_gatsby/image/...`. On Netlify, these are served by a function that translates
Gatsby Image CDN URLs into Netlify Image CDN compatible URL.
14 changes: 14 additions & 0 deletions plugin/src/helpers/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -382,4 +382,18 @@ export function shouldSkip(publishDir: string): boolean {

return shouldSkipResult
}

export function checkNetlifyImageCdn({
netlifyConfig,
}: {
netlifyConfig: NetlifyConfig
}): void {
/* eslint-disable no-param-reassign */
const { NETLIFY_IMAGE_CDN } = netlifyConfig.build.environment

if (NETLIFY_IMAGE_CDN === 'true') {
netlifyConfig.build.environment.GATSBY_CLOUD_IMAGE_CDN = 'true'
}
/* eslint-enable no-param-reassign */
}
/* eslint-enable max-lines */
65 changes: 50 additions & 15 deletions plugin/src/helpers/functions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,14 @@ export const setupImageCdn = async ({
constants: NetlifyPluginConstants
netlifyConfig: NetlifyConfig
}) => {
const { GATSBY_CLOUD_IMAGE_CDN } = netlifyConfig.build.environment

if (GATSBY_CLOUD_IMAGE_CDN !== '1' && GATSBY_CLOUD_IMAGE_CDN !== 'true') {
const { GATSBY_CLOUD_IMAGE_CDN, NETLIFY_IMAGE_CDN } =
netlifyConfig.build.environment

if (
NETLIFY_IMAGE_CDN !== `true` &&
GATSBY_CLOUD_IMAGE_CDN !== '1' &&
GATSBY_CLOUD_IMAGE_CDN !== 'true'
) {
return
}

Expand All @@ -92,26 +97,56 @@ export const setupImageCdn = async ({
join(constants.INTERNAL_FUNCTIONS_SRC, '_ipx.ts'),
)

if (NETLIFY_IMAGE_CDN === `true`) {
await copyFile(
join(__dirname, '..', '..', 'src', 'templates', 'image.ts'),
join(constants.INTERNAL_FUNCTIONS_SRC, '__image.ts'),
)

netlifyConfig.redirects.push(
{
from: '/_gatsby/image/:unused/:unused2/:filename',
// eslint-disable-next-line id-length
query: { u: ':url', a: ':args', cd: ':cd' },
to: '/.netlify/functions/__image/image_query_compat?url=:url&args=:args&cd=:cd',
status: 301,
force: true,
},
{
from: '/_gatsby/image/*',
to: '/.netlify/functions/__image',
status: 200,
force: true,
},
)
} else if (
GATSBY_CLOUD_IMAGE_CDN === '1' ||
GATSBY_CLOUD_IMAGE_CDN === 'true'
) {
netlifyConfig.redirects.push(
{
from: `/_gatsby/image/:unused/:unused2/:filename`,
// eslint-disable-next-line id-length
query: { u: ':url', a: ':args' },
to: `/.netlify/builders/_ipx/image_query_compat/:args/:url/:filename`,
status: 301,
},
{
from: '/_gatsby/image/*',
to: '/.netlify/builders/_ipx',
status: 200,
},
)
}

netlifyConfig.redirects.push(
{
from: `/_gatsby/image/:unused/:unused2/:filename`,
// eslint-disable-next-line id-length
query: { u: ':url', a: ':args' },
to: `/.netlify/builders/_ipx/image_query_compat/:args/:url/:filename`,
status: 301,
},
{
from: `/_gatsby/file/:unused/:filename`,
// eslint-disable-next-line id-length
query: { u: ':url' },
to: `/.netlify/functions/_ipx/file_query_compat/:url/:filename`,
status: 301,
},
{
from: '/_gatsby/image/*',
to: '/.netlify/builders/_ipx',
status: 200,
},
{
from: '/_gatsby/file/*',
to: '/.netlify/functions/_ipx',
Expand Down
3 changes: 3 additions & 0 deletions plugin/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import {
modifyConfig,
shouldSkipBundlingDatastore,
shouldSkip,
checkNetlifyImageCdn,
} from './helpers/config'
import { modifyFiles } from './helpers/files'
import { deleteFunctions, writeFunctions } from './helpers/functions'
Expand Down Expand Up @@ -42,6 +43,8 @@ export async function onPreBuild({
await restoreCache({ utils, publish: PUBLISH_DIR })

await checkConfig({ utils, netlifyConfig })

await checkNetlifyImageCdn({ netlifyConfig })
}

export async function onBuild({
Expand Down
88 changes: 88 additions & 0 deletions plugin/src/templates/image.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
import { Buffer } from 'buffer'

import { Handler } from '@netlify/functions'

type Event = Parameters<Handler>[0]

function generateURLFromQueryParamsPath(uParam, cdParam, argsParam) {
try {
const newURL = new URL('.netlify/images', 'https://example.com')
newURL.searchParams.set('url', uParam)
newURL.searchParams.set('cd', cdParam)

const aParams = new URLSearchParams(argsParam)
aParams.forEach((value, key) => {
newURL.searchParams.set(key, value)
})

return newURL.pathname + newURL.search
} catch (error) {
console.error('Error constructing URL:', error)
return null
}
}

function generateURLFromBase64EncodedPath(path) {
const [, , , encodedUrl, encodedArgs] = path.split('/')

const decodedUrl = Buffer.from(encodedUrl, 'base64').toString('utf8')
const decodedArgs = Buffer.from(encodedArgs, 'base64').toString('utf8')

let sourceURL
try {
sourceURL = new URL(decodedUrl)
} catch (error) {
console.error('Decoded string is not a valid URL:', error)
return
}

const newURL = new URL('.netlify/images', 'https://example.com')
newURL.searchParams.set('url', sourceURL.href)

const aParams = new URLSearchParams(decodedArgs)
aParams.forEach((value, key) => {
newURL.searchParams.set(key, value)
})

return newURL.pathname + newURL.search
}

// eslint-disable-next-line require-await
export const handler: Handler = async (event: Event) => {
const QUERY_PARAM_PATTERN =
/^\/\.netlify\/functions\/__image\/image_query_compat\/?$/i

const { pathname } = new URL(event.rawUrl)
const match = pathname.match(QUERY_PARAM_PATTERN)

let newURL

if (match) {
// Extract the query parameters
const {
url: uParam,
cd: cdParam,
args: argsParam,
} = event.queryStringParameters

newURL = generateURLFromQueryParamsPath(uParam, cdParam, argsParam)
} else {
newURL = generateURLFromBase64EncodedPath(pathname)
}

const cachingHeaders = {
'Cache-Control': 'public,max-age=31536000,immutable',
'Netlify-CDN-Cache-Control': 'public,max-age=31536000,immutable',
'Netlify-Vary': 'query',
}

return newURL
? {
statusCode: 301,
headers: {
Location: newURL,
...cachingHeaders,
},
}
: { statusCode: 400, body: 'Invalid request', headers: cachingHeaders }
}