Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP 2024 Chapter #3828

Merged
merged 43 commits into from
Dec 10, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
b6821e3
Initial tryout commit for HTTP chapter
rmarx Oct 24, 2024
43b4524
First part of the HTTP 2024 chapter with images
rmarx Nov 4, 2024
ba4377b
Merge branch 'main' into http2024-markdown
tunetheweb Nov 4, 2024
8f9a2d0
Optimised images with calibre/image-actions
github-actions[bot] Nov 4, 2024
7321984
Update contributors
tunetheweb Nov 4, 2024
ac087ae
Merge branch 'http2024-markdown' of github.com:HTTPArchive/almanac.ht…
tunetheweb Nov 4, 2024
0eb889b
Standardise decimal places
tunetheweb Nov 4, 2024
5a7f4ef
Edits
tunetheweb Nov 4, 2024
ad20895
Links
tunetheweb Nov 4, 2024
7eb9405
Fixes
tunetheweb Nov 4, 2024
c0bcdd7
alt-svc
tunetheweb Nov 4, 2024
d9aba5d
More edits
tunetheweb Nov 4, 2024
613188e
Fix SQL names
tunetheweb Nov 5, 2024
a0220da
Merge branch 'main' into http2024-markdown
tunetheweb Nov 5, 2024
d551cf2
Merge branch 'main' into http2024-markdown
tunetheweb Nov 10, 2024
74050b4
Fix config
tunetheweb Nov 10, 2024
30deb8e
Merge branch 'main' into http2024-markdown
tunetheweb Nov 10, 2024
9387843
Merge branch 'main' into http2024-markdown
tunetheweb Nov 11, 2024
20cef9a
Merge branch 'main' into http2024-markdown
tunetheweb Nov 11, 2024
2c295b4
Merge branch 'main' into http2024-markdown
tunetheweb Nov 12, 2024
0dbf229
Merge branch 'main' into http2024-markdown
tunetheweb Nov 18, 2024
2cb7690
Merge branch 'main' into http2024-markdown
tunetheweb Nov 21, 2024
8ef52f4
Fix contributors
tunetheweb Nov 21, 2024
63d2714
First section of part 2 on resource hints v1 written
rmarx Nov 29, 2024
6908fed
Merge branch 'main' into http2024-markdown
tunetheweb Nov 29, 2024
77a9326
Fix preload queries
tunetheweb Nov 29, 2024
d0dd82b
merge
rmarx Nov 29, 2024
d477c96
Merge branch 'main' into http2024-markdown
tunetheweb Dec 2, 2024
a5e57fb
Finalizing part 2 writing. Finishing edits to part 1.
rmarx Dec 2, 2024
3dd154f
merge
rmarx Dec 2, 2024
92fd4eb
Optimised images with calibre/image-actions
github-actions[bot] Dec 2, 2024
04a766f
Merge branch 'main' into http2024-markdown
tunetheweb Dec 3, 2024
788e8af
Markup cleanups
tunetheweb Dec 3, 2024
e148066
Retake images
tunetheweb Dec 3, 2024
502d2a5
Edits
tunetheweb Dec 3, 2024
72e9ef8
Editing pass Robin
rmarx Dec 3, 2024
31f3165
Update contributor info for author
rmarx Dec 3, 2024
29437bb
Nits
tunetheweb Dec 3, 2024
f5ae1c5
Add and update image/figure descriptions
rmarx Dec 4, 2024
2e8b8cc
Merge branch 'main' into http2024-markdown
tunetheweb Dec 10, 2024
f8b61c0
Extra queries for Robin
tunetheweb Dec 10, 2024
026a1fc
Integrated new results from final 2 queries
rmarx Dec 10, 2024
49699aa
Final final final final final tweaks
tunetheweb Dec 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions sql/2024/http/h3_switches.sql
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
#standardSQL
# Pages which had resources from domains with both h2 and h3 requests
# Note this returns 0 rows at the moment
#

SELECT
Expand All @@ -26,14 +25,12 @@ GROUP BY
date,
client,
page,
url_host,
protocol
url_host
HAVING
h2_requests > 0 AND
h3_requests > 0
ORDER BY
date,
client,
page,
url_host,
protocol
url_host
78 changes: 78 additions & 0 deletions sql/2024/http/h3_switches_pct.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
WITH requests AS (
SELECT DISTINCT
date,
client,
page,
NET.HOST(url) AS url_host,
JSON_EXTRACT_SCALAR(summary, '$.respHttpVersion') AS protocol,
resp_headers.value AS alt_svc
FROM
`httparchive.crawl.requests`
JOIN
UNNEST(response_headers) AS resp_headers ON LOWER(resp_headers.name) = 'alt-svc'
WHERE
date = '2024-06-01' AND
is_root_page
),

url_host_totals AS (
SELECT
date,
client,
url_host,
COUNT(DISTINCT page) AS total_pages
FROM
requests
WHERE
alt_svc LIKE '%h3%' AND
protocol IN ('HTTP/2', 'h2')
GROUP BY
date,
client,
url_host
)

SELECT
client,
url_host,
COUNT(DISTINCT page) AS switched_pages,
total_pages,
COUNT(DISTINCT page) / total_pages AS pct_pages,
ARRAY_TO_STRING(ARRAY_AGG(DISTINCT page LIMIT 5), ', ') AS sample_urls
FROM
url_host_totals
JOIN (
SELECT
date,
client,
page,
url_host,
COUNTIF(protocol IN ('HTTP/2', 'h2')) AS h2_requests,
COUNTIF(protocol IN ('HTTP/3', 'h3', 'h3-29')) AS h3_requests
FROM
requests
GROUP BY
date,
client,
page,
url_host
HAVING
h2_requests > 0 AND
h3_requests > 0
ORDER BY
date,
client,
page,
url_host
)
USING (date, client, url_host)
GROUP BY
client,
url_host,
total_pages
HAVING
switched_pages > 10
ORDER BY
switched_pages DESC,
client,
url_host
51 changes: 51 additions & 0 deletions sql/2024/http/h3_usage_site_with_dns.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
SELECT
date,
client,
dns_https_or_svcb,
CASE
WHEN protocol IN ('HTTP/3', 'h3', 'h3-29') OR
protocol IN ('HTTP/3', 'h3', 'h3-29') THEN 'h3_used'
ELSE 'h3_not_used'
END AS h3_used,
CASE
WHEN protocol IN ('HTTP/3', 'h3', 'h3-29') OR
protocol IN ('HTTP/3', 'h3', 'h3-29') OR
alt_svc LIKE '%h3=%' OR
alt_svc LIKE '%h3-29=%' THEN 'h3_supported'
ELSE 'h3_not_supported'
END AS h3_supported,
COUNT(DISTINCT page) AS num_pages,
SUM(COUNT(DISTINCT page)) OVER (PARTITION BY date, client) AS total_pages,
COUNT(DISTINCT page) / SUM(COUNT(DISTINCT page)) OVER (PARTITION BY date, client) AS pct_pages
FROM (
SELECT
date,
client,
page,
is_main_document,
JSON_EXTRACT(p.payload, '$._origin_dns.https') != '[]' OR JSON_EXTRACT(p.payload, '$._origin_dns.svcb') != '[]' AS dns_https_or_svcb,
JSON_EXTRACT_SCALAR(r.summary, '$.respHttpVersion') AS protocol,
resp_headers.value AS alt_svc
FROM
`httparchive.all.pages` p
JOIN
`httparchive.all.requests` r
USING (date, client, page, is_root_page)
LEFT OUTER JOIN
UNNEST(response_headers) AS resp_headers ON LOWER(resp_headers.name) = 'alt-svc'
WHERE
date = '2024-06-01' AND
is_root_page AND
is_main_document)
GROUP BY
date,
client,
dns_https_or_svcb,
h3_used,
h3_supported
ORDER BY
date,
client,
dns_https_or_svcb,
h3_used,
h3_supported
24 changes: 3 additions & 21 deletions sql/2024/http/lcp_element_data_by_type.sql
Original file line number Diff line number Diff line change
Expand Up @@ -42,24 +42,6 @@ CREATE TEMP FUNCTION getLoadingClasses(attributes STRING) RETURNS STRING LANGUAG
}
''';

CREATE TEMPORARY FUNCTION getResourceHints(linkNodes STRING)
RETURNS STRUCT<preload BOOLEAN, prefetch BOOLEAN, preconnect BOOLEAN, prerender BOOLEAN, `dns-prefetch` BOOLEAN, `modulepreload` BOOLEAN>
LANGUAGE js AS '''
var hints = ['preload', 'prefetch', 'preconnect', 'prerender', 'dns-prefetch', 'modulepreload'];
try {
var linkNodes = JSON.parse(linkNodes);
return hints.reduce((results, hint) => {
results[hint] = !!linkNodes.nodes.find(link => link.rel.toLowerCase() == hint);
return results;
}, {});
} catch (e) {
return hints.reduce((results, hint) => {
results[hint] = false;
return results;
}, {});
}
''';

WITH lcp_stats AS (
SELECT
client,
Expand All @@ -77,7 +59,7 @@ WITH lcp_stats AS (
getFetchPriorityAttr(JSON_EXTRACT(custom_metrics, '$.performance.lcp_elem_stats.attributes')) AS fetchPriority,
LOWER(JSON_EXTRACT_SCALAR(custom_metrics, '$.performance.lcp_resource.initialPriority')) AS initalPriority,
LOWER(JSON_EXTRACT_SCALAR(custom_metrics, '$.performance.lcp_resource.priority')) AS priority,
getResourceHints(JSON_EXTRACT(custom_metrics, '$.almanac.link-nodes')) AS hints
CAST(JSON_EXTRACT(custom_metrics, '$.performance.is_lcp_preloaded') AS BOOL) AS preloaded
FROM
`httparchive.all.pages`
WHERE
Expand Down Expand Up @@ -116,8 +98,8 @@ SELECT
COUNTIF(initalPriority = 'high') / COUNT(DISTINCT page) AS pct_inital_priority_high,
COUNTIF(initalPriority = 'high' AND fetchPriority = 'high') / COUNT(DISTINCT page) AS pct_inital_priority_high_and_fetchpriority,
COUNTIF(loading = 'lazy' AND fetchPriority = 'high') / COUNT(DISTINCT page) AS pct_native_lazyload_and_fetch_priority,
COUNTIF(hints.preload) AS preload,
COUNTIF(hints.preload) / COUNT(DISTINCT page) AS pct_preload
COUNTIF(preloaded) AS preload,
COUNTIF(preloaded) / COUNT(DISTINCT page) AS pct_preload
FROM
lcp_stats
JOIN (
Expand Down
24 changes: 3 additions & 21 deletions sql/2024/http/lcp_element_data_with_urls.sql
Original file line number Diff line number Diff line change
Expand Up @@ -42,24 +42,6 @@ CREATE TEMP FUNCTION getLoadingClasses(attributes STRING) RETURNS STRING LANGUAG
}
''';

CREATE TEMPORARY FUNCTION getResourceHints(linkNodes STRING)
RETURNS STRUCT<preload BOOLEAN, prefetch BOOLEAN, preconnect BOOLEAN, prerender BOOLEAN, `dns-prefetch` BOOLEAN, `modulepreload` BOOLEAN>
LANGUAGE js AS '''
var hints = ['preload', 'prefetch', 'preconnect', 'prerender', 'dns-prefetch', 'modulepreload'];
try {
var linkNodes = JSON.parse(linkNodes);
return hints.reduce((results, hint) => {
results[hint] = !!linkNodes.nodes.find(link => link.rel.toLowerCase() == hint);
return results;
}, {});
} catch (e) {
return hints.reduce((results, hint) => {
results[hint] = false;
return results;
}, {});
}
''';

WITH lcp_stats AS (
SELECT
client,
Expand All @@ -77,7 +59,7 @@ WITH lcp_stats AS (
getFetchPriorityAttr(JSON_EXTRACT(custom_metrics, '$.performance.lcp_elem_stats.attributes')) AS fetchPriority,
LOWER(JSON_EXTRACT_SCALAR(custom_metrics, '$.performance.lcp_resource.initialPriority')) AS initalPriority,
LOWER(JSON_EXTRACT_SCALAR(custom_metrics, '$.performance.lcp_resource.priority')) AS priority,
getResourceHints(JSON_EXTRACT(custom_metrics, '$.almanac.link-nodes')) AS hints
CAST(JSON_EXTRACT(custom_metrics, '$.performance.is_lcp_preloaded') AS BOOL) AS preloaded
FROM
`httparchive.all.pages`
WHERE
Expand Down Expand Up @@ -117,8 +99,8 @@ SELECT
COUNTIF(initalPriority = 'high') / COUNT(DISTINCT page) AS pct_inital_priority_high,
COUNTIF(initalPriority = 'high' AND fetchPriority = 'high') / COUNT(DISTINCT page) AS pct_inital_priority_high_and_fetchpriority,
COUNTIF(loading = 'lazy' AND fetchPriority = 'high') / COUNT(DISTINCT page) AS pct_native_lazyload_and_fetch_priority,
COUNTIF(hints.preload) AS preload,
COUNTIF(hints.preload) / COUNT(DISTINCT page) AS pct_preload
COUNTIF(preloaded) AS preload,
COUNTIF(preloaded) / COUNT(DISTINCT page) AS pct_preload
FROM
lcp_stats
JOIN (
Expand Down
3 changes: 1 addition & 2 deletions src/config/2024.json
Original file line number Diff line number Diff line change
Expand Up @@ -157,8 +157,7 @@
"part": "IV",
"chapter_number": "20",
"title": "HTTP",
"slug": "http",
"todo": true
"slug": "http"
},
{
"part": "IV",
Expand Down
12 changes: 10 additions & 2 deletions src/config/contributors.json
Original file line number Diff line number Diff line change
Expand Up @@ -676,9 +676,10 @@
"reviewers"
],
"2024": [
"analysts",
"committee",
"developers",
"editors",
"committee",
"reviewers"
]
},
Expand Down Expand Up @@ -1026,13 +1027,15 @@
"website": "https://chrisadams.me.uk"
},
"ChrisBeeti": {
"avatar_url": "32492572",
"github": "ChrisBeeti",
"name": "Chris Böttger",
"teams": {
"2024": [
"analysts",
"authors",
"committee"
"committee",
"reviewers"
]
}
},
Expand Down Expand Up @@ -3856,6 +3859,8 @@
"rmarx": {
"avatar_url": "2240689",
"github": "rmarx",
"linkedin": "rmarx",
"bluesky": "programmingart.bsky.social",
"name": "Robin Marx",
"teams": {
"2019": [
Expand All @@ -3869,6 +3874,9 @@
],
"2022": [
"reviewers"
],
"2024": [
"authors"
]
},
"twitter": "programmingart",
Expand Down
4 changes: 2 additions & 2 deletions src/config/last_updated.json
Original file line number Diff line number Diff line change
Expand Up @@ -787,8 +787,8 @@
"hash": "0972e7e4c8b572e58aa8dead2b4d7e13"
},
"en/2024/chapters/http.html": {
"date_published": "2024-11-11T00:00:00.000Z",
"date_modified": "2024-11-16T00:00:00.000Z",
"date_published": "2024-12-10T00:00:00.000Z",
"date_modified": "2024-12-10T00:00:00.000Z",
"hash": "aa71b4be8ab44d4379d6ba6c8fbc78b7"
},
"en/2024/chapters/jamstack.html": {
Expand Down
Loading
Loading