Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apple Silicon macOS test/builds #3077

Merged
merged 30 commits into from
May 25, 2024
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
5fa00f3
Enable build/test on macos-14 Actions Runners, which are Silicon/ARM64
philrz May 15, 2024
17d65db
Merge Intel and ARM64 release info into single latest-mac.yml file
philrz May 21, 2024
f6467ca
Merge Intel and ARM64 release info for Zui Insiders releases also
philrz May 22, 2024
5f57378
Change GitHub issue comment link
philrz May 22, 2024
ff6e8a2
PR feedback: Fix redundant REPO const
philrz May 22, 2024
5f50914
Use artifactName in electron-builder config to add arch to DMG filenames
philrz May 23, 2024
ecfbff4
PR feedback: Adjust arch references in Actions name for merging lates…
philrz May 23, 2024
d8fbdac
PR feedback: Comment explaining macos-12 is Intel (x86) and macos-14 …
philrz May 23, 2024
c5fefd1
Merge script needs to expect x86 in DMG filenames
philrz May 23, 2024
f20c316
PR feedback: Simplify YAML merge spread
philrz May 23, 2024
84fa204
PR feedback: Drop else after return
philrz May 23, 2024
021eb54
PR feedback: Drop else after return and don't both making const
philrz May 23, 2024
4f127e6
PR feedback: Drop else after return
philrz May 23, 2024
95c542e
PR feedback: Readable.from() rather than assembling localAssetStream …
philrz May 23, 2024
85f4ace
PR feedback: Don't bother initializing remotePlatformFile
philrz May 23, 2024
019735b
PR feedback: Change how return is done
philrz May 23, 2024
eb9468f
PR feedback: Drop explicit return
philrz May 24, 2024
ef013d9
PR feedback: Drop explicit return
philrz May 24, 2024
4d7fa77
PR feedback: Readable.from() rather than assembling assetStream via push
philrz May 24, 2024
c60f7e3
PR feedback: dumpOptions on one line
philrz May 24, 2024
78fb254
PR feedback: return instead of creating mergedString
philrz May 24, 2024
b343b9c
Remove another else
philrz May 24, 2024
f5a34ed
Remove another else
philrz May 24, 2024
2cfd313
PR feedback: Drop semicolons
philrz May 24, 2024
03eb879
PR feedback: Move originalAsset block outside of try
philrz May 24, 2024
501c79b
Drop another explicit return
philrz May 24, 2024
465259c
x86 -> x64 (what electron-builder calls the Intel arch)
philrz May 24, 2024
c575d52
Merge branch 'main' into mac-silicon
philrz May 24, 2024
61538de
PR feedback: Move comment
philrz May 25, 2024
b3a0694
Remove comment in original location
philrz May 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/actions/build-zui/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ runs:
env:
GH_TOKEN: ${{ inputs.gh_token }}
APPLE_ID: ${{ inputs.apple_id }}
APPLE_ID_PASSWORD: ${{ inputs.apple_id_password }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ inputs.apple_id_password }}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we transition from the separate electron-builder-notarize to leveraging the notarization support that's in electron-builder, the name of this environment variable changes but everything else stays the same.

APPLE_TEAM_ID: ${{ inputs.apple_team_id }}
CODE_SIGN_SCRIPT_PATH: ${{ github.workspace }}/esigner-codesign/dist/index.js
INPUT_FILE_PATH: ${{ steps.paths.outputs.artifact }}
Expand All @@ -87,5 +87,5 @@ runs:
- name: Check notorization with gatekeeper
if: runner.os == 'macOS'
run: |
spctl --assess --type execute --verbose --ignore-cache --no-cache dist/apps/zui/mac/*.app
spctl --assess --type execute --verbose --ignore-cache --no-cache dist/apps/zui/mac*/*.app
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wildcard is needed here because the Apple Silicon builds end up in a subdirectory mac-arm64/ whereas the Intel ones were in mac/.

shell: bash
5 changes: 4 additions & 1 deletion .github/actions/setup-zui/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,13 @@ runs:
with:
go-version: '1.21'

# Caching is disabled because it resulted in getting x86_64 Zed binaries
# on arm64 builds. See:
# - https://github.com/actions/setup-node/issues/1008
- name: Install Node
uses: actions/setup-node@v3
with:
cache: yarn
# cache: yarn
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In an early test iteration, @mattnibs spotted that the zed binaries bundled with Zui were still showing up as Intel. I ultimately traced it to a side effect of this caching, then found open issue actions/setup-node#1008 that seems to indicate this is a known problem. I briefly went down the path of trying to find another way to keep caching in play via some other config, but I also noticed that disabling caching wasn't slowing down the build process much at all. In the interest of simplicity I figured we could just live without it for now. I'm Subscribed to that issue so if/when it gets addressed I could revisit this. In the meantime I've included this comment so someone doesn't turn it on again without recognizing the effect it'll have on the macOS builds.

philrz marked this conversation as resolved.
Show resolved Hide resolved
node-version-file: .node-version

- name: Cache NextJS Artifacts
Expand Down
13 changes: 12 additions & 1 deletion .github/actions/upload-build-artifacts/action.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,25 @@
name: Upload Build Artifacts
description: Upload artifacts for each platform
inputs:
gh_token:
required: true

runs:
using: 'composite'
steps:
- uses: actions/upload-artifact@v3
with:
name: Mac Artifact
name: Mac Artifact (${{ runner.arch }})
path: dist/apps/zui/*.dmg

- name: Merge latest-mac.yml Mac release files for x64/arm64
if: runner.os == 'macOS'
run: |
node apps/zui/scripts/merge-mac-release-files.mjs
env:
GH_TOKEN: ${{ inputs.gh_token }}
shell: bash

- uses: actions/upload-artifact@v3
with:
name: Windows Artifact
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/build-insiders.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ jobs:
needs: check_latest
strategy:
matrix:
platform: [windows-2019, macos-12, ubuntu-20.04]
# macos-12 is is Intel-based (x64), macos-14 is Apple Silicon (arm64)
platform: [windows-2019, macos-12, macos-14, ubuntu-20.04]
philrz marked this conversation as resolved.
Show resolved Hide resolved

runs-on: ${{ matrix.platform }}
steps:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ jobs:
release:
strategy:
matrix:
platform: [macos-12, ubuntu-20.04, windows-2019]
# macos-12 is is Intel-based (x64), macos-14 is Apple Silicon (arm64)
platform: [macos-12, macos-14, ubuntu-20.04, windows-2019]
runs-on: ${{ matrix.platform }}
steps:
- name: Checkout Zui
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ jobs:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [macos-12, ubuntu-20.04, windows-2019]
# macos-12 is is Intel-based (x64), macos-14 is Apple Silicon (arm64)
os: [macos-12, macos-14, ubuntu-20.04, windows-2019]
steps:
- run: git config --global core.autocrlf false
- uses: actions/checkout@v3
Expand Down
11 changes: 10 additions & 1 deletion .github/workflows/release-insiders.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ jobs:
if: ${{ needs.check_latest.outputs.latest_sha != github.sha }}
strategy:
matrix:
platform: [windows-2019, macos-12, ubuntu-20.04]
# macos-12 is is Intel-based (x64), macos-14 is Apple Silicon (arm64)
platform: [windows-2019, macos-12, macos-14, ubuntu-20.04]

runs-on: ${{ matrix.platform }}
steps:
Expand Down Expand Up @@ -67,6 +68,14 @@ jobs:
cert_p12: ${{ secrets.APPLE_DEVELOPER_ID_CERT_P12_BASE64 }}
cert_passphrase: ${{ secrets.APPLE_DEVELOPER_ID_CERT_PASSPHRASE }}

- name: Merge latest-mac.yml Mac release files for x64/arm64
if: runner.os == 'macOS'
run: |
node apps/zui/scripts/merge-mac-release-files.mjs
env:
GH_TOKEN: ${{ secrets.PAT_TOKEN }}
shell: bash

- name: Inform Slack users of failure
uses: tiloio/[email protected]
if: ${{ failure() }}
Expand Down
5 changes: 4 additions & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ jobs:
release:
strategy:
matrix:
platform: [macos-12, ubuntu-20.04, windows-2019]
# macos-12 is is Intel-based (x64), macos-14 is Apple Silicon (arm64)
platform: [macos-12, macos-14, ubuntu-20.04, windows-2019]
runs-on: ${{ matrix.platform }}
steps:
- name: Checkout Zui
Expand Down Expand Up @@ -38,3 +39,5 @@ jobs:

- name: Upload Artifacts
uses: ./.github/actions/upload-build-artifacts
with:
gh_token: ${{ secrets.GITHUB_TOKEN }}
8 changes: 8 additions & 0 deletions apps/zui/darwin.plist
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.cs.allow-jit</key>
<true/>
</dict>
</plist>
6 changes: 5 additions & 1 deletion apps/zui/electron-builder.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,15 @@
"sign": "./scripts/sign.js"
},
"linux": {"target": ["deb", "rpm"]},
"mac": {
"entitlements": "darwin.plist",
"notarize": {"teamId": "2DBXHXV7KJ"},
"artifactName": "${productName}-${version}-${arch}.${ext}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first when using electron-builder at its defaults, we'd get a a DMG for Intel like Zui-1.7.4.dmg and for Apple Silicon like Zui-1.7.4-arm64.dmg. This use of artifactName gives us symmetry by having the Intel one now have the arch included as well and hence Zui-1.7.4-x64.dmg.

},
"rpm": {"depends": ["openssl"]},
"deb": {"depends": ["openssl"]},
"nsis": {"oneClick": false, "perMachine": false},
"forceCodeSigning": true,
"afterSign": "electron-builder-notarize",
"publish": {
"provider": "github"
},
Expand Down
5 changes: 3 additions & 2 deletions apps/zui/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,7 @@
"debut-css": "^0.7.0",
"decompress": "^4.2.1",
"electron": "30.0.1",
"electron-builder": "^23.6.0",
"electron-builder-notarize": "^1.2.0",
"electron-builder": "^24.13.3",
"electron-devtools-assembler": "^1.2.0",
"electron-dl": "^3.0.1",
"electron-localshortcut": "^3.2.1",
Expand All @@ -110,6 +109,7 @@
"jest": "^28.0.0",
"jest-css-modules-transform": "^4.4.2",
"jest-environment-jsdom": "^28.0.0",
"js-yaml": "^4.1.0",
"jwt-decode": "^3.1.2",
"lint-staged": "^12.1.5",
"livereload": "^0.9.1",
Expand All @@ -123,6 +123,7 @@
"node-fetch": "^2.6.1",
"nodemon": "^2.0.22",
"npm-run-all": "^4.1.5",
"octokit": "^4.0.2",
"ohm-js": "^17.0.4",
"on-idle": "^3.1.4",
"polished": "^3.6.5",
Expand Down
196 changes: 196 additions & 0 deletions apps/zui/scripts/merge-mac-release-files.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
// Based on https://github.com/electron-userland/electron-builder/issues/5592#issuecomment-2004803764
import { Octokit } from "octokit"
import pkg from '../package.json' assert {type: 'json'}
import { TextDecoder } from 'node:util'
import fs from 'node:fs'
import { Readable } from 'node:stream'
import { Buffer } from 'node:buffer'
import yaml from 'js-yaml'

const token = process.env.GH_TOKEN

const client = new Octokit({
auth: token
})

// These are derived from settings in package.json so the script will work on
// both regular Zui and Zui Insiders.
const OWNER = pkg.repository.split('/')[3]
const REPO = pkg.repository.split('/')[4]
const PRODUCT_NAME= pkg.productName.replaceAll(' ', '-')
const URL = `/repos/${OWNER}/${REPO}/releases`
const VERSION = pkg.version
const RELEASE_NAME = (PRODUCT_NAME == 'Zui') ? 'v' + VERSION : VERSION
const FILE_NAME = 'latest-mac.yml'
const LOCAL_FILE_PATH = `dist/apps/zui/${FILE_NAME}`

const mergeFiles = (intel, arm) => {
const intelObject = yaml.load(intel)
const armObject = yaml.load(arm)

const mergedObject = {
...intelObject,
files: [...intelObject.files, ...armObject.files]
}

// avoids moving the sha512 checksum into its own line
const dumpOptions = { lineWidth: -1 }

return yaml.dump(mergedObject, dumpOptions)
}

const getPlatformFromLatestMacYml = (content) => {
const intelRe = `${PRODUCT_NAME}-${VERSION}-x64.dmg`
const armRe = `${PRODUCT_NAME}-${VERSION}-arm64.dmg`
const isIntel = content.includes(intelRe)
const isArm = content.includes(armRe)

if (isIntel && isArm) return 'both'
if (isIntel && !isArm) return 'intel'
if (!isIntel && isArm) return 'arm'

return 'none'
}

(async () => {
const allReleases = await client.request(`GET ${URL}`)
const currentRelease = allReleases.data.find(release => {
return release.name === RELEASE_NAME
})

if (!currentRelease) {
console.log('No release found. Skipping merge')
return
}
console.log('Release found')

if (!fs.existsSync(LOCAL_FILE_PATH)) {
console.log(`[local] could not find ${FILE_NAME}. Skipping merge`)
return
}
console.log(`[local] ${FILE_NAME} found`)

const localLatestMacYmlContent = fs.readFileSync(LOCAL_FILE_PATH, { encoding: 'utf8' })

const localPlatform = getPlatformFromLatestMacYml(localLatestMacYmlContent)

if (localPlatform === 'none' || localPlatform === 'both') {
console.log(`[local] ${FILE_NAME} invalid. Platform: ${localPlatform}. Skipping merge`)
return
}
console.log(`[local] ${FILE_NAME} valid: Platform: ${localPlatform}`)

const localPlatformPresentRemotely = currentRelease.assets.find(asset => asset.name === `latest-mac-${localPlatform}.yml`)

if (localPlatformPresentRemotely) {
try {
await client.request(`DELETE ${URL}/assets/${localPlatformPresentRemotely.id}`)
console.log(`[remote] deleted latest-mac-${localPlatform}.yml`)
} catch(e) {
console.log(`[remote] error deleting latest-mac-${localPlatform}.yml. Skipping merge`)
console.log(e)
return
}
}

const uploadUrl = currentRelease.upload_url
try {
await client.rest.repos.uploadReleaseAsset({
url: uploadUrl,
headers: {
'content-type': 'application/octet-stream',
'content-length': Buffer.byteLength(localLatestMacYmlContent),
},
name: `latest-mac-${localPlatform}.yml`,
data: Readable.from(localLatestMacYmlContent),
})
console.log(`[remote] latest-mac-${localPlatform}.yml uploaded`)
} catch(e) {
console.log(`[remote] error uploading latest-mac-${localPlatform}.yml. Skipping merge`)
console.log(e)
return
}

const remotePlatform = localPlatform === 'intel' ? 'arm' : 'intel'

const remotePlatformFileExists = currentRelease.assets.find(asset => asset.name === `latest-mac-${remotePlatform}.yml`)

if (!remotePlatformFileExists) {
console.log(`[remote] latest-mac-${remotePlatform}.yml does not exist. Skipping merge`)
return
}
console.log(`[remote] latest-mac-${remotePlatform}.yml found`)

let remotePlatformFile

try {
remotePlatformFile = await client.request(`GET ${URL}/assets/${remotePlatformFileExists.id}`, {
headers: {
accept: 'application/octet-stream'
}
})
console.log(`[remote] latest-mac-${remotePlatform}.yml downloaded`)
} catch(e) {
console.log(`[remote] error downloading latest-mac-${remotePlatform}.yml. Skipping merge`)
console.log(e)
return
}

const remoteLatestMacYmlContent = new TextDecoder().decode(remotePlatformFile.data)

const originalAsset = currentRelease.assets.find(asset => asset.name === FILE_NAME)

if (!originalAsset) {
console.log(`[remote] ${FILE_NAME} not found. Skipping merge`)
return
}
console.log(`[remote] ${FILE_NAME} found`)

try {
await client.request(`DELETE ${URL}/assets/${originalAsset.id}`)
console.log(`[remote] deleted ${FILE_NAME}`)
} catch(e) {
console.log(`[remote] error deleting ${FILE_NAME}. Skipping merge`)
console.log(e)
return
}

const mergedContent = remotePlatform === 'intel' ? mergeFiles(remoteLatestMacYmlContent, localLatestMacYmlContent) : mergeFiles(localLatestMacYmlContent, remoteLatestMacYmlContent)

try {
await client.rest.repos.uploadReleaseAsset({
url: uploadUrl,
headers: {
'content-type': 'application/octet-stream',
'content-length': Buffer.byteLength(mergedContent),
},
name: FILE_NAME,
data: Readable.from(mergedContent),
})
console.log(`[remote] uploaded merged ${FILE_NAME}`)
} catch(e) {
console.log(`[remote] error uploading merged ${FILE_NAME}. Skipping merge`)
console.log(e)
return
}

// cleanup
const updatedRelease = await client.request(`GET ${URL}`)
const updatedCurrentRelease = updatedRelease.data.find(release => release.name === RELEASE_NAME)

const assetsToClean = updatedCurrentRelease.assets.filter(asset => {
return asset.name === `latest-mac-arm.yml` || asset.name === `latest-mac-intel.yml`
})

for (const assetToClean of assetsToClean) {
try {
await client.request(`DELETE ${URL}/assets/${assetToClean.id}`)
console.log(`[remote:cleanup] deleted ${assetToClean.name}`)
} catch(e) {
console.log(`[remote:cleanup] error deleting ${assetToClean.name}`)
console.log(e)
}
}

console.log('Merge complete')
})()
Loading
Loading