Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

M1 fixes (#309) #316

Merged
merged 10 commits into from
Oct 11, 2021
Merged

M1 fixes (#309) #316

merged 10 commits into from
Oct 11, 2021

Conversation

dh-ecc
Copy link
Contributor

@dh-ecc dh-ecc commented Oct 1, 2021

[Addressing #309]

This is the absolute bare minimum diff to get the pod integration working for M1 machines, which can be demonstrated in this repo by pod installing in the Example/ZcashLightClientSample/ directory and running on simulator.

In doing so we remove some rust build dependencies, but (at the time of writing) can only build on the rust nightly toolchain, so we add this as a dependency.

Caveats / Known issues

  • This works only for cocoapods integration, other modes of building have not been updated.
  • We currently only build the rust library for a single architecture at a time, so this prevents some types of archive builds. (This is addressed with xcframeworks in the assessment of [Assessment/Research Spike] Simplifying Cocoapods build and Distributing over SPM #311) I've added back in cargo-lipo for the time being so we can build universal builds, but this will still be superseded when moving to xcframeworks.
  • The docs have not been updated since they will likely be superceded to remove rust dependency a consumer (and to some extent a contributor) of the SDK.
  • The Cargo.lock updates are sufficient to make the rust lib build correctly on the nightly toolchain, however I don't know if this is the most idiomatic way to declare this, I feel like there could be a better alternative by declaring something in the Cargo.toml instead.

The following commands should be sufficient to test this working and should work on both intel and m1 macs.
It assumes you have working rustup and pod commands, try running it in a /tmp/ folder 😄

rustup toolchain add nightly
rustup +nightly target add \
x86_64-apple-ios \
aarch64-apple-ios \
aarch64-apple-ios-sim
git clone [email protected]:dh-ecc/ZcashLightClientKit.git
cd ZcashLightClientKit
git checkout m1-fixes
cd Example/ZcashLightClientSample
pod install
SIMULATOR_ID="$(xcrun simctl list devices available 'iPhone' | grep -E -o -i '([0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12})' | head -n 1)"
xcodebuild \
-workspace ZcashLightClientSample.xcworkspace \
-scheme ZcashLightClientSample \
-destination "id=$SIMULATOR_ID" \
clean \
build

The build command is a very fancy way of saying "build it for an iOS Simulator on my machine" without having to assume an OS version or device that the host might have 😅, Some inspiration from https://nshipster.com/simctl/
. Would be much nicer if it was possibe to something like-destination "generic/platform=iOS Simulator" but as far as I know this is not possible... please let me know if there is!

Author

  • Self-review: Did you review your own code in GitHub's web interface? Code often looks different when reviewing the diff in a browser, making it easier to spot potential bugs.
  • Automated tests: Did you add appropriate automated tests for any code changes? N/A
  • Code coverage: Did you check the code coverage report for the automated tests? While we are not looking for perfect coverage, the tool can point out potential cases that have been missed. N/A
  • Documentation: Did you update Docs as appropiate? (E.g README.md, etc.) Explicitly deferred
  • Run the app: Did you run the app and try the changes?
  • Did you provide Screenshots of what the App looks like before and after your changes as part of the description of this PR? (only applicable to UI Changes) N/A
  • Rebase and squash: Did you pull in the latest changes from the main branch and squash your commits before assigning a reviewer? Having your code up to date and squashed will make it easier for others to review. Use best judgement when squashing commits, as some changes (such as refactoring) might be easier to review as a separate commit. Rebased + attempt at useful individual commits

Reviewer

  • Checklist review: Did you go through the code with the Code Review Guidelines checklist?
  • Ad hoc review: Did you perform an ad hoc review? In addition to a first pass using the code review guidelines, do a second pass using your best judgement and experience which may identify additional questions or comments. Research shows that code review is most effective when done in multiple passes, where reviewers look for different things through each pass.
  • Automated tests: Did you review the automated tests?
  • Manual tests: Did you review the manual tests?You will find manual testing guidelines under our manual testing section
  • How is Code Coverage affected by this PR? We encourage you to compare coverage befor and after your changes and when possible, leave it in a better place. Learn More...
  • Documentation: Did you review Docs, README.md, LICENSE.md, and Architecture.md as appropriate?
  • Run the app: Did you run the app and try the changes? While the CI server runs the app to look for build failures or crashes, humans running the app are more likely to notice unexpected log messages, UI inconsistencies, or bad output data.

dh-ecc and others added 4 commits September 28, 2021 18:14
We no longer want to exclude arm64 for simulator SDKs as this excludes simulators running on M1 machines
Previously, the script would build for the `x86_64-apple-ios` whenever building for the `iphonesimulator` platform,
however, M1 macs introduce a new target architecture for the iOS Simulator, `aarch64-apple-ios-sim`.

Luckily the (now three) targets can be uniquely determined by the combination of the $PLATFORM_NAME + the architecture of the host computer.

Unfortunately This new architecture is currently only supported by the nightly toolchain,
so we must update the build command accordingly.

Since a build only requires one target architecture for `librustzcash` we can drop the `cargo-lipo` dependency,
which makes the resulting build command easier.

Using nightly broke some `num-bigint` dependencies, so we update these to a compatible patch version.
For some reason `orchard` builds on revision `d0baa18f` were hanging indefinitely _only_ when building in release mode.

Did a git bisect to find a more recent revision that still builds, (but does not fail due to other upstream requirements)
and found `624a834` to be the first working one!
@pacu
Copy link
Contributor

pacu commented Oct 4, 2021

Travis CI Build is failing with

=== BUILD TARGET ZcashLightClientKit OF PROJECT Pods WITH CONFIGURATION Debug ===
Building Rust backend
platform name
iphonesimulator
fix 'permission denied issue'
cargo +nightly build --manifest-path /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../Cargo.toml --target x86_64-apple-ios --release
export LIBRARY_PATH="/Applications/Xcode-12.5.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/lib:"
error: toolchain 'nightly-x86_64-apple-darwin' is not installed
/Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../Scripts/build_librustzcash_xcode.sh: line 44: persist_environment: command not found
clean up existing artifacts: rm -f /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../ZcashLightClientKit/zcashlc/libzcashlc.a
clean up sdk lib path: rm -f /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../lib/libzcashlc.a
copying artifacts: cp -f /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../ZcashLightClientKit/zcashlc/libzcashlc.a
cp: /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a: No such file or directory
copying artifacts: cp -f /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../lib/libzcashlc.a
cp: /Users/travis/build/zcash/ZcashLightClientKit/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a: No such file or directory
** BUILD FAILED **
/Users/travis/.travis/functions: line 607:  4158 Terminated: 15          travis_jigger "${!}" "${timeout}" "${cmd[@]}"
The command "travis_wait 60 xcodebuild -quiet -UseModernBuildSystem=NO -workspace ./Example/ZcashLightClientSample/ZcashLightClientSample.xcworkspace -scheme ZcashLightClientSample -destination platform\=iOS\ Simulator,OS\=14.5,name\=iPhone\ 8 build" exited with 65.

@pacu
Copy link
Contributor

pacu commented Oct 4, 2021

Bitrise CI is failing with this error

▸ Running script '[CP-User] Build generate constants and build librustzcash'
❌  error: toolchain 'nightly-x86_64-apple-darwin' is not installed
Last lines of the Xcode's build log:
cp: /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a: No such file or directory
copying artifacts: cp -f /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../lib/libzcashlc.a
cp: /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a: No such file or directory
** BUILD FAILED **
The following build commands failed:
   PhaseScriptExecution [CP-User]\ Build\ generate\ constants\ and\ build\ librustzcash /Users/vagrant/Library/Developer/Xcode/DerivedData/ZcashLightClientSample-bgjqxuwsktfenyabttyfngigqezy/Build/Intermediates.noindex/Pods.build/Release-iphonesimulator/ZcashLightClientKit.build/Script-C9462958FFCBCF2AA9566CCBDC027C0C.sh
(1 failure)
You can find the last couple of lines of Xcode's build log above, but the full log is also available in the raw-xcodebuild-output.log
The log file is stored in $BITRISE_DEPLOY_DIR, and its full path is available in the $BITRISE_XCODE_RAW_RESULT_TEXT_PATH environment variable
(value: /Users/vagrant/deploy/raw-xcodebuild-output.log)
Build failed, error: exit status 65

Copy link
Contributor

@pacu pacu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dh-ecc please review the comments from Str4d and the errors on CI

@dh-ecc
Copy link
Contributor Author

dh-ecc commented Oct 4, 2021

[@pacu]

Travis CI Build is failing with [...]

I'll get onto this now 👍 Need to update the rust environment setup.

[@pacu]

Bitrise CI is failing with this error

I might need your help to update some pre build steps for setting up the rust environment, should be basically the same as the update to the travis script.

This adds the nightly toolchain as well as the required targets
The upstream revision for `librustzcash` no longer exists! (It's pointing to a branch that is still being worked on, though should be merged soon)

This change updates the reference in `Cargo.toml` to a compatible one and updates:
- `rust/src/lib.rs`
- `ZcashLightClientKit/zcashlc/zcashlc.h`
- `ZcashLightClientKit/Rust/ZcashRustBackend.swift`

... to account for the updates in the new upstream reference.
Copy link
Contributor

@nuttycom nuttycom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! nuttycom/librustzcash@cc58a21 is the most recent commit on the librustzcash branch and I hope the commit that will actually be merged soon, but the commit you're at here should be fine.

ZCASH_ACTIVE_ARCHITECTURE="aarch64-apple-ios"
fi

echo "fix 'permission denied issue'"
chmod -R +w ${PODS_TARGET_SRCROOT}

echo "cargo lipo --manifest-path ${PODS_TARGET_SRCROOT}/Cargo.toml --targets $ZCASH_ACTIVE_ARCHITECTURE --release"
echo "cargo +nightly build --manifest-path ${PODS_TARGET_SRCROOT}/Cargo.toml --target $ZCASH_ACTIVE_ARCHITECTURE --release"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@str4d suggested that we don't build nightly when the Xcode target is release. This would make every build, to run in nightly.

nightly should be used (for now) only for targets that require so. otherwise do it as always.

NOTE: This could be kind of confusing. The --release argument is for rust, which is fine and should be built that way, otherwise the rust compiler will run without optimizations.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeh this sounds like a good idea 👍 I'll add this change.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in 7cec435.

dh-ecc added 2 commits October 5, 2021 19:19
We now explicitely pin a nightly toolchain version for building for arm64 iOS simulator and use stable toolchain for any other architecture
@pacu
Copy link
Contributor

pacu commented Oct 5, 2021

Bitrise is failing with

Running script '[CP-User] Build generate constants and build librustzcash'
    Patch `jubjub v0.7.0 (https://github.com/zkcrypto/jubjub.git?rev=96ab4162b83303378eae32a326b54d88b75bffc2#96ab4162)` was not used in the crate graph.
❌  error: could not compile `cfg-if` due to previous error
    build failed, waiting for other jobs to finish...
❌  error: build failed
Last lines of the Xcode's build log:
copying artifacts: cp -f /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../lib/libzcashlc.a
+ cp -f /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../lib/libzcashlc.a
cp: /Users/vagrant/git/Example/ZcashLightClientSample/Pods/../../../target/x86_64-apple-ios/release/libzcashlc.a: No such file or directory
** BUILD FAILED **

@dh-ecc dh-ecc force-pushed the m1-fixes branch 2 times, most recently from 3b59336 to 6161e30 Compare October 6, 2021 10:54
Reintroduce `cargo-lipo` as a dependency since bitrise is building for multiple architectures.
@dh-ecc
Copy link
Contributor Author

dh-ecc commented Oct 6, 2021

@pacu Bitrise is failing because it is building in Release configuration which builds all available platform architectures, since it's for simulator it will need to have an x86_64 and arm64 slice.

To sort this I reintroduced cargo-lipo 😅, this will be temporary since we'll remove all of the rust tooling in the near future.

Would you be able to add a cargo install cargo-lipo to the bitrise setup? (again 😓 sorry)

@pacu pacu self-requested a review October 11, 2021 16:23
@pacu pacu merged commit 2b8a63d into Electric-Coin-Company:master Oct 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants