Spark NLP on Apple new M1 Chip #2282

maziyarpanahi · 2021-02-12T12:22:05Z

maziyarpanahi
Feb 12, 2021
Maintainer

I personally don't have a Macbook with M1 Chip, but from what I've read so far the issue with Apache Spark and Spark NLP comes from the JDK. I suggest the following link to download Java JDK that is compatible with M1 (must be 8 and must be arm-64):

https://www.azul.com/downloads/zulu-community/?version=java-8-lts&os=macos&architecture=arm-64-bit&package=jdk

Let me know if you have M1 Chip and facing any issue using Spark NLP :)

aizazulhaq · 2021-09-02T23:15:50Z

aizazulhaq
Sep 2, 2021

I am trying to run spark nlp on my m1. I installed the zulu jdk8 using sdkman. It is giving me the following error.

Warning: Could not load Loader: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path
Exception in thread "main" java.lang.UnsatisfiedLinkError: no jnitensorflow in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1875)
	at java.lang.Runtime.loadLibrary0(Runtime.java:872)
	at java.lang.System.loadLibrary(System.java:1124)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1683)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1300)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1123)
	at org.tensorflow.internal.c_api.global.tensorflow.<clinit>(tensorflow.java:12)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1139)
	at org.tensorflow.NativeLibrary.load(NativeLibrary.java:48)
	at org.tensorflow.TensorFlow.init(TensorFlow.java:111)
	at org.tensorflow.TensorFlow.<clinit>(TensorFlow.java:126)
	at org.tensorflow.Graph.<clinit>(Graph.java:849)
	at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.readGraph(TensorflowWrapper.scala:368)
	at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.unpackWithoutBundle(TensorflowWrapper.scala:297)
	at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:426)
	at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel$class.readTensorflowModel(TensorflowSerializeModel.scala:145)
	at com.johnsnowlabs.nlp.annotators.sentence_detector_dl.SentenceDetectorDLModel$.readTensorflowModel(SentenceDetectorDLModel.scala:453)
	at com.johnsnowlabs.nlp.annotators.sentence_detector_dl.ReadsSentenceDetectorDLGraph$class.readSentenceDetectorDLGraph(SentenceDetectorDLModel.scala:431)
	at com.johnsnowlabs.nlp.annotators.sentence_detector_dl.SentenceDetectorDLModel$.readSentenceDetectorDLGraph(SentenceDetectorDLModel.scala:453)
	at com.johnsnowlabs.nlp.annotators.sentence_detector_dl.ReadsSentenceDetectorDLGraph$$anonfun$6.apply(SentenceDetectorDLModel.scala:435)
	at com.johnsnowlabs.nlp.annotators.sentence_detector_dl.ReadsSentenceDetectorDLGraph$$anonfun$6.apply(SentenceDetectorDLModel.scala:435)
	at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead$1.apply(ParamsAndFeaturesReadable.scala:31)
	at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead$1.apply(ParamsAndFeaturesReadable.scala:30)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$class.com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead(ParamsAndFeaturesReadable.scala:30)
	at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read$1.apply(ParamsAndFeaturesReadable.scala:41)
	at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read$1.apply(ParamsAndFeaturesReadable.scala:41)
	at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:19)
	at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:8)
	at org.apache.spark.ml.util.DefaultParamsReader$.loadParamsInstance(ReadWrite.scala:652)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$4.apply(Pipeline.scala:274)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$4.apply(Pipeline.scala:272)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$.load(Pipeline.scala:272)
	at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:348)
	at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:342)
	at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:397)
	at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:391)
	at com.johnsnowlabs.nlp.pretrained.PretrainedPipeline.<init>(PretrainedPipeline.scala:65)
	at DIRECTORY.pipeline.step.MLStep$.classifyActivityType(MLStep.scala:76)
	at DIRECTORY.pipeline.step.MLStep$.classifyActivityAndWorkType(MLStep.scala:33)
	at DIRECTORY.pipeline.Pipeline$.runStep2ML(Pipeline.scala:49)
	at DIRECTORY.Main$.raw2end(Main.scala:118)
	at DIRECTORY.Main$.main(Main.scala:88)
	at DIRECTORY.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsatisfiedLinkError: no tensorflow_cc in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1875)
	at java.lang.Runtime.loadLibrary0(Runtime.java:872)
	at java.lang.System.loadLibrary(System.java:1124)
	at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1683)
	at org.bytedeco.javacpp.Loader.load(Loader.java:1227)
	... 61 more

0 replies

pabshazon · 2021-09-16T20:12:17Z

pabshazon
Sep 16, 2021

Same problem here, any idea how to make it work?

0 replies

aizazulhaq · 2021-09-17T01:20:29Z

aizazulhaq
Sep 17, 2021

Hey @maziyarpanahi, any idea about this?

0 replies

DevinTDHa · 2021-10-20T11:03:19Z

DevinTDHa
Oct 20, 2021
Collaborator

Hi Everyone,

seems like currently it's almost impossible to get Tensorflow working on M1 Macs (which is a requirement for Spark NLP):

tensorflow/java#252 (comment)

We might have better luck, once macOS Monterey is released (25th October). In that update, Tensorflow will be available as tensorflow-metal, which is a version released by Apple (even with GPU acceleration).

I will try again to get it running, once the new version is released.

In the meantime we recommend using Spark NLP on Google Colab, that way the installation is simple and you can use the library without any hassle 👍

4 replies

kgoderis Nov 5, 2021

@DevinTDHa Any update on this?

DevinTDHa Nov 8, 2021
Collaborator

@DevinTDHa Any update on this?

Hi @kgoderis,
I have seen that you have commented over at the tensorflow-java repository as well. Similarly, I was trying to compile tf and tf-java from source on my M1 MacBook Air but I am also encountering some issues in the build process. I am keeping this thread updated with any successes I might have.

kgoderis Nov 8, 2021

It is really a pain. When you refer to using Spark NLP on Colab, do you mean the python version, or the java version ?

maziyarpanahi Nov 8, 2021
Maintainer Author

Spark NLP is built on top of Apache Spark and TensorFlow on Java. We have to obey their compatibilities regarding dependencies such as Java, Python, Operating Systems, gcc, etc.

At this moment, it's not possible to use TensorFlow in Java on any M1 processors. The only workaround is to use Google Colab to use Spark NLP (everything here has a Colab button which is as easy as pip install pyspark==3.1.2 spark-nlp: https://github.com/JohnSnowLabs/spark-nlp-workshop)

mwunderlich · 2021-11-29T11:10:30Z

mwunderlich
Nov 29, 2021

I have been able to get SparkNLP 3.x.x working on an M1 Macbook. Initially, I tried with a Anaconda environment, but this didn't work. Then, I switched to Miniforge which got it working.
For detailed instructions, see this post here: https://fathinah.medium.com/installing-data-science-libraries-in-mac-m1-using-miniforge-2e7378d73e9c

7 replies

DevinTDHa Nov 29, 2021
Collaborator

@mwunderlich Sorry about that, I actually meant miniforge.

I had installed the Apple recommended version from here
https://developer.apple.com/metal/tensorflow-plugin/

I have also installed the homebrew version that you have linked and it seems to have the same functionality. For example, running

conda install -c apple tensorflow-deps
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal

for the arm64 miniforge environment, tensorflow is able to list the M1 GPU as an available device:

>>> import tensorflow as tf
>>> import platform
>>> platform.platform()
'macOS-12.0.1-arm64-arm-64bit'
>>> tf.config.list_physical_devices()
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'),
 PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

I am just wondering, if you are able to use tensorflow within Spark NLP.

Running the pipe above, even after importing tensorflow and listing the devices will result in an error for me.
Could you try to run it and see what happens?
If you are able to run it, then that would be very important for me.

Thanks in advance.

aizazulhaq Dec 17, 2021

@mwunderlich Did you do this using Mac OS 12 or 11?

mwunderlich Dec 20, 2021

@aizazulhaq That was on OS 11.6 Big sur.

hpareek1996 May 11, 2022

@DevinTDHa So were you able to fix this error. If yes, please let me know.

maziyarpanahi May 11, 2022
Maintainer Author

@hpareek1996 This error cannot be fixed and it's also not a bug. Unfortunately, TensorFlow for Java doesn't support M1. (there are no CD/CI features on Github for M1 so they cannot build/test/release) - to be fair, even TensorFlow for Python doesn't support it, it's an entirely different package, but still gets the job done.

We are in the process of supporting M1 in the next major release (5-6 weeks timeframe). Still, even then, our own team built it and continue to maintain it until the official team behind TensorFlow for Java starts supporting M1.

maziyarpanahi · 2022-06-19T13:09:40Z

maziyarpanahi
Jun 19, 2022
Maintainer Author

The Apple silicon M1/M2 chips should be supported in Spark NLP 4.0.0 release: https://github.com/JohnSnowLabs/spark-nlp/releases/tag/4.0.0

1 reply

rwoodard-prog Nov 12, 2022

This does not seem to be working as indicated in the documentation. I created an issue for it. #13079

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Spark NLP on Apple new M1 Chip #2282

{{title}}

Replies: 6 comments 12 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Spark NLP on Apple new M1 Chip #2282

maziyarpanahi Feb 12, 2021 Maintainer

Replies: 6 comments · 12 replies

DevinTDHa Oct 20, 2021 Collaborator

DevinTDHa Nov 8, 2021 Collaborator

maziyarpanahi Nov 8, 2021 Maintainer Author

DevinTDHa Nov 29, 2021 Collaborator

maziyarpanahi May 11, 2022 Maintainer Author

maziyarpanahi Jun 19, 2022 Maintainer Author

maziyarpanahi
Feb 12, 2021
Maintainer

Replies: 6 comments 12 replies

DevinTDHa
Oct 20, 2021
Collaborator

DevinTDHa Nov 8, 2021
Collaborator

maziyarpanahi Nov 8, 2021
Maintainer Author

DevinTDHa Nov 29, 2021
Collaborator

maziyarpanahi May 11, 2022
Maintainer Author

maziyarpanahi
Jun 19, 2022
Maintainer Author