Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stocator fails with Spark Streaming #271

Open
VidyasagarMSC opened this issue Mar 2, 2021 · 1 comment
Open

Stocator fails with Spark Streaming #271

VidyasagarMSC opened this issue Mar 2, 2021 · 1 comment

Comments

@VidyasagarMSC
Copy link

VidyasagarMSC commented Mar 2, 2021

I tried reading and writing objects to COS using Stocator with Spark streaming but it fails with the below error

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.ibm.stocator.fs.ObjectStoreFileSystem not found

Here's what I tried to overcome the error and failed

  1. Built Stocator jar file and moved the jar to /opt/spark/jars with COS access key and secret key. Fails with jackson.databinding not found exception
  2. Built SDK jar file and used the IAM api key Fails with jackson.databinding not found exception

This is what worked as an alternative to Stocator
I have to fall back to S3A by moving the required jar files to /opt/spark/jars folder and use s3a:// instead of cos://

@mrmadira
Copy link
Collaborator

Have you tried using the ready jar from https://mvnrepository.com/artifact/com.ibm.stocator/stocator ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants