toDF() isn't working on the shell

[rush4ratio]

I get the same error (see attached) when trying orgs.toDF().show() or memberships.select_fields(['organization_id']).toDF().distinct().show()

[mohitsax]

Thanks for using AWS Glue.

Please refer to the step 5 in AWS Glue documentation on using a REPL shell at: http://docs.aws.amazon.com/glue/latest/dg/tutorial-development-endpoint-repl.html
The solution to resolve this error is as follows – you would have to stop the existing SparkContext and create a new one using GlueContext.
spark.stop()
glueContext = GlueContext(SparkContext.getOrCreate())

If you have further questions, you can also use the AWS Glue Forum: https://forums.aws.amazon.com/forum.jspa?forumID=262

[rush4ratio]

Thanks for the suggestion. I've tried it but unfortunately I still get the same error.

[mohitsax]

Thanks for trying out the fix. We were not able to reproduce the error on a REPL shell after using the above fix.
Could you please open up a support ticket.

[Sergeant007]

The fix with spark.stop() worked for me. Let me also post the exact error message here for better indexing by search engines:

Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /home/glue/metastore_db.

[yupinh]

One workaround would be to disable hive support when sparkContext is initialized.

newconf = sc._conf.set("spark.sql.catalogImplementation", "in-memory")
sc.stop()
sc = sc.getOrCreate(newconf)

Let me know if this causes you additional problems.

[laurikoobas]

The spark.stop() "fix" worked for me as well.
The specific error message was:

ERROR Schema: Failed initialising database.
Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Failed to start database 'metastore_db'

[jgoeglein]

I ran into this as well. Why isn't the development environment setup to support this from the beginning? There's lots of glue documentation out there using .toDF() that doesn't work out of the box (the first example in https://github.com/aws-samples/aws-glue-samples/blob/master/FAQ_and_How_to.md for example)

[zalmane]

Just ran into this. The above did not work for me.
Ended up starting pyspark with the following flag:
./bin/gluepyspark --conf spark.sql.catalogImplementation=in-memory

[davehowell]

apache/spark@ac9c053 this is a spark patch that should fix this in the Spark 3.0.0 release