Spark shell is great but one of the major issues is the amount of logging it dishes out, it can get frustrating when you are trying to debug things.

Easily solved though.

In your SPARK_HOME/conf directory you’ll find a Make a copy of it.


Edit with your favourite text editor and change:

log4j.rootCategory=INFO, console


log4j.rootCategory=WARN, console

When you restart the Spark shell you’ll have a fighting chance of seeing the output.