Spark shell is great but one of the major issues is the amount of logging it dishes out, it can get frustrating when you are trying to debug things.
Easily solved though.
In your SPARK_HOME/conf directory you’ll find a log4j.properties.template. Make a copy of it.
cp log4j.properties.template log4.properties
Edit log4j.properties with your favourite text editor and change:
When you restart the Spark shell you’ll have a fighting chance of seeing the output.