Printing the spark, hadoop, system configs


I changed the Google storage connector write buffer size when starting up an experimental Dataproc cluster with

  --properties ''

and found these commands (modified from a SO post) useful for verifying such changes. You’ll need access to hc:

from import Env

To print the Spark config:


To print the Hadoop config:

hadoopConf = {}
iterator = Env().hc().sc._jsc.hadoopConfiguration().iterator()
while iterator.hasNext():
    prop =
    hadoopConf[prop.getKey()] = prop.getValue()
for item in sorted(hadoopConf.items()): print(item)

To print system properties:

import os
for item in sorted(os.environ.items()): print(item)

GCP defaults are here: