User Defined Configuration in Spark

Call the User defined configuration file as below,

def getConf(confName: String) = {
val conf = new Configuration()
conf.addResource(getClass.getClassLoader.getResource(confName))
conf
}

//Example Use Case
val sparkConf = new SparkConf().setAppName(conf.get(“appName”)).setMaster(conf.get(“locality”))
//memory tuning
sparkConf.set(“spark.executor.memory”, conf.get(“spark.executor.memory”))
//spark.executor.memory is defined in the user defined configuration files.