[SPARK-28119] Cannot read environment variable inside custom property file

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Affects Version/s: 2.4.3
• Component/s: Input/Output
• Labels: bulk-closed

Environment

Linux ubuntu 16.10, spark standalone 2.4.3

• Assignee: Unassigned
• Reporter: MINET J-Sébastien
• Created: 20/Jun/19 07:11
• Updated: 25/May/21 01:51
• Resolved: 25/May/21 01:39

Description

Spark is compiled with commons-configuration version 1.6 due to hadoop-client library dependency

[INFO] | +- org.apache.hadoop:hadoop-client:jar:2.6.5:provided
[INFO] | | +- org.apache.hadoop:hadoop-common:jar:2.6.5:provided
[INFO] | | | +- commons-cli:commons-cli:jar:1.2:provided
[INFO] | | | +- xmlenc:xmlenc:jar:0.52:provided
[INFO] | | | +- commons-httpclient:commons-httpclient:jar:3.1:provided
[INFO] | | | +- commons-io:commons-io:jar:2.4:provided
[INFO] | | | +- commons-collections:commons-collections:jar:3.2.2:provided
[INFO] | | | +- commons-configuration:commons-configuration:jar:1.6:provided

Here is my code

import org.apache.commons.configuration.ConfigurationException;
import org.apache.commons.configuration.PropertiesConfiguration;
import org.apache.spark.sql.SparkSession;

public class SparkPropertyTest {
   public static void main(String... args) throws ConfigurationException {
     SparkSession sp = SparkSession.builder().getOrCreate();
     PropertiesConfiguration config = new PropertiesConfiguration();
     String file = sp.sparkContext().getConf().get("spark.files");
     sp.log().warn("Using property file {}", file);
     config.load(file);
     sp.log().warn(config.getString("env.path"));
  }
}

Here is the added contains to log4j.properties

env.path=${env:PATH}

If I launch spark job with following vm options

-Dspark.master=local[2] -Dspark.files=src/main/resources/log4j.properties

I get the result where the environement variable is printed as is

2019-06-20 07:09:03 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-06-20 07:09:05 WARN SparkSession:11 - Using property file src/main/resources/log4j.properties
2019-06-20 07:09:05 WARN SparkSession:13 - ${env:PATH}

Now I update my pom.xml

<dependency>
   <groupId>commons-configuration</groupId>
   <artifactId>commons-configuration</artifactId>
   <version>1.10</version>
</dependency>

So the new result is

2019-06-20 07:09:40 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-06-20 07:09:42 WARN SparkSession:11 - Using property file src/main/resources/log4j.properties
2019-06-20 07:09:42 WARN SparkSession:13 - /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games