Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28119

Cannot read environment variable inside custom property file

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Incomplete
    • Affects Version/s: 2.4.3
    • Fix Version/s: None
    • Component/s: Input/Output
    • Labels:
    • Environment:

      Linux ubuntu 16.10, spark standalone 2.4.3

      Description

      Spark is compiled with commons-configuration version 1.6 due to hadoop-client library dependency

      [INFO] | +- org.apache.hadoop:hadoop-client:jar:2.6.5:provided
      [INFO] | | +- org.apache.hadoop:hadoop-common:jar:2.6.5:provided
      [INFO] | | | +- commons-cli:commons-cli:jar:1.2:provided
      [INFO] | | | +- xmlenc:xmlenc:jar:0.52:provided
      [INFO] | | | +- commons-httpclient:commons-httpclient:jar:3.1:provided
      [INFO] | | | +- commons-io:commons-io:jar:2.4:provided
      [INFO] | | | +- commons-collections:commons-collections:jar:3.2.2:provided
      [INFO] | | | +- commons-configuration:commons-configuration:jar:1.6:provided
      

      Here is my code

      import org.apache.commons.configuration.ConfigurationException;
      import org.apache.commons.configuration.PropertiesConfiguration;
      import org.apache.spark.sql.SparkSession;
      
      public class SparkPropertyTest {
         public static void main(String... args) throws ConfigurationException {
           SparkSession sp = SparkSession.builder().getOrCreate();
           PropertiesConfiguration config = new PropertiesConfiguration();
           String file = sp.sparkContext().getConf().get("spark.files");
           sp.log().warn("Using property file {}", file);
           config.load(file);
           sp.log().warn(config.getString("env.path"));
        }
      }
      

      Here is the added contains to log4j.properties

      env.path=${env:PATH}
      

      If I launch spark job with following vm options

      -Dspark.master=local[2] -Dspark.files=src/main/resources/log4j.properties
      

      I get the result where the environement variable is printed as is

      2019-06-20 07:09:03 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2019-06-20 07:09:05 WARN SparkSession:11 - Using property file src/main/resources/log4j.properties
      2019-06-20 07:09:05 WARN SparkSession:13 - ${env:PATH}
      

      Now I update my pom.xml

      <dependency>
         <groupId>commons-configuration</groupId>
         <artifactId>commons-configuration</artifactId>
         <version>1.10</version>
      </dependency>
      

      So the new result is

      2019-06-20 07:09:40 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2019-06-20 07:09:42 WARN SparkSession:11 - Using property file src/main/resources/log4j.properties
      2019-06-20 07:09:42 WARN SparkSession:13 - /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
      

       

       

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              JSFM MINET J-Sébastien
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: