Pig
  1. Pig
  2. PIG-3135

HExecutionEngine should look for resources in user passed Properties

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Looking at this snippet:

          private void init(Properties properties) throws ExecException {
                .
                .
                .
      
                  // Check existence of hadoop-site.xml or core-site.xml
                  Configuration testConf = new Configuration();
                  ClassLoader cl = testConf.getClassLoader();
                  URL hadoop_site = cl.getResource( HADOOP_SITE );
                  URL core_site = cl.getResource( CORE_SITE );
                 
                  if( hadoop_site == null && core_site == null ) {
                      throw new ExecException("Cannot find hadoop configurations in classpath (neither hadoop-site.xml nor core-site.xml was found in the classpath)." +
                              " If you plan to use local mode, please put -x local option in command line",
                              4010);
                  }
      

      This assumes the resources (*-site.xml) are set on the classpath, but this will not always be the case when run with Pig's Java APIs. One could want to programatically set the resources and the code here should additionally check if they are available in there.

      Example: When a Configuration object is created and resources are added before passing it on to Pig.

      Configuration conf = new Configuration(false);
      conf.addResource("foo/core-site.xml");
      conf.addResource("bar/hadoop-site.xml");
      
      PigServer pServer = new PigServer(ExecType.MAPREDUCE, conf);
      

      The above conf is not used right now to obtain resources.

      1. PIG-3135_1.patch
        6 kB
        Prashant Kommireddi
      2. PIG-3135.patch
        5 kB
        Prashant Kommireddi

        Activity

        Prashant Kommireddi created issue -
        Prashant Kommireddi made changes -
        Field Original Value New Value
        Assignee Prashant Kommireddi [ prkommireddi ]
        Prashant Kommireddi made changes -
        Attachment PIG-3135.patch [ 12566436 ]
        Prashant Kommireddi made changes -
        Patch Info Patch Available [ 10042 ]
        Prashant Kommireddi made changes -
        Attachment PIG-3135_1.patch [ 12567801 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.12 [ 12323380 ]
        Resolution Fixed [ 1 ]
        Cheolsoo Park made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Cheolsoo Park made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Prashant Kommireddi
            Reporter:
            Prashant Kommireddi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development