Hadoop Common
  1. Hadoop Common
  2. HADOOP-7056

NN unintentionally reads mapred-site.xml

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 0.22.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It seems that mapred-site.xml is now unintentionally being read by the NN, and probably all the daemons.

      This is because the Groups constructor calls ReflectionUtils.newInstance(...), which in turn calls ReflectionUtils.setConf(...), which calls ReflectionUtils.setJobConf(...), which creates an instance of o.a.h.mapred.JobConf, which calls o.a.h.mapreduce.util.ConfigUtil.loadResources(...), which reads in mapred-site.xml and mapred-default.xml. Phew!

      I suspect the appropriate solution is for ReflectionUtils.setConf(...) to be changed to not call (the mapred-specific) ReflectionUtils.setJobConf(...), but there's a comment in there which says it's being left in place for backward compatibility. Can somebody with more knowledge of that situation please comment?

        Activity

        Hide
        Masatake Iwasaki added a comment -

        I saw this on the DataNode of recent trunk too. The cause was the same.

            [1] org.apache.hadoop.util.ReflectionUtils.setJobConf (ReflectionUtils.java:91)                                                                                                              
            [2] org.apache.hadoop.util.ReflectionUtils.setConf (ReflectionUtils.java:75)                                                                                                                 
            [3] org.apache.hadoop.util.ReflectionUtils.newInstance (ReflectionUtils.java:133)                                                                                                            
            [4] org.apache.hadoop.security.Groups.<init> (Groups.java:64)                                                                                                                                
            [5] org.apache.hadoop.security.Groups.getUserToGroupsMappingService (Groups.java:240)                                                                                                        
            [6] org.apache.hadoop.security.UserGroupInformation.initialize (UserGroupInformation.java:266)                                                                                               
            [7] org.apache.hadoop.security.UserGroupInformation.setConfiguration (UserGroupInformation.java:294)                                                                                         
            [8] org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode (DataNode.java:1,770)                                                                                                
            [9] org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode (DataNode.java:1,813)                                                                                                     
            [10] org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain (DataNode.java:1,990)                                                                                                        
            [11] org.apache.hadoop.hdfs.server.datanode.DataNode.main (DataNode.java:2,014)                                                                                                              
        
        Show
        Masatake Iwasaki added a comment - I saw this on the DataNode of recent trunk too. The cause was the same. [1] org.apache.hadoop.util.ReflectionUtils.setJobConf (ReflectionUtils.java:91) [2] org.apache.hadoop.util.ReflectionUtils.setConf (ReflectionUtils.java:75) [3] org.apache.hadoop.util.ReflectionUtils.newInstance (ReflectionUtils.java:133) [4] org.apache.hadoop.security.Groups.<init> (Groups.java:64) [5] org.apache.hadoop.security.Groups.getUserToGroupsMappingService (Groups.java:240) [6] org.apache.hadoop.security.UserGroupInformation.initialize (UserGroupInformation.java:266) [7] org.apache.hadoop.security.UserGroupInformation.setConfiguration (UserGroupInformation.java:294) [8] org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode (DataNode.java:1,770) [9] org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode (DataNode.java:1,813) [10] org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain (DataNode.java:1,990) [11] org.apache.hadoop.hdfs.server.datanode.DataNode.main (DataNode.java:2,014)
        Hide
        Masatake Iwasaki added a comment -

        just for noting. o.a.h.mapreduce.util.ConfigUtil#loadResources which is called in JobConf loads not only mapred-{default,site}.xml but also yarn-{default,site}.xml.

          public static void loadResources() {
            addDeprecatedKeys();
            Configuration.addDefaultResource("mapred-default.xml");
            Configuration.addDefaultResource("mapred-site.xml");
            Configuration.addDefaultResource("yarn-default.xml");
            Configuration.addDefaultResource("yarn-site.xml");
          }
        
        Show
        Masatake Iwasaki added a comment - just for noting. o.a.h.mapreduce.util.ConfigUtil#loadResources which is called in JobConf loads not only mapred-{default,site}.xml but also yarn-{default,site}.xml. public static void loadResources() { addDeprecatedKeys(); Configuration.addDefaultResource( "mapred- default .xml" ); Configuration.addDefaultResource( "mapred-site.xml" ); Configuration.addDefaultResource( "yarn- default .xml" ); Configuration.addDefaultResource( "yarn-site.xml" ); }

          People

          • Assignee:
            Unassigned
            Reporter:
            Aaron T. Myers
          • Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

            • Created:
              Updated:

              Development