[OOZIE-3135] Configure log4j2 in SqoopMain

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.0.0b1
• Assignee: Unassigned
• Reporter: Attila Sasvári
• Created: 30/Nov/17 13:29
• Updated: 05/Aug/19 13:26

Description

In Hadoop 3, MAPREDUCE-6983 switched to use slfj4 & log4j2 in org.apache.hadoop.mapreduce.Job (that prints out MR job id-s needed for Oozie). We need to setup log4j accordingly (it is also related to HADOOP-12956).
Without proper configuration in the Sqoop action, we won't be able to get external job id-s (SqoopActionExecutor unit tests and real action would be also affected).

The API for Log4j 2 is not compatible with Log4j 1.x, but we will need to support both hadoop 2 and hadoop 3 profiles for a while.

We could use reflection to determine the type of the logger object in org.apache.hadoop.mapreduce.Job and configure log4j settings based on it, but there might be a better way.

For example we could do something like this:

• add a new method for configuring log4j2:

      private String setUpSqoopLog4J2(final String rootLogLevel) throws IOException {
          System.out.println("Setting up log4j2");
  
          final String logFile = getSqoopLogFile();
          final File log4j2Xml = new File(SQOOP_LOG4J2_XML);
          try (Writer writer = new FileWriter(log4j2Xml))
          {
              final String logj2SettingsXml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
                      "<Configuration status=\"WARN\">\n" +
                      "    <Appenders>\n" +
                      "        <Console name=\"Console\" target=\"SYSTEM_OUT\">\n" +
                      "            <PatternLayout pattern=\"%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n\"/>\n" +
                      "        </Console>\n" +
                      "        <File name=\"File\" fileName=\"" + logFile + "\">  \n" +
                      "            <PatternLayout pattern=\"%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n\"/>\n" +
                      "        </File> \n" +
                      "    </Appenders>\n" +
                      "    <Loggers>\n" +
                      "        <Root level=\"" + rootLogLevel.toLowerCase() + "\">\n" +
                      "            <AppenderRef ref=\"Console\"/>\n" +
                      "            <AppenderRef ref=\"File\"/>\n" +
                      "        </Root>\n" +
                      "    </Loggers>\n" +
                      "</Configuration>";
              writer.write(logj2SettingsXml);
          }
  
          System.out.printf("log4j2 configuration file created at %s%n", log4j2Xml.getAbsolutePath());
  
          final   LoggerContext context = (LoggerContext) LogManager.getContext(false);
          context.setConfigLocation(log4j2Xml.toURI()); // forces log4j2 reconfiguration
          return logFile;
      }
  

  and call it in the run() method if the mapreduce client is using slf4j for logging:

          String logFile;
          // MAPREDUCE-6983 switches to slfj4 & log4j2. Need to setup log4j accordingly
          if (org.apache.hadoop.mapreduce.Job.class.getDeclaredField("LOG").getType().
                  isAssignableFrom(org.slf4j.Logger.class)) {
              logFile = setUpSqoopLog4J2(rootLogLevel);
          }
          else {
              logFile = setUpSqoopLog4J(rootLogLevel, logLevel);
          }
  

Issue Links

• is related to: OOZIE-3136 Upgrade from Log4j 1.x to 2.x
• is related to: OOZIE-2973 Make sure Oozie works with Hadoop 3