Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1686

ClassNotFoundException for custom format classes provided in libjars

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.22.0
    • Component/s: contrib/streaming, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The StreamUtil::goodClassOrNull method assumes user-provided classes have package names and if not, they are part of the Hadoop Streaming package. For example, using custom InputFormat or OutputFormat classes without package names will fail with a ClassNotFound exception which is not indicative given the classes are provided in the libjars option. Admittedly, most Java packages should have a package name so this should rarely come up.

      Possible resolution options:

      1) modify the error message to include the actual classname that was attempted in the goodClassOrNull method
      2) call the Configuration::getClassByName method first and if class not found check for default package name and try the call again

          public static Class goodClassOrNull(Configuration conf, String className, String defaultPackage) {
              Class clazz = null;
              try {
                  clazz = conf.getClassByName(className);
              } catch (ClassNotFoundException cnf) {
              }
              if (clazz == null) {
                  if (className.indexOf('.') == -1 && defaultPackage != null) {
                      className = defaultPackage + "." + className;
                      try {
                          clazz = conf.getClassByName(className);
                      } catch (ClassNotFoundException cnf) {
                      }
                  }
              }
              return clazz;
          }
      

        Attachments

        1. HADOOP-1686-2.patch
          5 kB
          Paul Burkhardt
        2. HADOOP-1686-1.patch
          5 kB
          Paul Burkhardt
        3. HADOOP-1686.patch
          5 kB
          Paul Burkhardt

          Activity

            People

            • Assignee:
              pburkhardt Paul Burkhardt
              Reporter:
              pburkhardt Paul Burkhardt
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: