Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1686

ClassNotFoundException for custom format classes provided in libjars

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.22.0
    • Component/s: contrib/streaming, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The StreamUtil::goodClassOrNull method assumes user-provided classes have package names and if not, they are part of the Hadoop Streaming package. For example, using custom InputFormat or OutputFormat classes without package names will fail with a ClassNotFound exception which is not indicative given the classes are provided in the libjars option. Admittedly, most Java packages should have a package name so this should rarely come up.

      Possible resolution options:

      1) modify the error message to include the actual classname that was attempted in the goodClassOrNull method
      2) call the Configuration::getClassByName method first and if class not found check for default package name and try the call again

          public static Class goodClassOrNull(Configuration conf, String className, String defaultPackage) {
              Class clazz = null;
              try {
                  clazz = conf.getClassByName(className);
              } catch (ClassNotFoundException cnf) {
              }
              if (clazz == null) {
                  if (className.indexOf('.') == -1 && defaultPackage != null) {
                      className = defaultPackage + "." + className;
                      try {
                          clazz = conf.getClassByName(className);
                      } catch (ClassNotFoundException cnf) {
                      }
                  }
              }
              return clazz;
          }
      
      1. HADOOP-1686-2.patch
        5 kB
        Paul Burkhardt
      2. HADOOP-1686-1.patch
        5 kB
        Paul Burkhardt
      3. HADOOP-1686.patch
        5 kB
        Paul Burkhardt

        Activity

          People

          • Assignee:
            Paul Burkhardt
            Reporter:
            Paul Burkhardt
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development