Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1686

ClassNotFoundException for custom format classes provided in libjars

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.22.0
    • Component/s: contrib/streaming, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The StreamUtil::goodClassOrNull method assumes user-provided classes have package names and if not, they are part of the Hadoop Streaming package. For example, using custom InputFormat or OutputFormat classes without package names will fail with a ClassNotFound exception which is not indicative given the classes are provided in the libjars option. Admittedly, most Java packages should have a package name so this should rarely come up.

      Possible resolution options:

      1) modify the error message to include the actual classname that was attempted in the goodClassOrNull method
      2) call the Configuration::getClassByName method first and if class not found check for default package name and try the call again

          public static Class goodClassOrNull(Configuration conf, String className, String defaultPackage) {
              Class clazz = null;
              try {
                  clazz = conf.getClassByName(className);
              } catch (ClassNotFoundException cnf) {
              }
              if (clazz == null) {
                  if (className.indexOf('.') == -1 && defaultPackage != null) {
                      className = defaultPackage + "." + className;
                      try {
                          clazz = conf.getClassByName(className);
                      } catch (ClassNotFoundException cnf) {
                      }
                  }
              }
              return clazz;
          }
      
      1. HADOOP-1686-2.patch
        5 kB
        Paul Burkhardt
      2. HADOOP-1686-1.patch
        5 kB
        Paul Burkhardt
      3. HADOOP-1686.patch
        5 kB
        Paul Burkhardt

        Activity

        Paul Burkhardt created issue -
        Paul Burkhardt made changes -
        Field Original Value New Value
        Priority Trivial [ 5 ] Minor [ 4 ]
        Paul Burkhardt made changes -
        Attachment HADOOP-1686.patch [ 12450189 ]
        Paul Burkhardt made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 0.20.1 [ 12314047 ]
        Affects Version/s 0.20.2 [ 12314205 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Paul Burkhardt made changes -
        Attachment HADOOP-1686-1.patch [ 12450710 ]
        Paul Burkhardt made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Paul Burkhardt made changes -
        Attachment HADOOP-1686-2.patch [ 12450844 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314184 ]
        Resolution Fixed [ 1 ]
        Tsz Wo Nicholas Sze made changes -
        Assignee Paul Burkhardt [ pburkhardt ]
        Component/s test [ 12312904 ]

          People

          • Assignee:
            Paul Burkhardt
            Reporter:
            Paul Burkhardt
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development