Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1859

sorter extension depends on hadoop but does not declare as such in repository artifact

    XMLWordPrintableJSON

    Details

      Description

      When SortValues is used via org.apache.beam:beam-sdks-java-extensions-sorter:0.6.0, this exception is raised:

      Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
      	at org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
      	at org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      	at org.apache.beam.sdk.extensions.sorter.BufferedExternalSorter.create(BufferedExternalSorter.java:98)
      	at org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn.processElement(SortValues.java:153)
      	at org.apache.beam.sdk.extensions.sorter.SortValues$SortValuesDoFn$auxiliary$uK25yOmK.invokeProcessElement(Unknown Source)
      	at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:198)
      	at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:159)
      	at org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElement(PushbackSideInputDoFnRunner.java:111)
      	at org.apache.beam.runners.core.PushbackSideInputDoFnRunner.processElementInReadyWindows(PushbackSideInputDoFnRunner.java:77)
      	at org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:134)
      	at org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:51)
      	at org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
      	at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      I think the issue is that beam-sdks-java-extensions-sorter should declare that it depends on that hadoop library but does not?

        Attachments

          Activity

            People

            • Assignee:
              davor Davor Bonaci
              Reporter:
              wtanaka Wesley Tanaka
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: