Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.13.0
    • Labels:
      None
    • Environment:

      The command line drivers spark-itemsimilarity and spark-naivebayes using a remote or pseudo-clustered Spark

    • Sprint:
      Jan/Feb-2017

      Description

      Missing classes when running these jobs because the dependencies-reduced jar, passed to Spark for serialization purposes, does not contain all needed classes.

      Found by a user.

        Issue Links

          Activity

          Hide
          pferrel Pat Ferrel added a comment -

          The jar isn't supposed to have all deps, only the ones not provided by the environment. In fact it is supposed to have the minimum.

          So it appears some of the provided classes for previous platforms (Spark etc) have change in new versions? We then need to add to the dependency reduced jar but first check to see if a newer version of some provided dep will fill the bill or dependency-reduced will bloat needlessly.

          What specifically is the error, what is missing.

          Show
          pferrel Pat Ferrel added a comment - The jar isn't supposed to have all deps, only the ones not provided by the environment. In fact it is supposed to have the minimum. So it appears some of the provided classes for previous platforms (Spark etc) have change in new versions? We then need to add to the dependency reduced jar but first check to see if a newer version of some provided dep will fill the bill or dependency-reduced will bloat needlessly. What specifically is the error, what is missing.
          Hide
          pferrel Pat Ferrel added a comment -

          Test thoroughly, not sure of side effects of the fix

          Show
          pferrel Pat Ferrel added a comment - Test thoroughly, not sure of side effects of the fix
          Hide
          pferrel Pat Ferrel added a comment -

          Oops misnamed the commit message for MAHOUT-1950. The fix is in master, unit tested, and driver integration tested on remote spark and HDFS.

          Just removed line 83 in mahout/spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala

          Needs to be tested thoroughly since I have no idea of the ramifications of removing the line. See Andrew Palumbo who added it but can't recall the reason either.

          Cross your fingers.

          Show
          pferrel Pat Ferrel added a comment - Oops misnamed the commit message for MAHOUT-1950 . The fix is in master, unit tested, and driver integration tested on remote spark and HDFS. Just removed line 83 in mahout/spark/src/main/scala/org/apache/mahout/sparkbindings/package.scala Needs to be tested thoroughly since I have no idea of the ramifications of removing the line. See Andrew Palumbo who added it but can't recall the reason either. Cross your fingers.
          Hide
          pferrel Pat Ferrel added a comment -

          Andrew Palumbo Suneel Marthi There seems to be some question about who made the commit but I'm sure it wasn't me. I have no idea what is causing this, as I said, and the only thing suspicious in it (the one before works BTW) is the mahout jars line change in the Spark module. the rest of the changes are in Flink afaict.

          Show
          pferrel Pat Ferrel added a comment - Andrew Palumbo Suneel Marthi There seems to be some question about who made the commit but I'm sure it wasn't me. I have no idea what is causing this, as I said, and the only thing suspicious in it (the one before works BTW) is the mahout jars line change in the Spark module. the rest of the changes are in Flink afaict.
          Hide
          pferrel Pat Ferrel added a comment -

          A quick way to test this is:

          1) get Spark and HDFS running locally in pseudo-cluster mode.
          2) build the version of Mahout under test I use simply "mvn clean install -DskipTests"
          3) "hdfs dfs -rm -r test-results" to removes any old results
          4) run the script below and look for exceptions in the output, they will look like the above errors

          #!/usr/bin/env bash
          #begin script
          mahout spark-itemsimilarity \
          --input test.csv \
          --output test-result \
          --master spark://Maclaurin.local:7077 \
          --filter1 purchase \
          --filter2 view \
          --itemIDColumn 2 \
          --rowIDColumn 0 \
          --filterColumn 1
          #end-script

          test.csv file for the script

          u1,purchase,iphone
          u1,purchase,ipad
          u2,purchase,nexus
          u2,purchase,galaxy
          u3,purchase,surface
          u4,purchase,iphone
          u4,purchase,galaxy
          u1,view,iphone
          u1,view,ipad
          u1,view,nexus
          u1,view,galaxy
          u2,view,iphone
          u2,view,ipad
          u2,view,nexus
          u2,view,galaxy
          u3,view,surface
          u3,view,nexus
          u4,view,iphone
          u4,view,ipad

          Show
          pferrel Pat Ferrel added a comment - A quick way to test this is: 1) get Spark and HDFS running locally in pseudo-cluster mode. 2) build the version of Mahout under test I use simply "mvn clean install -DskipTests" 3) "hdfs dfs -rm -r test-results" to removes any old results 4) run the script below and look for exceptions in the output, they will look like the above errors #!/usr/bin/env bash #begin script mahout spark-itemsimilarity \ --input test.csv \ --output test-result \ --master spark://Maclaurin.local:7077 \ --filter1 purchase \ --filter2 view \ --itemIDColumn 2 \ --rowIDColumn 0 \ --filterColumn 1 #end-script test.csv file for the script u1,purchase,iphone u1,purchase,ipad u2,purchase,nexus u2,purchase,galaxy u3,purchase,surface u4,purchase,iphone u4,purchase,galaxy u1,view,iphone u1,view,ipad u1,view,nexus u1,view,galaxy u2,view,iphone u2,view,ipad u2,view,nexus u2,view,galaxy u3,view,surface u3,view,nexus u4,view,iphone u4,view,ipad
          Hide
          pferrel Pat Ferrel added a comment -

          scratch that PR. We do not have a fix for this but I have narrowed down the commit where it first starts to occur.

          in Mahout 0.12.2 the drivers work with remote Spark

          They work in all commits until https://github.com/apache/mahout/commit/8e0e8b5572e0d24c1930ed60fec6d02693b41575 which would say that something in this commit broke things. This is mainly Flink but there is a change to how mahout jars are packaged and the error is shown below. The error wording is a bit mysterious, it seem to be missing MahoutKryoRegistrator but could also be from a class that cannot be serialized, really not sure.

          Exception in thread "main" 17/03/06 18:15:04 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
          org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, 192.168.0.6): java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo
          at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212)
          at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
          at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
          at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
          at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
          at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
          at org.apache.spark.scheduler.Task.run(Task.scala:89)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          Caused by: org.apache.spark.SparkException: Failed to register classes with Kryo
          at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:128)
          at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:273)
          at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:258)
          at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:174)
          at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:215)
          at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
          at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1205)
          ... 11 more
          Caused by: java.lang.ClassNotFoundException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
          at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:348)
          at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
          at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
          at scala.Option.map(Option.scala:145)
          at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123)
          ... 17 more
          Driver stacktrace:
          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
          at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
          at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
          at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
          at scala.Option.foreach(Option.scala:236)
          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
          at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
          at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
          at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
          at org.apache.spark.rdd.RDD.collect(RDD.scala:926)
          at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:81)
          at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:319)
          at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:319)
          at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75)
          at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:319)
          at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382)
          at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39)
          at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372)
          at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152)
          at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201)
          at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112)
          at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110)
          at scala.Option.map(Option.scala:145)
          at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110)
          at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
          at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
          at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
          Caused by: java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo
          at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212)
          at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
          at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
          at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
          at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
          at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
          at org.apache.spark.scheduler.Task.run(Task.scala:89)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          Caused by: org.apache.spark.SparkException: Failed to register classes with Kryo
          at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:128)
          at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:273)
          at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:258)
          at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:174)
          at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:215)
          at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
          at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1205)
          ... 11 more
          Caused by: java.lang.ClassNotFoundException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
          at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:348)
          at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
          at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
          at scala.Option.map(Option.scala:145)
          at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123)
          ... 17 more

          Show
          pferrel Pat Ferrel added a comment - scratch that PR. We do not have a fix for this but I have narrowed down the commit where it first starts to occur. in Mahout 0.12.2 the drivers work with remote Spark They work in all commits until https://github.com/apache/mahout/commit/8e0e8b5572e0d24c1930ed60fec6d02693b41575 which would say that something in this commit broke things. This is mainly Flink but there is a change to how mahout jars are packaged and the error is shown below. The error wording is a bit mysterious, it seem to be missing MahoutKryoRegistrator but could also be from a class that cannot be serialized, really not sure. Exception in thread "main" 17/03/06 18:15:04 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, 192.168.0.6): java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88) at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.spark.SparkException: Failed to register classes with Kryo at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:128) at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:273) at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:258) at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:174) at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:215) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1205) ... 11 more Caused by: java.lang.ClassNotFoundException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at scala.Option.map(Option.scala:145) at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123) ... 17 more Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929) at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.RDD.collect(RDD.scala:926) at org.apache.mahout.drivers.TDIndexedDatasetReader$class.elementReader(TextDelimitedReaderWriter.scala:81) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:319) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.elementReader(TextDelimitedReaderWriter.scala:319) at org.apache.mahout.math.indexeddataset.Reader$class.readElementsFrom(ReaderWriter.scala:75) at org.apache.mahout.drivers.TextDelimitedIndexedDatasetReader.readElementsFrom(TextDelimitedReaderWriter.scala:319) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:382) at org.apache.mahout.sparkbindings.SparkEngine$.indexedDatasetDFSReadElements(SparkEngine.scala:39) at org.apache.mahout.math.indexeddataset.package$.indexedDatasetDFSReadElements(package.scala:372) at org.apache.mahout.drivers.ItemSimilarityDriver$.readIndexedDatasets(ItemSimilarityDriver.scala:152) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:201) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:112) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:110) at scala.Option.map(Option.scala:145) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:110) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88) at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) at org.apache.spark.scheduler.Task.run(Task.scala:89) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.spark.SparkException: Failed to register classes with Kryo at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:128) at org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:273) at org.apache.spark.serializer.KryoSerializerInstance.<init>(KryoSerializer.scala:258) at org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:174) at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:215) at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178) at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1205) ... 11 more Caused by: java.lang.ClassNotFoundException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at scala.Option.map(Option.scala:145) at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123) ... 17 more
          Hide
          pferrel Pat Ferrel added a comment -
          Show
          pferrel Pat Ferrel added a comment - fix being tested in https://github.com/apache/mahout/pull/292
          Hide
          pferrel Pat Ferrel added a comment -

          Trevor Grant added the use of spark-submit to the Mahout script for launching the drivers, this potentially has some side effects since much of the work of spark-submit was done in the drivers and I am not sure if there is a way to pass params through to spark-submit. In other words the driver may no permit unrecognized params on the command line. Therefor we will leave the drivers as they are, doing more work than they should but mark this as deprecated and remove in a future release.

          Show
          pferrel Pat Ferrel added a comment - Trevor Grant added the use of spark-submit to the Mahout script for launching the drivers, this potentially has some side effects since much of the work of spark-submit was done in the drivers and I am not sure if there is a way to pass params through to spark-submit. In other words the driver may no permit unrecognized params on the command line. Therefor we will leave the drivers as they are, doing more work than they should but mark this as deprecated and remove in a future release.
          Hide
          pferrel Pat Ferrel added a comment -

          User found the following error running the spark-itemsimilarity driver (affect the NB driver too) on a remote Spark master:

          17/03/03 10:08:40 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, reco-master): java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo
          at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212)
          at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
          ...
          Caused by: java.lang.ClassNotFoundException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
          at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          at java.lang.Class.forName0(Native Method)
          at java.lang.Class.forName(Class.java:348)
          at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
          at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
          at scala.Option.map(Option.scala:145)
          at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123)

          When I run the exactly same command on the 0.12.2 release distribution against the same Spark cluster, the
          command completes sucessfully.

          My Environment is:

          Show
          pferrel Pat Ferrel added a comment - User found the following error running the spark-itemsimilarity driver (affect the NB driver too) on a remote Spark master: 17/03/03 10:08:40 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, reco-master): java.io.IOException: org.apache.spark.SparkException: Failed to register classes with Kryo at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) ... Caused by: java.lang.ClassNotFoundException: org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123) at scala.Option.map(Option.scala:145) at org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123) When I run the exactly same command on the 0.12.2 release distribution against the same Spark cluster, the command completes sucessfully. My Environment is: Ubuntu 14.04 Oracle-JDK 1.8.0_121 Spark standalone cluster using this distribution: http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz Mahout 0.13.0-RC: https://repository.apache.org/content/repositories/orgapachemahout-1034/org/apache/mahout/apache-mahout-distribution/0.13.0/apache-mahout-distribution-0.13.0.tar.gz

            People

            • Assignee:
              pferrel Pat Ferrel
              Reporter:
              pferrel Pat Ferrel
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile