Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.11.2
    • Fix Version/s: 1.0.0
    • Component/s: Flink
    • Labels:
      None

      Description

      dals test fails in Flink bindings with an OOM. Numerically the test passes, when the matrix being decomposed in the test lowered to the size 50 x 50. But the default size of the matrix in the DistributedDecompositionsSuiteBase is 500 x 500.

      java.lang.OutOfMemoryError: Java heap space
      	at java.util.Arrays.copyOf(Arrays.java:2271)
      	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
      	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
      	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
      	at java.io.ObjectOutputStream$BlockDataOutputStream.writeBlockHeader(ObjectOutputStream.java:1893)
      	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1874)
      	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1785)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1188)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
      	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
      	at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:300)
      	at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:252)
      	at org.apache.flink.runtime.operators.util.TaskConfig.setStubWrapper(TaskConfig.java:273)
      	at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.createDataSourceVertex(JobGraphGenerator.java:893)
      	at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:286)
      	at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.preVisit(JobGraphGenerator.java:109)
      	at org.apache.flink.optimizer.plan.SourcePlanNode.accept(SourcePlanNode.java:86)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.SingleInputPlanNode.accept(SingleInputPlanNode.java:199)
      	at org.apache.flink.optimizer.plan.OptimizedPlan.accept(OptimizedPlan.java:128)
      	at org.apache.flink.optimizer.plantranslate.JobGraphGenerator.compileJobGraph(JobGraphGenerator.java:188)
      

        Activity

        Hide
        smarthi Suneel Marthi added a comment -

        We don't see the OOM anymore but the below error also from FlinkOpAtA.slim()

        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(1/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(2/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(3/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(4/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(5/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(6/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(7/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(8/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(1/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(2/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(3/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(4/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(5/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(6/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(7/8) switched to CANCELED 
        04/08/2016 03:52:13	CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(8/8) switched to CANCELED 
        04/08/2016 03:52:13	Reduce (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(1/1) switched to CANCELED 
        04/08/2016 03:52:13	DataSink (org.apache.flink.api.java.Utils$CollectHelper@2c663246)(1/1) switched to CANCELED 
        04/08/2016 03:52:13	Job execution switched to status FAILED.
        - dals *** FAILED ***
          org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
          at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:716)
          at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662)
          at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662)
          at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
          at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
          at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
          at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
          at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
          at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
          at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
          ...
          Cause: java.lang.IllegalStateException: unread block data
          at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
          at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
          at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:290)
          at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:248)
          at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:282)
          ...
        
        
        Show
        smarthi Suneel Marthi added a comment - We don't see the OOM anymore but the below error also from FlinkOpAtA.slim() 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(1/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(2/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(3/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(4/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(5/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(6/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(7/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.drm.RowsFlinkDrm.asBlockified(FlinkDrm.scala:52)) -> Map (Map at org.apache.mahout.flinkbindings.blas.FlinkOpMapBlock$.apply(FlinkOpMapBlock.scala:38)) -> FlatMap (FlatMap at org.apache.mahout.flinkbindings.drm.BlockifiedFlinkDrm.asRowWise(FlinkDrm.scala:93))(8/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(1/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(2/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(3/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(4/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(5/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(6/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(7/8) switched to CANCELED 04/08/2016 03:52:13 CHAIN MapPartition (MapPartition at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:54)) -> Combine (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(8/8) switched to CANCELED 04/08/2016 03:52:13 Reduce (Reduce at org.apache.mahout.flinkbindings.blas.FlinkOpAtA$.slim(FlinkOpAtA.scala:101))(1/1) switched to CANCELED 04/08/2016 03:52:13 DataSink (org.apache.flink.api.java.Utils$CollectHelper@2c663246)(1/1) switched to CANCELED 04/08/2016 03:52:13 Job execution switched to status FAILED. - dals *** FAILED *** org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:716) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:662) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346) ... Cause: java.lang.IllegalStateException: unread block data at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2431) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:290) at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:248) at org.apache.flink.runtime.operators.util.TaskConfig.getStubWrapper(TaskConfig.java:282) ...
        Hide
        Andrew_Palumbo Andrew Palumbo added a comment -

        for future reference after tweaking some Flink vars the problem with dals ​is​ a serialization error:

        Process finished with exit code 137
        Caused by: java.lang.Exception: Deserializing the InputFormat ([(0,{0:0.3947476722883563,1:-0.08695028358267716,2:-1.0574297632219802,3:0.3268090996516988,4:-1.3667553319818917,5:-0.1794776700908003,6:1.078276508767426,7:-1.19520500669697,8:-0.48920817822415197,9:-0.01611590341576673,10:-0.3924584320254835,11:1.1084504280408736,12:-0.7766818602582699,13:-1.745148020967139,14:-0.30702403178017207,15:1.0870667203881104,16:0.5743916990799559,17:1.1374342122090273,18:-1.0523085600170734,19:-1.3638541557908512,20:-1.3315774874522164,21:0.13871074941128161,22:-0.1 ...
        
        Show
        Andrew_Palumbo Andrew Palumbo added a comment - for future reference after tweaking some Flink vars the problem with dals ​is​ a serialization error: Process finished with exit code 137 Caused by: java.lang.Exception: Deserializing the InputFormat ([(0,{0:0.3947476722883563,1:-0.08695028358267716,2:-1.0574297632219802,3:0.3268090996516988,4:-1.3667553319818917,5:-0.1794776700908003,6:1.078276508767426,7:-1.19520500669697,8:-0.48920817822415197,9:-0.01611590341576673,10:-0.3924584320254835,11:1.1084504280408736,12:-0.7766818602582699,13:-1.745148020967139,14:-0.30702403178017207,15:1.0870667203881104,16:0.5743916990799559,17:1.1374342122090273,18:-1.0523085600170734,19:-1.3638541557908512,20:-1.3315774874522164,21:0.13871074941128161,22:-0.1 ...
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user andrewpalumbo opened a pull request:

        https://github.com/apache/mahout/pull/218

        MAHOUT-1818: Workaround: Create a FlinkDistributedDecomposionSuite and clean up Tests…

        Create `FlinkDistributedDecompositionSuite` identical to that of `DistributedDecompositionSuiteBase` with the exception of the `dals` test which uses a 350 x 350 matrix rather than a 500 x 500 due to some flink Serialization issues.

        Also remove unneeded tests ahead of release.

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/andrewpalumbo/mahout flink-tests

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/mahout/pull/218.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #218


        commit 9a55b49a01c7ce0c60084d2b0ab8c5ce0ca0df8e
        Author: Andrew Palumbo <apalumbo@apache.org>
        Date: 2016-04-10T22:51:49Z

        (NOJIRA) Create a FlinkDistributedDecomposionSuite and clean up Tests for Flink Release


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user andrewpalumbo opened a pull request: https://github.com/apache/mahout/pull/218 MAHOUT-1818 : Workaround: Create a FlinkDistributedDecomposionSuite and clean up Tests… Create `FlinkDistributedDecompositionSuite` identical to that of `DistributedDecompositionSuiteBase` with the exception of the `dals` test which uses a 350 x 350 matrix rather than a 500 x 500 due to some flink Serialization issues. Also remove unneeded tests ahead of release. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewpalumbo/mahout flink-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/mahout/pull/218.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #218 commit 9a55b49a01c7ce0c60084d2b0ab8c5ce0ca0df8e Author: Andrew Palumbo <apalumbo@apache.org> Date: 2016-04-10T22:51:49Z (NOJIRA) Create a FlinkDistributedDecomposionSuite and clean up Tests for Flink Release
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user andrewpalumbo commented on the pull request:

        https://github.com/apache/mahout/pull/218#issuecomment-208088609

        committed to flink branch by 472438bc83a51bcc08518e44daff4ec5f3bf81e9

        Show
        githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo commented on the pull request: https://github.com/apache/mahout/pull/218#issuecomment-208088609 committed to flink branch by 472438bc83a51bcc08518e44daff4ec5f3bf81e9
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user andrewpalumbo closed the pull request at:

        https://github.com/apache/mahout/pull/218

        Show
        githubbot ASF GitHub Bot added a comment - Github user andrewpalumbo closed the pull request at: https://github.com/apache/mahout/pull/218
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Mahout-Quality #3324 (See https://builds.apache.org/job/Mahout-Quality/3324/)
        MAHOUT-1818 workaround and test cleanup for Flink release closes (apalumbo: rev 472438bc83a51bcc08518e44daff4ec5f3bf81e9)

        • flink/src/test/scala/org/apache/mahout/flinkbindings/standard/DistributedDecompositionsSuite.scala
        • flink/src/test/scala/org/apache/mahout/flinkbindings/standard/FlinkDistributedDecompositionsSuite.scala
        • flink/src/test/scala/org/apache/mahout/flinkbindings/FailingTestsSuite.scala
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Mahout-Quality #3324 (See https://builds.apache.org/job/Mahout-Quality/3324/ ) MAHOUT-1818 workaround and test cleanup for Flink release closes (apalumbo: rev 472438bc83a51bcc08518e44daff4ec5f3bf81e9) flink/src/test/scala/org/apache/mahout/flinkbindings/standard/DistributedDecompositionsSuite.scala flink/src/test/scala/org/apache/mahout/flinkbindings/standard/FlinkDistributedDecompositionsSuite.scala flink/src/test/scala/org/apache/mahout/flinkbindings/FailingTestsSuite.scala
        Hide
        smarthi Suneel Marthi added a comment - - edited

        Could we mark this as resolved with 'Will not Fix' as resolution ? Andrew Palumbo

        Show
        smarthi Suneel Marthi added a comment - - edited Could we mark this as resolved with 'Will not Fix' as resolution ? Andrew Palumbo
        Hide
        Andrew_Palumbo Andrew Palumbo added a comment -

        Yes lets resolve. We wrote a special test for this for a smaller matrix. Some memory issue on the backend kills it.

        Show
        Andrew_Palumbo Andrew Palumbo added a comment - Yes lets resolve. We wrote a special test for this for a smaller matrix. Some memory issue on the backend kills it.

          People

          • Assignee:
            Unassigned
            Reporter:
            Andrew_Palumbo Andrew Palumbo
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development