Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1711

Mahout DSL for Flink: implement broadcasting

    Details

    • Type: Task
    • Status: Closed
    • Priority: Minor
    • Resolution: Workaround
    • Affects Version/s: 0.10.1
    • Fix Version/s: 0.12.0
    • Component/s: Flink
    • Labels:
      None

      Description

      as a part of MAHOUT-1570 implement drmBroadcast for flink

        Issue Links

          Activity

          Hide
          agrigorev Alexey Grigorev added a comment - - edited

          drmBroadcast is now implemented by using standard Java serialization and shipping the vector/matrix along with UDFs (see here: https://github.com/alexeygrigorev/mahout/blob/flink-binding/flink/src/main/scala/org/apache/mahout/flinkbindings/FlinkByteBCast.scala)

          It can be improved with some macro that would rewrite the UDFs and would append `withBroadcastSet` to Flink's DataSet

          Show
          agrigorev Alexey Grigorev added a comment - - edited drmBroadcast is now implemented by using standard Java serialization and shipping the vector/matrix along with UDFs (see here: https://github.com/alexeygrigorev/mahout/blob/flink-binding/flink/src/main/scala/org/apache/mahout/flinkbindings/FlinkByteBCast.scala ) It can be improved with some macro that would rewrite the UDFs and would append `withBroadcastSet` to Flink's DataSet
          Hide
          Andrew_Palumbo Andrew Palumbo added a comment - - edited

          There still may be some work to do here. Broadcasting Matrices using .withBroadcast() does not seem to work.

          The current ByteBroadcast implementation seems to be working but we still have a few failing tests that may rely on it so we shouldn't rule it out as a cause/

          Show
          Andrew_Palumbo Andrew Palumbo added a comment - - edited There still may be some work to do here. Broadcasting Matrices using .withBroadcast() does not seem to work. The current ByteBroadcast implementation seems to be working but we still have a few failing tests that may rely on it so we shouldn't rule it out as a cause/
          Hide
          smarthi Suneel Marthi added a comment -

          We don't need this given that we can leverage Samsara-Flink 'drmBroadcast' to broadcast Matrix and Vector. Resolving this as 'Not Required' with a work around.

          Show
          smarthi Suneel Marthi added a comment - We don't need this given that we can leverage Samsara-Flink 'drmBroadcast' to broadcast Matrix and Vector. Resolving this as 'Not Required' with a work around.
          Hide
          dlyubimov Dmitriy Lyubimov added a comment -

          bulk-closing resolved issues

          Show
          dlyubimov Dmitriy Lyubimov added a comment - bulk-closing resolved issues
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Mahout-Quality #3324 (See https://builds.apache.org/job/Mahout-Quality/3324/)
          MAHOUT-1711: Flink: drmBroadcast implemented (alexey.s.grigoriev: rev 08ad113f732adfe844cb5fb68ab5897f75aa2456)

          • flink/src/test/scala/org/apache/mahout/flinkbindings/UseCasesSuite.scala
          • flink/src/test/scala/org/apache/mahout/flinkbindings/examples/ReadCsvExample.scala
          • flink/src/main/scala/org/apache/mahout/flinkbindings/DataSetOps.scala
          • flink/src/main/scala/org/apache/mahout/flinkbindings/blas/FlinkOpMapBlock.scala
          • flink/src/test/scala/org/apache/mahout/flinkbindings/FlinkByteBCastSuite.scala
          • flink/src/main/scala/org/apache/mahout/flinkbindings/drm/CheckpointedFlinkDrm.scala
          • flink/src/main/scala/org/apache/mahout/flinkbindings/package.scala
          • flink/src/main/scala/org/apache/mahout/flinkbindings/FlinkByteBCast.scala
          • flink/src/test/scala/org/apache/mahout/flinkbindings/RLikeOpsSuite.scala
          • flink/src/main/scala/org/apache/mahout/flinkbindings/blas/FlinkOpAtB.scala
          • flink/src/main/scala/org/apache/mahout/flinkbindings/FlinkEngine.scala
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Mahout-Quality #3324 (See https://builds.apache.org/job/Mahout-Quality/3324/ ) MAHOUT-1711 : Flink: drmBroadcast implemented (alexey.s.grigoriev: rev 08ad113f732adfe844cb5fb68ab5897f75aa2456) flink/src/test/scala/org/apache/mahout/flinkbindings/UseCasesSuite.scala flink/src/test/scala/org/apache/mahout/flinkbindings/examples/ReadCsvExample.scala flink/src/main/scala/org/apache/mahout/flinkbindings/DataSetOps.scala flink/src/main/scala/org/apache/mahout/flinkbindings/blas/FlinkOpMapBlock.scala flink/src/test/scala/org/apache/mahout/flinkbindings/FlinkByteBCastSuite.scala flink/src/main/scala/org/apache/mahout/flinkbindings/drm/CheckpointedFlinkDrm.scala flink/src/main/scala/org/apache/mahout/flinkbindings/package.scala flink/src/main/scala/org/apache/mahout/flinkbindings/FlinkByteBCast.scala flink/src/test/scala/org/apache/mahout/flinkbindings/RLikeOpsSuite.scala flink/src/main/scala/org/apache/mahout/flinkbindings/blas/FlinkOpAtB.scala flink/src/main/scala/org/apache/mahout/flinkbindings/FlinkEngine.scala

            People

            • Assignee:
              smarthi Suneel Marthi
              Reporter:
              agrigorev Alexey Grigorev
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development

                  Agile