[MAHOUT-1597] A + 1.0 (element-wise scala operation) gives wrong result if rdd is missing rows, Spark side

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9
• Fix Version/s: 0.10.0
• Assignee: Dmitriy Lyubimov
• Reporter: Dmitriy Lyubimov
• Created: 22/Jul/14 21:03
• Updated: 13/Apr/15 10:21
• Resolved: 28/Jul/14 17:35

Description

    // Concoct an rdd with missing rows
    val aRdd: DrmRdd[Int] = sc.parallelize(
      0 -> dvec(1, 2, 3) ::
          3 -> dvec(3, 4, 5) :: Nil
    ).map { case (key, vec) => key -> (vec: Vector)}

    val drmA = drmWrap(rdd = aRdd)

    val controlB = inCoreA + 1.0

    val drmB = drmA + 1.0

    (drmB -: controlB).norm should be < 1e-10

should not fail.

it was failing due to elementwise scalar operator only evaluates rows actually present in dataset.

In case of Int-keyed row matrices, there are implied rows that yet may not be present in RDD.

Our goal is to detect the condition and evaluate missing rows prior to physical operators that don't work with missing implied rows.