Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14671

Pipeline.setStages needs to handle Array non-covariance

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.1, 2.0.0
    • Fix Version/s: 1.6.2, 2.0.0
    • Component/s: ML
    • Labels:
      None

      Description

      The following code worked for Spark 1.5 but fails for 1.6 (using the WritableStage, UnWritableStage classes in PipelineSuite):

      val stages0 = Array(new UnWritableStage("b"))
      val stages1 = Array(new WritableStage("a"))
      val steps = stages0 ++ stages1
      val p = new Pipeline().setStages(steps)
      
      p.stages.w(steps)
      new ParamPair(p.stages, steps)
      

      This also occurs with a mix of transformers from ml.feature. It is because Java Arrays are non-covariant and the addition of MLWritable to some transformers means the stages0/1 arrays above are not of type Array[PipelineStage].

        Attachments

          Activity

            People

            • Assignee:
              josephkb Joseph K. Bradley
              Reporter:
              josephkb Joseph K. Bradley
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: