Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1836

REPL $outer type mismatch causes lookup() and equals() problems

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.9.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Anand Avati partially traced the cause to REPL wrapping classes in $outer classes. There are at least two major symptoms:

      1. equals()
      =========

      In REPL equals() (required in custom classes used as a key for groupByKey) seems to have to be written using instanceOf[] instead of the canonical match{}

      Spark Shell (equals uses match{}):

      class C(val s:String) extends Serializable {
        override def equals(o: Any) = o match {
          case that: C => that.s == s
          case _ => false
        }
      }
      
      val x = new C("a")
      val bos = new java.io.ByteArrayOutputStream()
      val out = new java.io.ObjectOutputStream(bos)
      out.writeObject(x);
      val b = bos.toByteArray();
      out.close
      bos.close
      val y = new java.io.ObjectInputStream(new ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
      x.equals(y)
      
      res: Boolean = false
      

      Spark Shell (equals uses isInstanceOf[]):

      class C(val s:String) extends Serializable {
        override def equals(o: Any) = if (o.isInstanceOf[C]) (o.asInstanceOf[C].s = s) else false
      }
      
      val x = new C("a")
      val bos = new java.io.ByteArrayOutputStream()
      val out = new java.io.ObjectOutputStream(bos)
      out.writeObject(x);
      val b = bos.toByteArray();
      out.close
      bos.close
      val y = new java.io.ObjectInputStream(new ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
      x.equals(y)
      
      res: Boolean = true
      

      Scala Shell (equals uses match{}):

      class C(val s:String) extends Serializable {
        override def equals(o: Any) = o match {
          case that: C => that.s == s
          case _ => false
        }
      }
      
      val x = new C("a")
      val bos = new java.io.ByteArrayOutputStream()
      val out = new java.io.ObjectOutputStream(bos)
      out.writeObject(x);
      val b = bos.toByteArray();
      out.close
      bos.close
      val y = new java.io.ObjectInputStream(new java.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
      x.equals(y)
      
      res: Boolean = true
      

      2. lookup()
      =========

      class C(val s:String) extends Serializable {
        override def equals(o: Any) = if (o.isInstanceOf[C]) o.asInstanceOf[C].s == s else false
        override def hashCode = s.hashCode
        override def toString = s
      }
      val r = sc.parallelize(Array((new C("a"),11),(new C("a"),12)))
      r.lookup(new C("a"))
      <console>:17: error: type mismatch;
       found   : C
       required: C
                    r.lookup(new C("a"))
                             ^
      

      See
      http://mail-archives.apache.org/mod_mbox/spark-dev/201405.mbox/%3C1400019424.80629.YahooMailNeo%40web160801.mail.bf1.yahoo.com%3E

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                michaelmalak Michael Malak
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: