[SPARK-1836] REPL $outer type mismatch causes lookup() and equals() problems

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.9.0
• Assignee: Unassigned
• Reporter: Michael Malak
• Created: 14/May/14 21:23
• Updated: 28/May/14 19:40
• Resolved: 28/May/14 19:40

Description

Anand Avati partially traced the cause to REPL wrapping classes in $outer classes. There are at least two major symptoms:

1. equals()
=========

In REPL equals() (required in custom classes used as a key for groupByKey) seems to have to be written using instanceOf[] instead of the canonical match{}

Spark Shell (equals uses match{}):

class C(val s:String) extends Serializable {
  override def equals(o: Any) = o match {
    case that: C => that.s == s
    case _ => false
  }
}

val x = new C("a")
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)

res: Boolean = false

Spark Shell (equals uses isInstanceOf[]):

class C(val s:String) extends Serializable {
  override def equals(o: Any) = if (o.isInstanceOf[C]) (o.asInstanceOf[C].s = s) else false
}

val x = new C("a")
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new ava.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)

res: Boolean = true

Scala Shell (equals uses match{}):

class C(val s:String) extends Serializable {
  override def equals(o: Any) = o match {
    case that: C => that.s == s
    case _ => false
  }
}

val x = new C("a")
val bos = new java.io.ByteArrayOutputStream()
val out = new java.io.ObjectOutputStream(bos)
out.writeObject(x);
val b = bos.toByteArray();
out.close
bos.close
val y = new java.io.ObjectInputStream(new java.io.ByteArrayInputStream(b)).readObject().asInstanceOf[C]
x.equals(y)

res: Boolean = true

2. lookup()
=========

class C(val s:String) extends Serializable {
  override def equals(o: Any) = if (o.isInstanceOf[C]) o.asInstanceOf[C].s == s else false
  override def hashCode = s.hashCode
  override def toString = s
}
val r = sc.parallelize(Array((new C("a"),11),(new C("a"),12)))
r.lookup(new C("a"))
<console>:17: error: type mismatch;
 found   : C
 required: C
              r.lookup(new C("a"))
                       ^

See
http://mail-archives.apache.org/mod_mbox/spark-dev/201405.mbox/%3C1400019424.80629.YahooMailNeo%40web160801.mail.bf1.yahoo.com%3E

Issue Links

• duplicates: SPARK-1199 Type mismatch in Spark shell when using case class defined in shell