[SPARK-43198] Fix "Could not initialise class ammonite..." error when using filter

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Connect
• Epic Link: Spark Connect Scala Client - Basic
• Assignee: Venkata Sai Akhil Gudesa
• Reporter: Venkata Sai Akhil Gudesa
• Created: 19/Apr/23 16:07
• Updated: 26/Apr/23 16:13
• Resolved: 26/Apr/23 16:13

Description

When

spark.range(10).filter(n => n % 2 == 0).collectAsList()`

 is run in the ammonite REPL (Spark Connect), the following error is thrown:

io.grpc.StatusRuntimeException: UNKNOWN: ammonite/repl/ReplBridge$
  io.grpc.Status.asRuntimeException(Status.java:535)
  io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
  org.apache.spark.sql.connect.client.SparkResult.org$apache$spark$sql$connect$client$SparkResult$$processResponses(SparkResult.scala:62)
  org.apache.spark.sql.connect.client.SparkResult.length(SparkResult.scala:114)
  org.apache.spark.sql.connect.client.SparkResult.toArray(SparkResult.scala:131)
  org.apache.spark.sql.Dataset.$anonfun$collect$1(Dataset.scala:2687)
  org.apache.spark.sql.Dataset.withResult(Dataset.scala:3088)
  org.apache.spark.sql.Dataset.collect(Dataset.scala:2686)
  org.apache.spark.sql.Dataset.collectAsList(Dataset.scala:2700)
  ammonite.$sess.cmd0$.<init>(cmd0.sc:1)
  ammonite.$sess.cmd0$.<clinit>(cmd0.sc)