[SPARK-1018] take and collect don't work on HadoopRDD

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.8.1
• Component/s: Spark Core
• Labels: hadoop
• Assignee: Unassigned
• Reporter: Diana Carroll
• Created: 09/Jan/14 10:43
• Updated: 19/Sep/16 14:16
• Resolved: 16/Jan/16 13:21

Description

I am reading a simple text file using hadoopFile as follows:
var hrdd1 = sc.hadoopFile("/home/training/testdata.txt",classOf[TextInputFormat], classOf[LongWritable], classOf[Text])

Testing using this simple text file:
001 this is line 1
002 this is line two
003 yet another line

the data read is correct, as I can tell using println
scala> hrdd1.foreach(println):
(0,001 this is line 1)
(19,002 this is line two)
(40,003 yet another line)

But neither collect nor take work properly. Take prints out the key (byte offset) of the last (non-existent) line repeatedly:
scala> hrdd1.take(4):
res146: Array[(org.apache.hadoop.io.LongWritable, org.apache.hadoop.io.Text)] = Array((61,), (61,), (61,))

Collect is even worse: it complains:
java.io.NotSerializableException: org.apache.hadoop.io.LongWritable at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)

The problem appears to be the LongWritable in both cases, because if I map to a new RDD, converting the values from Text objects to strings, it works:
scala> hrdd1.map(pair => (pair._1.toString,pair._2.toString)).take(4)
res148: Array[(java.lang.String, java.lang.String)] = Array((0,001 this is line 1), (19,002 this is line two), (40,003 yet another line))

Seems to me either rdd.collect and rdd.take ought to handle non-serializable types gracefully, or hadoopFile should return a mapped RDD that converts the hadoop types into the appropriate serializable Java objects. (Or at very least the docs for the API should indicate that the usual RDD methods don't work on HadoopRDDs).

BTW, this behavior is the same for both the old and new API versions of hadoopFile. It also is the same whether the file is from HDFS or a plain old text file.

Issue Links

• duplicates: SPARK-993 Don't reuse Writable objects in HadoopRDDs by default
• is duplicated by: SPARK-3693 Cached Hadoop RDD always return rows with the same value