[SPARK-25315] setting "auto.offset.reset" to "earliest" has no effect in Structured Streaming with Spark 2.3.1 and Kafka 1.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: 2.3.1
• Component/s: Structured Streaming

Environment

Standalone; running in IDEA

• Assignee: Unassigned
• Reporter: Zhenhao Li
• Created: 03/Sep/18 13:50
• Updated: 01/Oct/18 18:12
• Resolved: 01/Oct/18 18:12

Description

The following code won't read from the beginning of the topic

```

val kafkaOptions = Map[String, String](
 "kafka.bootstrap.servers" -> KAFKA_BOOTSTRAP_SERVERS,
 "subscribe" -> TOPIC,
 "group.id" -> GROUP_ID,
 "auto.offset.reset" -> "earliest"
)

val myStream = sparkSession
    .readStream
    .format("kafka")
    .options(kafkaOptions)
    .load()
    .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")

  myStream
    .writeStream
    .format("console")
    .start()
    .awaitTermination()

```