[SPARK-16428] Spark file system watcher not working on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.6.2
• Component/s: Examples, Input/Output, Spark Core, Windows
• Labels: bulk-closed

Environment

Ubuntu 15.10 64 bit, Windows 7 Enterprise 64 bit

• Flags: Important
• External issue URL: http://stackoverflow.com/questions/38254405/spark-file-system-watcher-not-working-on-windows
• Assignee: Unassigned
• Reporter: John-Michael Reed
• Created: 07/Jul/16 20:34
• Updated: 12/Dec/22 18:10
• Resolved: 21/May/19 04:15

Description

Two people tested Apache Spark on their computers...

[Spark Download - http://i.stack.imgur.com/z1oqu.png]

We downloaded the version of Spark prebuild for Hadoop 2.6, went to the folder /spark-1.6.2-bin-hadoop2.6/, created a "tmp" directory, went to that directory, and ran:

$ bin/run-example org.apache.spark.examples.streaming.HdfsWordCount tmp

I added arbitrary files content1 and content2dssdgdg to that "tmp" directory.

-------------------------------------------
Time: 1467921704000 ms
-------------------------------------------
(content1,1)
(content2dssdgdg,1)

-------------------------------------------
Time: 1467921706000 ms

Spark detected those files with the above terminal output on my Ubuntu 15.10 laptop, but not on my colleague's Windows 7 Enterprise laptop.

This is preventing us from getting work done with Spark.

Link: http://stackoverflow.com/questions/38254405/spark-file-system-watcher-not-working-on-windows