[HADOOP-16770] Compare two directories in HDFS filesystem for every 5 mins interval for same cluster. (smiliar like diff command in linux)

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.10.0
• Component/s: hdfs-client
• Assignee: Unassigned
• Reporter: GanGSTR
• Created: 18/Dec/19 07:23
• Updated: 18/Dec/19 22:34
• Resolved: 18/Dec/19 22:34

Description

Hi team,

Created two hadoop clusters, one cluster is storing files in new directories based on TIME based directories are created in Hadoop FileSystem say /a/b/time/a.txt b.txt..

For every 5 mins, compare this cluster 1 filesytem for two different directories whether any new directories with list of files are updated or not , if its updated in dir 1, then update those files only to be moved to dir 2. Later those new directories files copied to HDFS cluster 2 file system. 

Currently HDFS not supported hdfs dfs -diff command,  Any solution for this?

Have tried  -copyFromLocal and copyToLocal command, it uses lot of diskspace while copying local to hdfs & hdfs to local.