[#HADOOP-4176] Implement getFileChecksum(Path) in HftpFileSystem

Hadoop Common

Implement getFileChecksum(Path) in HftpFileSystem

Created: 15/Sep/08 06:30 PM Updated: 08/Jul/09 04:51 PM

Return to search

Component/s:

None

Affects Version/s:

None

Fix Version/s:

0.19.0

Time Tracking:

Not Specified

File Attachments:

File Name	Date Attached	Attached By	Size
4176_20080917.patch	2008-09-18 12:30 AM	Tsz Wo (Nicholas), SZE	14 kB
4176_20080918.patch	2008-09-18 08:14 PM	Tsz Wo (Nicholas), SZE	20 kB
4176_20080918b.patch	2008-09-18 09:15 PM	Tsz Wo (Nicholas), SZE	20 kB
4176_20080918c.patch	2008-09-18 11:44 PM	Tsz Wo (Nicholas), SZE	21 kB
4176_20080919.patch	2008-09-19 07:04 AM	Tsz Wo (Nicholas), SZE	20 kB

Issue Links:

Reference

This issue relates to:

~~HADOOP-3981~~ Need a distributed file checksum algorithm for HDFS

Hadoop Flags:	Reviewed
Release Note:	Implemented getFileChecksum(Path) in HftpFileSystemfor distcp support.
Resolution Date:	19/Sep/08 07:16 AM

Description

« Hide

In order to use FileChecksum in DistCp, we should implement getFileChecksum(Path) in HftpFileSystem

All

Comments

Work Log

Change History

Subversion Commits

Sort Order:

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 18/Sep/08 12:30 AM

4176_20080917.patch: first complete patch for reviewing.

[ Permlink | « Hide ]

Chris Douglas added a comment - 18/Sep/08 01:16 AM

This looks good. Only a couple minor suggestions in DFSClient:

Since callGetBlockLocations is private, there's no compatibility issue and its signature can be changed and all the calls updated instead of redirecting again.
In the javadoc for getFileChecksum, @see is more standard than "Similar to..."

Would you expect the load on the namenode to be significant for large numbers of checksum requests? Might it make sense to redirect this request to a datanode that can compute and return the checksum?

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 18/Sep/08 08:14 PM

> Would you expect the load on the namenode to be significant for large numbers of checksum requests? Might it make sense to redirect this request to a datanode that can compute and return the checksum?

Yes, we should do URL redirect.

4176_20080918.patch: Incorporated all Chris's comments.

[ Permlink | « Hide ]

Tsz Wo (Nicholas), SZE added a comment - 18/Sep/08 09:03 PM

     [exec] +1 overall.  

     [exec]     +1 @author.  The patch does not contain any @author tags.

     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.

     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.

     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.

Testing it locally.