HDFS rename does not work correctly if src contains Java regular expression special characters

• Key: HADOOP-4806
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Tsz Wo (Nicholas), SZE
• Reporter: Tsz Wo (Nicholas), SZE
• Affects Version/s: 0.18.2
• Fix Version/s: 0.18.3

Description

For examples,

1. create a file /a+b/foo but not close it
2. rename /a+b/foo to /a+b/bar/foo
   Then, the corresponding INodeFileUnderConstruction will be moved from /a+b/foo to /a+b/bar/foo but the lease path remains unchanged.

Comments

[Tsz Wo (Nicholas), SZE]

In LeaseManager.changeLease(...), the variable overwrite is not a regular expression pattern but is used as a regular expression pattern in

final String newpath = oldpath.replaceFirst(overwrite, replaceBy);

[Tsz Wo (Nicholas), SZE]

4806_20081208.patch: fixed string replacement bug.

[Tsz Wo (Nicholas), SZE]

S3FileSystem.renameRecursive(...) also has this problem.

[Tsz Wo (Nicholas), SZE]

4806_20081208_0.18.patch: for 0.18 (added TestRenameWhileOpen)

[Hairong Kuang]

Could we Pattern#quote "overwrite"? I just want to be safe...

[Tsz Wo (Nicholas), SZE]

4806_20081209.patch: using Pattern.quote(..).

[Tsz Wo (Nicholas), SZE]

4806_20081209_0.18.patch: for 0.18

[Hairong Kuang]

+1

[Tsz Wo (Nicholas), SZE]

     [exec] +1 overall.  

     [exec]     +1 @author.  The patch does not contain any @author tags.

     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.

     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.

     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.

     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.

Running tests locally.

[Tsz Wo (Nicholas), SZE]

Only org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.TestStartAtOffset failed and it is not related to this patch.

[Tsz Wo (Nicholas), SZE]

4806_20081209_0.18.patch: the previous file was wrong. Should be this one.

[Tsz Wo (Nicholas), SZE]

I just committed this.

[Hudson]

Integrated in Hadoop-trunk #685 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/685/)
. HDFS rename should not use src path as a regular expression. (szetszwo)

[Tsz Wo (Nicholas), SZE]

It turned out that the first patch is safer. See HADOOP-6017.