Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1252

TestDFSConcurrentFileOperations broken in 0.20-appendj

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20-append
    • Fix Version/s: 0.20.205.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This test currently has several flaws:

      • It calls DN.updateBlock with a BlockInfo instance, which then causes java.lang.RuntimeException: java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.server.namenode.BlocksMap$BlockInfo.<init>() in the logs when the DN tries to send blockReceived for the block
      • It assumes that getBlockLocations returns an up-to-date length block after a sync, which is false. It happens to work because it calls getBlockLocations directly on the NN, and thus gets a direct reference to the block in the blockmap, which later gets updated

      This patch fixes this test to use the AppendTestUtil functions to intiiate recovery, and generally pass more reliably.

      1. HDFS-1252.20s.1.patch
        3 kB
        Suresh Srinivas
      2. HDFS-1252.20s.patch
        3 kB
        Suresh Srinivas
      3. hdfs-1252.txt
        5 kB
        Todd Lipcon

        Activity

        Hide
        Todd Lipcon added a comment -

        Patch that goes on top of some other uncommitted ones (relies on refactor of recoverFile into AppendTestUtil)

        Show
        Todd Lipcon added a comment - Patch that goes on top of some other uncommitted ones (relies on refactor of recoverFile into AppendTestUtil)
        Hide
        sam rash added a comment -

        does the patch preserver the essence of the test: a file that is about to be closed is moved and lease recovery should still work (ie, recover blocks that are already finalized on DNs)

        Show
        sam rash added a comment - does the patch preserver the essence of the test: a file that is about to be closed is moved and lease recovery should still work (ie, recover blocks that are already finalized on DNs)
        Hide
        Todd Lipcon added a comment -

        Yes, it should preserve the original test intention. It just uses the normal access methods instead of calling functions directly on in the NN instance.

        Show
        Todd Lipcon added a comment - Yes, it should preserve the original test intention. It just uses the normal access methods instead of calling functions directly on in the NN instance.
        Hide
        Suresh Srinivas added a comment -

        I have attached 0.20-security version of the patch.

        Show
        Suresh Srinivas added a comment - I have attached 0.20-security version of the patch.
        Hide
        Jitendra Nath Pandey added a comment -

        +1.

        Show
        Jitendra Nath Pandey added a comment - +1.
        Hide
        Suresh Srinivas added a comment -

        The test TestDFSConcurrentFileOperations failed because append was not enabled. Enabling append in the test to fix the issue.

        Show
        Suresh Srinivas added a comment - The test TestDFSConcurrentFileOperations failed because append was not enabled. Enabling append in the test to fix the issue.
        Hide
        Jitendra Nath Pandey added a comment -

        looks good. +1

        Show
        Jitendra Nath Pandey added a comment - looks good. +1
        Hide
        Suresh Srinivas added a comment -

        I committed the patch to 0.20-security.

        Show
        Suresh Srinivas added a comment - I committed the patch to 0.20-security.
        Hide
        Todd Lipcon added a comment -

        Suresh committed this to 0.20.205

        Show
        Todd Lipcon added a comment - Suresh committed this to 0.20.205
        Hide
        Matt Foley added a comment -

        Closed upon release of 0.20.205.0

        Show
        Matt Foley added a comment - Closed upon release of 0.20.205.0

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development