Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4232

NN fails to write a fsimage with stale leases

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.3-alpha, 0.23.6
    • Component/s: namenode
    • Labels:
      None

      Description

      The reading of a fsimage will ignore leases for non-existent files, but the writing of an image will fail if there are leases for non-existent files. If the image contains leases that reference a non-existent file, then the NN will fail to start, and the 2NN will start but fail to ever write an image.

        Issue Links

          Activity

          Daryn Sharp created issue -
          Hide
          Daryn Sharp added a comment -

          I have not verified if the issue is present in other versions, but it's very likely to affect at least 1.x too.

          Show
          Daryn Sharp added a comment - I have not verified if the issue is present in other versions, but it's very likely to affect at least 1.x too.
          Daryn Sharp made changes -
          Field Original Value New Value
          Description The reading of a fsimage will ignore leases for non-existent files, but the
          writing of an image will fail if there are leases for non-existent files. If
          an image is written when the namesystem and lease manager are temporarily out
          of sync after a delete, then the NN will fail to start, and
          the 2NN will start but fail to ever write an image.
          The reading of a fsimage will ignore leases for non-existent files, but the writing of an image will fail if there are leases for non-existent files. If the image contains leases that reference a non-existent file, then the NN will fail to start, and the 2NN will start but fail to ever write an image.
          Hide
          Daryn Sharp added a comment -

          Ok, I need some input here. I'm attacking the symptom of a problem encountered on a production cluster that encountered a series of complex issues that we can't piece together. Long story short, we wound up with leases to non-existent files and the NN couldn't start. When believed the dangling leases were the result of deletions but the cause is ambiguous.

          My original intent was to make dangling leases non-fatal. It's arguably reasonable because nothing will be lost - that's not already lost (the file). However that masks whatever bug may have caused the issue in the first place.

          1. Looking at the code, the inodes are deleted, then the corresponding leases are deleted, and then the edit log is synced outside the write lock so perhaps it's plausible that the edits were not atomically written? I don't think I can write a test that might induce that possible bug.
          2. The other possibility is that LeaseManager#removeLeaseWithPathPrefix is malfunctioning. It takes a SortedSet, makes another SortedSet backed by the former set, makes a List of Map.Entry from the set of a set, then iterates and the list and performs operations that will affect the original set. Now the javadocs for Map.Entry state:

          Map.Entry objects are valid only for the duration of the iteration; more formally, the behavior of a map entry is undefined if the backing map has been modified after the entry was returned by the iterator, except through the setValue operation on the map entry.

          It sounds like #2 is relying on undefined behavior twofold: using the entries after the iteration is complete, and using the entries while modifying their set. I can easily fix that but I can't write a test that can reliably induce the former undefined behavior to occur.

          So I'm at the crossroads of:

          1. As a safety measure, make dangling leases non-fatal
          2. Fix removeLeaseWithPrefix to avoid undefined behavior, and hope it solves the problem
          3. 1+2

          Advice?

          Show
          Daryn Sharp added a comment - Ok, I need some input here. I'm attacking the symptom of a problem encountered on a production cluster that encountered a series of complex issues that we can't piece together. Long story short, we wound up with leases to non-existent files and the NN couldn't start. When believed the dangling leases were the result of deletions but the cause is ambiguous. My original intent was to make dangling leases non-fatal. It's arguably reasonable because nothing will be lost - that's not already lost (the file). However that masks whatever bug may have caused the issue in the first place. Looking at the code, the inodes are deleted, then the corresponding leases are deleted, and then the edit log is synced outside the write lock so perhaps it's plausible that the edits were not atomically written? I don't think I can write a test that might induce that possible bug. The other possibility is that LeaseManager#removeLeaseWithPathPrefix is malfunctioning. It takes a SortedSet , makes another SortedSet backed by the former set, makes a List of Map.Entry from the set of a set, then iterates and the list and performs operations that will affect the original set. Now the javadocs for Map.Entry state: Map.Entry objects are valid only for the duration of the iteration; more formally, the behavior of a map entry is undefined if the backing map has been modified after the entry was returned by the iterator, except through the setValue operation on the map entry. It sounds like #2 is relying on undefined behavior twofold: using the entries after the iteration is complete, and using the entries while modifying their set. I can easily fix that but I can't write a test that can reliably induce the former undefined behavior to occur. So I'm at the crossroads of: As a safety measure, make dangling leases non-fatal Fix removeLeaseWithPrefix to avoid undefined behavior, and hope it solves the problem 1+2 Advice?
          Hide
          Daryn Sharp added a comment -

          A possible "safety" patch.

          Show
          Daryn Sharp added a comment - A possible "safety" patch.
          Daryn Sharp made changes -
          Attachment HDFS-4232.patch [ 12555267 ]
          Hide
          Daryn Sharp added a comment -

          As I was writing tests, I found that renames can cause double leases for old and new path! Will have a patch shortly.

          Show
          Daryn Sharp added a comment - As I was writing tests, I found that renames can cause double leases for old and new path! Will have a patch shortly.
          Daryn Sharp made changes -
          Link This issue is related to HDFS-2875 [ HDFS-2875 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Hi Daryn,

          Good catch on #2! It is a valid concern but I think it unlikely causes the problem here. Let's fix it separately.

          How do renames cause double leases?

          Show
          Tsz Wo Nicholas Sze added a comment - Hi Daryn, Good catch on #2! It is a valid concern but I think it unlikely causes the problem here. Let's fix it separately. How do renames cause double leases?
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-4242 [ HDFS-4242 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Filed HDFS-4242 for #2.

          Show
          Tsz Wo Nicholas Sze added a comment - Filed HDFS-4242 for #2.
          Hide
          Daryn Sharp added a comment -

          Thanks Nicholas! Although I already had a patch for the problem (with a few more critical fixes) that I've been testing.

          Show
          Daryn Sharp added a comment - Thanks Nicholas! Although I already had a patch for the problem (with a few more critical fixes) that I've been testing.
          Hide
          Daryn Sharp added a comment -

          This jira was split into subtasks, those are complete.

          Show
          Daryn Sharp added a comment - This jira was split into subtasks, those are complete.
          Daryn Sharp made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Fix Version/s 2.0.3-alpha [ 12323274 ]
          Fix Version/s 0.23.6 [ 12323503 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          19d 18h 55m 1 Daryn Sharp 18/Dec/12 15:41
          Resolved Resolved Closed Closed
          58d 21h 29m 1 Arun C Murthy 15/Feb/13 13:11

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development