Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-608

BlockReceiver:receivePacket(): packet's header parsing logic is complicated. Refactoring will help w/ testing efforts

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Current logic of getting an information from a packet header is kinda complicated with all shifting along the internal data buffer back and forth.
      Besides, an absence of clear API to access a packet header's information makes pipeline testing effort (even with aspects' application) to be a very complicated or even impossible.

      It'd be very nice to have a separate data container which will keep all the information about a packet header within and will initialized once when a new packet is acquired.

      In order to speed-up pipeline testing needed for append (HDFS-265) this modification might implemented in three different states:

      • introduce new header structure and refactor receivePacket()
      • refactor readNextPacket so it it will acquire next packet in initialize its header
      • refactor the sender side to use the packet's header structure as the receiver does.
      1. HDFS-608.patch
        4 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Hide
          Konstantin Boudnik added a comment -

          The patch provides the fix for the first step of the refactoring in the receivePacket().
          test-core has passed without any problems.

          Please review.

          Show
          Konstantin Boudnik added a comment - The patch provides the fix for the first step of the refactoring in the receivePacket(). test-core has passed without any problems. Please review.
          Hide
          Konstantin Boudnik added a comment -

          After conversing with Konstantin and Raghu I've been convinced that the fix has to be applied on both sides - i.e. BlockReceiver and DFSClient - at the same time.

          Because the effort is likely to require quite a bit of testing efforts it seems to be impractical to undertake in before 0.21 (and append) are completed. Thus, it's going to be postponed until after 0.21 time frame.

          Show
          Konstantin Boudnik added a comment - After conversing with Konstantin and Raghu I've been convinced that the fix has to be applied on both sides - i.e. BlockReceiver and DFSClient - at the same time. Because the effort is likely to require quite a bit of testing efforts it seems to be impractical to undertake in before 0.21 (and append) are completed. Thus, it's going to be postponed until after 0.21 time frame.
          Hide
          dhruba borthakur added a comment -

          Also, it might make sense to do a benchmark before and after this patch is applied, especially because it is tricky code and can have a profound influence on performance.

          Show
          dhruba borthakur added a comment - Also, it might make sense to do a benchmark before and after this patch is applied, especially because it is tricky code and can have a profound influence on performance.
          Hide
          Konstantin Boudnik added a comment -

          Yes, thank you for the hint - makes a lot of sense!

          Show
          Konstantin Boudnik added a comment - Yes, thank you for the hint - makes a lot of sense!
          Hide
          Konstantin Boudnik added a comment -

          This is DUP of HDFS-881

          Show
          Konstantin Boudnik added a comment - This is DUP of HDFS-881

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development