Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3077 Quorum-based protocol for reading and writing edit logs
  3. HDFS-4025

QJM: Sychronize past log segments to JNs that missed them

    XMLWordPrintableJSON

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently, if a JournalManager crashes and misses some segment of logs, and then comes back, it will be re-added as a valid part of the quorum on the next log roll. However, it will not have a complete history of log segments (i.e any individual JN may have gaps in its transaction history). This mirrors the behavior of the NameNode when there are multiple local directories specified.

      However, it would be better if a background thread noticed these gaps and "filled them in" by grabbing the segments from other JournalNodes. This increases the resilience of the system when JournalNodes get reformatted or otherwise lose their local disk.

        Attachments

        1. HDFS-4025.000.patch
          24 kB
          Hanisha Koneru
        2. HDFS-4025.001.patch
          34 kB
          Hanisha Koneru
        3. HDFS-4025.002.patch
          35 kB
          Hanisha Koneru
        4. HDFS-4025.003.patch
          34 kB
          Hanisha Koneru
        5. HDFS-4025.004.patch
          38 kB
          Hanisha Koneru
        6. HDFS-4025.005.patch
          39 kB
          Hanisha Koneru
        7. HDFS-4025.006.patch
          34 kB
          Hanisha Koneru
        8. HDFS-4025.007.patch
          38 kB
          Hanisha Koneru
        9. HDFS-4025.008.patch
          46 kB
          Hanisha Koneru
        10. HDFS-4025.009.patch
          47 kB
          Hanisha Koneru
        11. HDFS-4025.010.patch
          47 kB
          Hanisha Koneru
        12. HDFS-4025.011.patch
          47 kB
          Hanisha Koneru

          Issue Links

            Activity

              People

              • Assignee:
                hanishakoneru Hanisha Koneru
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                21 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: