Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-1843

Undetectable corruption after restart of a datanode

    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      Right now, all write chunks use BufferedIO ie, sync flag is disabled by default. Also, Rocks Db metadata updates are done in Rocks DB cache first at Datanode. In case, there comes a situation where the buffered chunk data as well as the corresponding metadata update is lost as a part of datanode restart, it may lead to a situation where, it will not be possible to detect the corruption (not even with container scanner) of this nature in a reasonable time frame, until and unless there is a client IO failure or Recon server detects it over time. In order to atleast to detect the problem, Ratis snapshot on datanode should sync the rocks db file . In such a way, ContainerScanner will be able to detect this.We can also add a metric around sync to measure how much of a throughput loss it can incurr.

      Thanks Mukul Kumar Singh for suggesting this.

        Attachments

        1. HDDS-1843.000.patch
          30 kB
          Shashikant Banerjee

          Issue Links

            Activity

              People

              • Assignee:
                shashikant Shashikant Banerjee
                Reporter:
                shashikant Shashikant Banerjee
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10h
                  10h