Hadoop Common
  1. Hadoop Common
  2. HADOOP-1680

Improvements to Block CRC upgrade messages

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: None
    • Labels:
      None

      Description

      Nigel has good suggestions regd various log messages during Block CRC upgrade. I will include the suggestions in the next comment.

        Activity

        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Raghu!

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Raghu!
        Hide
        Nigel Daley added a comment -

        +1 code review

        Show
        Nigel Daley added a comment - +1 code review
        Hide
        Raghu Angadi added a comment -

        Most of the corrections Nigel suggested are made

        Only one not included is about not printing a warning "Could not find metadata file for ..." during datanode upgade. This warning is printed by the core part of datanode which is blissfully unaware of upgrade. It needs a few code changes to for this part of the DataNode to be aware of upgrade. We will include a note about this warning in the 'quick guide for uprade' we will have for this release.

        Show
        Raghu Angadi added a comment - Most of the corrections Nigel suggested are made Only one not included is about not printing a warning "Could not find metadata file for ..." during datanode upgade. This warning is printed by the core part of datanode which is blissfully unaware of upgrade. It needs a few code changes to for this part of the DataNode to be aware of upgrade. We will include a note about this warning in the 'quick guide for uprade' we will have for this release.
        Hide
        Raghu Angadi added a comment -

        NAMENODE:

        • Safe mode will be turned off automatically upon completion of the distributed upgrade: status = 0%
          • change 'status' to 'upgrade progress'
        • 2007-07-27 21:05:20,069 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Marking Datanode Upgrade done
          • change to 'Upgrade of all DataNode blocks is complete.'
        • 2007-07-27 21:05:20,069 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Deleting ".crc" files. This can take few minutes ...
          • change to 'Deleting ".crc" files. This may take a few minutes.'
        • 2007-07-27 21:05:20,599 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Deleted 46 files
          • change to 'Deleted 46 ".crc" files.'
        • 2007-07-27 21:05:20,632 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Upgrade completed.
          • change to 'DataNode upgrade is complete.'
        • Avg completion on Datanodes: 100.00% with 0 errors.
          • change to 'Average completion of all DataNodes: 100.00% with 0 errors.'

        DATANODE:

        • 2007-07-27 21:05:06,867 WARN org.apache.hadoop.dfs.DataNode: Could not find metadata file for blk_-5527862901389508424
          • try turning this message off during distributed upgrade
        • 2007-07-27 22:12:31,666 INFO org.apache.hadoop.dfs.BlockCrcUpgradeUtils: readCrcBuf: 2 out of 3 matched for blk_1227930759043708693 (file: /user/ndaley/0.13.1/part-18)
          • 'readCrcBuf: 2 out of 3 matched' should be 'at least 2 of the 3 replicated CRC files agree'
          • 'file:' should be 'from file:'
        • 2007-07-27 22:12:31,661 INFO org.apache.hadoop.dfs.BlockCrcUpgradeUtils: readCrcBuf: 4 out of 20 matched for blk_1122855582873753750 (file: /20replicationSmallChecksum)
          • 'readCrcBuf: 4 out of 20 matched' should be 'at least 3 of the 4 replicated CRC files agree'.
            ie the 2nd number should be the max that are online
        Show
        Raghu Angadi added a comment - NAMENODE: Safe mode will be turned off automatically upon completion of the distributed upgrade: status = 0% change 'status' to 'upgrade progress' 2007-07-27 21:05:20,069 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Marking Datanode Upgrade done change to 'Upgrade of all DataNode blocks is complete.' 2007-07-27 21:05:20,069 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Deleting ".crc" files. This can take few minutes ... change to 'Deleting ".crc" files. This may take a few minutes.' 2007-07-27 21:05:20,599 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Deleted 46 files change to 'Deleted 46 ".crc" files.' 2007-07-27 21:05:20,632 INFO org.apache.hadoop.dfs.BlockCrcUpgradeNamenode: Upgrade completed. change to 'DataNode upgrade is complete.' Avg completion on Datanodes: 100.00% with 0 errors. change to 'Average completion of all DataNodes: 100.00% with 0 errors.' DATANODE: 2007-07-27 21:05:06,867 WARN org.apache.hadoop.dfs.DataNode: Could not find metadata file for blk_-5527862901389508424 try turning this message off during distributed upgrade 2007-07-27 22:12:31,666 INFO org.apache.hadoop.dfs.BlockCrcUpgradeUtils: readCrcBuf: 2 out of 3 matched for blk_1227930759043708693 (file: /user/ndaley/0.13.1/part-18) 'readCrcBuf: 2 out of 3 matched' should be 'at least 2 of the 3 replicated CRC files agree' 'file:' should be 'from file: ' 2007-07-27 22:12:31,661 INFO org.apache.hadoop.dfs.BlockCrcUpgradeUtils: readCrcBuf: 4 out of 20 matched for blk_1122855582873753750 (file: /20replicationSmallChecksum) 'readCrcBuf: 4 out of 20 matched' should be 'at least 3 of the 4 replicated CRC files agree'. ie the 2nd number should be the max that are online

          People

          • Assignee:
            Raghu Angadi
            Reporter:
            Raghu Angadi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development