Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5996

hadoop 1.1.2. hdfs write bug

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.2
    • Fix Version/s: 1.1.2
    • Component/s: fuse-dfs
    • Labels:
    • Environment:

      one master and three slave ,all of them are normal

    • Release Note:
      someone has discovered this bug and it has been resolved


      I am a student from China ,my research is Hive data storage on hadoop .There is a hdfs-write bug when I used sql : insert overwrite table wangmeng select * from testTable (this sql is translated into N map( no Reduce) jobs,each map .corresponding to a HDFS file output On disk. ) No matter what value N is , there will always exists some DfsdataoutputStream buffer can not write to disk at last ,such as N=160 files ,then there my be about 5 write-faliure files .,the write-failured hdfs--file size on disk is always 0 bytes rather than a value which is between 0 and zhe correct size. .There does not have any exceptions to throw . and the HDFS WRITTEN statistical data is absolutely correct .
      When I debug , I find those write-failed DFS-buffer own absolutely correct values on its buffer ,but the buffer can not write to disk at last although I use Dfsdataoutputstream.flush() , Dfsdataoutputstream close() .
      .I can not find the reason those dfs-buffer can not write success. Now I choose a method to avoide this problem by using a temporary file : for example , if the DFS-buffer will write to its destination FINAL, now I will let this DFS-buffer write to a temporary file TEM first ,and then I move the TEM data to the destination just by change the hdfs-- file path. This method can avoid the DFS-buffer write -failure .Now I want to fix this problem radically , so How can I patch my codes about this problem and is there anything I can do ? Many Thanks.



          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users



              • Created:

              Time Tracking

              Original Estimate - 504h
              Remaining Estimate - 504h
              Time Spent - Not Specified
              Not Specified

                Issue deployment