Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6914

Resolve huge memory consumption Issue with OIV processing PB-based fsimages

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 2.4.1
    • None
    • None

    Description

      For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing.

      However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server.

      To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%.

      Attachments

        1. HDFS-6914.patch
          7 kB
          Hao Chen
        2. HDFS-6914.v2.patch
          4 kB
          Hao Chen

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            haoch Hao Chen

            Dates

              Created:
              Updated:

              Slack

                Issue deployment