Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4304

Make FSEditLogOp.MAX_OP_SIZE configurable

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 2.0.3-alpha
    • Fix Version/s: 2.1.0-beta
    • Component/s: namenode
    • Labels:
      None

      Description

      Today we ran into an issue where a NN had logged a very large op, greater than the 1.5MB MAX_OP_SIZE constant. In order to successfully load the edits, we had to patch with a larger constant. This constant should be configurable so that we wouldn't have to recompile in these odd cases. Additionally, I think the default should be bumped a bit higher, since it's only a safeguard against OOME, and people tend to run NNs with multi-GB heaps.

      1. HDFS-4304.001.patch
        14 kB
        Colin Patrick McCabe
      2. HDFS-4304.002.patch
        15 kB
        Colin Patrick McCabe
      3. HDFS-4304.003.patch
        15 kB
        Colin Patrick McCabe
      4. HDFS-4304.004.patch
        15 kB
        Colin Patrick McCabe
      5. HDFS-4304.005.patch
        15 kB
        Colin Patrick McCabe

        Activity

        Todd Lipcon created issue -
        Colin Patrick McCabe made changes -
        Field Original Value New Value
        Attachment HDFS-4304.001.patch [ 12560473 ]
        Colin Patrick McCabe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Colin Patrick McCabe made changes -
        Attachment HDFS-4304.002.patch [ 12560479 ]
        Colin Patrick McCabe made changes -
        Attachment HDFS-4304.002.patch [ 12560479 ]
        Colin Patrick McCabe made changes -
        Attachment HDFS-4304.002.patch [ 12560480 ]
        Colin Patrick McCabe made changes -
        Attachment HDFS-4304.003.patch [ 12560658 ]
        Colin Patrick McCabe made changes -
        Attachment HDFS-4304.004.patch [ 12560911 ]
        Colin Patrick McCabe made changes -
        Attachment HDFS-4304.005.patch [ 12563382 ]
        Aaron T. Myers made changes -
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Target Version/s 3.0.0, 2.0.3-alpha [ 12320356, 12323274 ] 2.0.3-alpha, 3.0.0 [ 12323274, 12320356 ]
        Aaron T. Myers made changes -
        Target Version/s 3.0.0, 2.0.3-alpha [ 12320356, 12323274 ] 3.0.0, 2.0.4-beta [ 12320356, 12324031 ]
        Aaron T. Myers made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.0.4-beta [ 12324031 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Colin Patrick McCabe
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development