Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5788

listLocatedStatus response can be very large

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 0.23.10, 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently we limit the size of listStatus requests to a default of 1000 entries. This works fine except in the case of listLocatedStatus where the location information can be quite large. As an example, a directory with 7000 entries, 4 blocks each, 3 way replication - a listLocatedStatus response is over 1MB. This can chew up very large amounts of memory in the NN if lots of clients try to do this simultaneously.

      Seems like it would be better if we also considered the amount of location information being returned when deciding how many files to return.

      Patch will follow shortly.

      1. HDFS-5788.patch
        6 kB
        Nathan Roberts

        Issue Links

          Activity

          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320356 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Fix Version/s 2.3.0 [ 12325255 ]
          Fix Version/s 2.4.0 [ 12324588 ]
          Kihwal Lee made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320356 ]
          Fix Version/s 2.4.0 [ 12324588 ]
          Resolution Fixed [ 1 ]
          Nathan Roberts made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Nathan Roberts made changes -
          Attachment HDFS-5788.patch [ 12624374 ]
          Jason Lowe made changes -
          Field Original Value New Value
          Link This issue relates to HADOOP-8942 [ HADOOP-8942 ]
          Nathan Roberts created issue -

            People

            • Assignee:
              Nathan Roberts
              Reporter:
              Nathan Roberts
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development