Hadoop Common
  1. Hadoop Common
  2. HADOOP-1093

NNBench generates millions of NotReplicatedYetException in Namenode log

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Running NNBench on latest trunk (0.12.1 candidate) on a few hundred nodes yielded 2.3 million of these exceptions in the NN log:

      2007-03-08 09:23:03,053 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020 call error:
      org.apache.hadoop.dfs.NotReplicatedYetException: Not replicated yet
      at org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:803)
      at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:309)
      at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:336)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:559)

      I run NNBench to create files with block size set to 1 and replication set to 1. NNBench then writes 1 byte to the file. Minimum replication for the cluster is the default, ie 1. If it encounters an exception while trying to do either the create or write operations, it loops and tries again. Multiply this by 1000 files per node and a few hundred nodes.

      1. nyr5.patch
        12 kB
        dhruba borthakur

        Activity

        Nigel Daley created issue -
        Nigel Daley made changes -
        Field Original Value New Value
        Assignee dhruba borthakur [ dhruba ]
        dhruba borthakur made changes -
        Link This issue is cloned as HADOOP-1108 [ HADOOP-1108 ]
        Hairong Kuang made changes -
        Link This issue is cloned as HADOOP-1108 [ HADOOP-1108 ]
        Sameer Paranjpye made changes -
        Priority Blocker [ 1 ] Major [ 3 ]
        Nigel Daley made changes -
        Fix Version/s 0.12.1 [ 12312347 ]
        Fix Version/s 0.13.0 [ 12312348 ]
        dhruba borthakur made changes -
        Attachment notyetreplicated.patch [ 12354126 ]
        dhruba borthakur made changes -
        Attachment notyetreplicated.patch [ 12354126 ]
        dhruba borthakur made changes -
        Attachment nyr2.patch [ 12354542 ]
        dhruba borthakur made changes -
        Attachment nyr2.patch [ 12354542 ]
        dhruba borthakur made changes -
        Attachment nyr3.patch [ 12355197 ]
        dhruba borthakur made changes -
        Attachment nyr3.patch [ 12355197 ]
        dhruba borthakur made changes -
        Attachment nyr4.patch [ 12355272 ]
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        dhruba borthakur made changes -
        Attachment nyr4.patch [ 12355272 ]
        dhruba borthakur made changes -
        Attachment nyr4.patch [ 12355285 ]
        dhruba borthakur made changes -
        Attachment nyr5.patch [ 12355372 ]
        dhruba borthakur made changes -
        Attachment nyr4.patch [ 12355285 ]
        Doug Cutting made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            Nigel Daley
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development