Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.0
    • Component/s: balancer & mover
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      start balancer failed with NPE
      File this issue to track for QE and dev take a look

      balancer.log:
      2013-03-06 00:19:55,174 ERROR org.apache.hadoop.hdfs.server.balancer.Balancer: java.lang.NullPointerException
      at org.apache.hadoop.hdfs.server.namenode.BlockPlacementPolicy.getInstance(BlockPlacementPolicy.java:165)
      at org.apache.hadoop.hdfs.server.balancer.Balancer.checkReplicationPolicyCompatibility(Balancer.java:799)
      at org.apache.hadoop.hdfs.server.balancer.Balancer.<init>(Balancer.java:808)
      at org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:831)

      Balancer.java
      private void checkReplicationPolicyCompatibility(Configuration conf)
      throws UnsupportedActionException {
      if (!(BlockPlacementPolicy.getInstance(conf, null, null) <== here
      instanceof BlockPlacementPolicyDefault))

      { throw new UnsupportedActionException( "Balancer without BlockPlacementPolicyDefault"); }

      }

      1. HDFS-4558.patch
        0.8 kB
        Junping Du
      2. HDFS-4558-v2.patch
        1 kB
        Junping Du
      3. HDFS-4558-v3.patch
        1 kB
        Junping Du

        Activity

        Wenwu Peng created issue -
        Hide
        Steve Loughran added a comment -

        What's your block placement policy? Something non-standard?

        Show
        Steve Loughran added a comment - What's your block placement policy? Something non-standard?
        Junping Du made changes -
        Field Original Value New Value
        Assignee Junping Du [ djp ]
        Hide
        Junping Du added a comment -

        The conf in running balancer is not initialized well so left as a null value. I will deliver a patch to fix it.

        Show
        Junping Du added a comment - The conf in running balancer is not initialized well so left as a null value. I will deliver a patch to fix it.
        Junping Du made changes -
        Affects Version/s 1.1.2 [ 12323593 ]
        Target Version/s 1.2.0 [ 12321657 ]
        Hide
        Junping Du added a comment -

        This is a pretty simple one, so not include a unit test. I verified it fix my issue (the same with here) in running balancer against branch-1.

        Show
        Junping Du added a comment - This is a pretty simple one, so not include a unit test. I verified it fix my issue (the same with here) in running balancer against branch-1.
        Junping Du made changes -
        Attachment HDFS-4558.patch [ 12572476 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        It seems that this is a 1.2.0 only issue. 1.1.2 should not have this problem.

        Show
        Tsz Wo Nicholas Sze added a comment - It seems that this is a 1.2.0 only issue. 1.1.2 should not have this problem.
        Tsz Wo Nicholas Sze made changes -
        Fix Version/s 1.2.0 [ 12321657 ]
        Affects Version/s 1.2.0 [ 12321657 ]
        Affects Version/s 1.1.2 [ 12323593 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Junping,

        How about we simply remove the default constructor and create a new conf in main(..)?

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Junping, How about we simply remove the default constructor and create a new conf in main(..)?
        Hide
        Junping Du added a comment -

        Yes. It should be 1.2.0 issue.
        That's a better idea as main is the only caller for this default constructor which is not necessary now. I will update a patch soon.

        Show
        Junping Du added a comment - Yes. It should be 1.2.0 issue. That's a better idea as main is the only caller for this default constructor which is not necessary now. I will update a patch soon.
        Hide
        Junping Du added a comment -

        Update as Nicholas suggestion in v2 patch. BTW, The default constructor of balancer consumed by main can work before because its config is initialized later in ToolRunner. After adding check replication policy, its conf cannot be null in construct stage, so here we remove the default constructor.

        Show
        Junping Du added a comment - Update as Nicholas suggestion in v2 patch. BTW, The default constructor of balancer consumed by main can work before because its config is initialized later in ToolRunner. After adding check replication policy, its conf cannot be null in construct stage, so here we remove the default constructor.
        Junping Du made changes -
        Attachment HDFS-4558-v2.patch [ 12572550 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Thanks Junping, the conf also needs to be passed to ToolRunner. Otherwise, we will create two configurations.

        Show
        Tsz Wo Nicholas Sze added a comment - Thanks Junping, the conf also needs to be passed to ToolRunner. Otherwise, we will create two configurations.
        Hide
        Junping Du added a comment -

        Ooops. You are right, Nicholas. v3 is the right one.

        Show
        Junping Du added a comment - Ooops. You are right, Nicholas. v3 is the right one.
        Junping Du made changes -
        Attachment HDFS-4558-v3.patch [ 12572692 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Tsz Wo Nicholas Sze made changes -
        Hadoop Flags Reviewed [ 10343 ]
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Junping!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Junping!
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 1.2.0 [ 12321657 ]
        Resolution Fixed [ 1 ]
        Hide
        Junping Du added a comment -

        Thanks for review and comments. Nicholas!

        Show
        Junping Du added a comment - Thanks for review and comments. Nicholas!
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop 1.2.0.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Junping Du
            Reporter:
            Wenwu Peng
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development