Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12822

HDFS unit test failure in AArch64. TestDirectoryScanner.testThrottling: Throttle is too permissive

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: test
    • Environment:

      ARMv8 AArch64, Ubuntu16.04

    • Flags:
      Important

      Description

      Description: Hi, When I ran the HDFS unit test and got a failure in TestDirectoryScanner.java test case :
      TestDirectoryScanner.testThrottling:624 Throttle is too permissive
      detail:
      Running org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
      Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 227.046 sec <<< FAILURE! - in org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner
      testThrottling(org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner) Time elapsed: 198.014 sec <<< FAILURE!
      java.lang.AssertionError: Throttle is too permissive
      at org.junit.Assert.fail(Assert.java:88)
      at org.junit.Assert.assertTrue(Assert.java:41)
      at org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner.testThrottling(TestDirectoryScanner.java:624)

      And below is the failure part of source code TestDirectoryScanner.java:

            ...........
            while ((retries > 0) && ((ratio < 7f) || (ratio > 10f))) {
              scanner = new DirectoryScanner(dataNode, fds, conf);
              ratio = runThrottleTest(blocks);
              retries -= 1;
            }
      
            // Waiting should be about 9x running.
            LOG.info("RATIO: " + ratio);
            assertTrue("Throttle is too restrictive", ratio <= 10f);
            assertTrue("Throttle is too permissive", ratio >= 7f);
          ............
          private float runThrottleTest(int blocks) throws IOException {
            scanner.setRetainDiffs(true);
            scan(blocks, 0, 0, 0, 0, 0);
            scanner.shutdown();
            assertFalse(scanner.getRunStatus());
            return (float)scanner.timeWaitingMs.get() / scanner.timeRunningMs.get();
          }
         .............
      
      

      The ratio in my test is 6.0578866, which is smaller than 7f in the code. So the code thrown out an assertTrue failure.
      My questions are:
      1. Why the ratio was set between 7f and 10f, is it a empirical value?
      2. The ratio is smaller than 7f in AArch64 platform, is this value within normal range?

      Could anyone help? Thanks a lot.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              guangming Guangming Zhang
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified