Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1614

TestDFSIO should allow to configure output directory

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: benchmarks
    • Labels:
      None

      Description

      TestDFSIO has a hardcoded location for its files to be written and read to or from. This location is /benchmarks. However, it might pose a problem if HDFS '/' doesn't allow anyone to write into it. It'd be convenient to have a command line option to specify an alternative location on demand.

      1. MAPREDUCE-1614.patch
        0.6 kB
        Konstantin Boudnik

        Issue Links

          Activity

          Konstantin Boudnik created issue -
          Hide
          Konstantin Boudnik added a comment -

          I have ran TestDFSIO on 0.20 secured cluster and used -Dtest.build.data to some place on HDFS. TestDFSIO failed because it couldn't create /benchmarks directory.
          Therefore I am changing this JIRA from 'Improvement' to 'Bug'

          Show
          Konstantin Boudnik added a comment - I have ran TestDFSIO on 0.20 secured cluster and used -Dtest.build.data to some place on HDFS. TestDFSIO failed because it couldn't create /benchmarks directory. Therefore I am changing this JIRA from 'Improvement' to 'Bug'
          Konstantin Boudnik made changes -
          Field Original Value New Value
          Issue Type Improvement [ 4 ] Bug [ 1 ]
          Hide
          Konstantin Boudnik added a comment -

          It turns out the the problem is pretty typical for the programs in the hadoop-test.jar. E.g. NNBench does't allow to configure a location of the results log thus dictating the working directory from where this test can be ran.

          Show
          Konstantin Boudnik added a comment - It turns out the the problem is pretty typical for the programs in the hadoop-test.jar. E.g. NNBench does't allow to configure a location of the results log thus dictating the working directory from where this test can be ran.
          Hide
          Konstantin Shvachko added a comment -

          Specifying "test.build.data" changes the output directory for TestDFSIO. -Dtest.build.data probably doesn't work, as this parameter should be configured.
          For NNBench there is an explicit -baseDir option.
          Could you please verify that it work.

          Show
          Konstantin Shvachko added a comment - Specifying "test.build.data" changes the output directory for TestDFSIO. -Dtest.build.data probably doesn't work, as this parameter should be configured. For NNBench there is an explicit -baseDir option. Could you please verify that it work.
          Hide
          Konstantin Boudnik added a comment -

          .bq Specifying "test.build.data" changes the output directory for TestDFSIO. -Dtest.build.data probably doesn't work, as this parameter should be configured.

          Yes, setting this parameter does not do any good - I still can see that Permission denied: user=cos, access=WRITE, inode="":hdfs:hdfs:rwxr-xr-x issue

          Show
          Konstantin Boudnik added a comment - .bq Specifying "test.build.data" changes the output directory for TestDFSIO. -Dtest.build.data probably doesn't work, as this parameter should be configured. Yes, setting this parameter does not do any good - I still can see that Permission denied: user=cos, access=WRITE, inode="":hdfs:hdfs:rwxr-xr-x issue
          Konstantin Shvachko made changes -
          Link This issue is part of MAPREDUCE-1832 [ MAPREDUCE-1832 ]
          Miguel Ochoa made changes -
          Link This issue is part of MAPREDUCE-2286 [ MAPREDUCE-2286 ]
          Konstantin Boudnik made changes -
          Assignee Konstantin Boudnik [ cos ]
          Hide
          Konstantin Boudnik added a comment -

          It seems that MAPREDUCE-1832 hasn't fixed this particular problem however has laid all the ground work for it. Simple fix to address the issue in question.

          Show
          Konstantin Boudnik added a comment - It seems that MAPREDUCE-1832 hasn't fixed this particular problem however has laid all the ground work for it. Simple fix to address the issue in question.
          Konstantin Boudnik made changes -
          Attachment MAPREDUCE-1614.patch [ 12477994 ]
          Hide
          Konstantin Boudnik added a comment -

          This has been fixed by MAPREDUCE-1832.

          Show
          Konstantin Boudnik added a comment - This has been fixed by MAPREDUCE-1832 .
          Konstantin Boudnik made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]

            People

            • Assignee:
              Konstantin Boudnik
              Reporter:
              Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 50h
                50h
                Remaining:
                Remaining Estimate - 50h
                50h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development