Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Need to verify that map task in 1.6.0 continuous verify run locally. Should test online and offline map reduce.

        Issue Links

          Activity

          Keith Turner created issue -
          Hide
          Josh Elser added a comment -

          FWIW, I already verified that 1.6.0 continuous verify mappers run locally. This is how I validated that ACCUMULO-2036 was successful.

          I did not test offline mapreduce.

          Show
          Josh Elser added a comment - FWIW, I already verified that 1.6.0 continuous verify mappers run locally. This is how I validated that ACCUMULO-2036 was successful. I did not test offline mapreduce.
          Hide
          Mike Drob added a comment -

          I think there is value in doing this check during the testing suite. We need to verify that locality does not regress between now and release.

          Show
          Mike Drob added a comment - I think there is value in doing this check during the testing suite. We need to verify that locality does not regress between now and release.
          Hide
          Keith Turner added a comment -

          I already verified that 1.6.0 continuous verify mappers run locally

          Right. I was thinking it would be nice to verify again for release candidates or automate it in some way. ugh, this really needs to be automated.

          Show
          Keith Turner added a comment - I already verified that 1.6.0 continuous verify mappers run locally Right. I was thinking it would be nice to verify again for release candidates or automate it in some way. ugh, this really needs to be automated.
          Hide
          Josh Elser added a comment -

          I was looking at MiniYARNCluster for some ideas. In the class-level javadoc, they mention an option that will treat unique "hosts" as the combination of hostname and port, instead of just hostname.

          I bet we could start up a test with two nodemanagers and create some RangeInputSplits with host:port on them instead of just hostname. It's a big of a contrived test since we'd have to inject the port, but maybe it's better than nothing?

          Show
          Josh Elser added a comment - I was looking at MiniYARNCluster for some ideas. In the class-level javadoc, they mention an option that will treat unique "hosts" as the combination of hostname and port, instead of just hostname. I bet we could start up a test with two nodemanagers and create some RangeInputSplits with host:port on them instead of just hostname. It's a big of a contrived test since we'd have to inject the port, but maybe it's better than nothing?
          Hide
          Eric Newton added a comment -

          Is this really a blocker?

          Show
          Eric Newton added a comment - Is this really a blocker?
          Josh Elser made changes -
          Field Original Value New Value
          Priority Blocker [ 1 ] Major [ 3 ]
          Hide
          Sean Busbey added a comment -

          Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks?

          Presumably, for release testing the continuous ingest test and the verification job will run on clusters with a node count >= 5.

          Show
          Sean Busbey added a comment - Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks? Presumably, for release testing the continuous ingest test and the verification job will run on clusters with a node count >= 5.
          Keith Turner made changes -
          Link This issue relates to ACCUMULO-2147 [ ACCUMULO-2147 ]
          Hide
          Keith Turner added a comment -

          Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks?

          That sounds like a good check. Not as rigorous as I would like, but much better than nothing. If there is a bug in the Accumulo code and its giving the wrong location for a tablet, then M/R would report task as local that are not. For example if Accumulo was giving M/R the last location instead of the current location.

          Show
          Keith Turner added a comment - Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks? That sounds like a good check. Not as rigorous as I would like, but much better than nothing. If there is a bug in the Accumulo code and its giving the wrong location for a tablet, then M/R would report task as local that are not. For example if Accumulo was giving M/R the last location instead of the current location.
          Sean Busbey made changes -
          Fix Version/s 1.6.1 [ 12325441 ]
          Fix Version/s 1.6.0 [ 12322468 ]
          Keith Turner made changes -
          Fix Version/s 1.6.0 [ 12322468 ]
          Fix Version/s 1.6.1 [ 12325441 ]
          Hide
          Keith Turner added a comment -

          verified mappers ran locally w/ 1.6.0-RC1

          Show
          Keith Turner added a comment - verified mappers ran locally w/ 1.6.0-RC1
          Keith Turner made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          98d 33m 1 Keith Turner 15/Apr/14 18:57

            People

            • Assignee:
              Keith Turner
              Reporter:
              Keith Turner
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development