Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Need to verify that map task in 1.6.0 continuous verify run locally. Should test online and offline map reduce.

        Issue Links

          Activity

          Hide
          Keith Turner added a comment -

          verified mappers ran locally w/ 1.6.0-RC1

          Show
          Keith Turner added a comment - verified mappers ran locally w/ 1.6.0-RC1
          Hide
          Keith Turner added a comment -

          Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks?

          That sounds like a good check. Not as rigorous as I would like, but much better than nothing. If there is a bug in the Accumulo code and its giving the wrong location for a tablet, then M/R would report task as local that are not. For example if Accumulo was giving M/R the last location instead of the current location.

          Show
          Keith Turner added a comment - Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks? That sounds like a good check. Not as rigorous as I would like, but much better than nothing. If there is a bug in the Accumulo code and its giving the wrong location for a tablet, then M/R would report task as local that are not. For example if Accumulo was giving M/R the last location instead of the current location.
          Hide
          Sean Busbey added a comment -

          Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks?

          Presumably, for release testing the continuous ingest test and the verification job will run on clusters with a node count >= 5.

          Show
          Sean Busbey added a comment - Would it be sufficient to have the verification MR job look at the job counters to verify that the number of data local map tasks are within some threshold of the total number of launched map tasks? Presumably, for release testing the continuous ingest test and the verification job will run on clusters with a node count >= 5.
          Hide
          Eric Newton added a comment -

          Is this really a blocker?

          Show
          Eric Newton added a comment - Is this really a blocker?
          Hide
          Josh Elser added a comment -

          I was looking at MiniYARNCluster for some ideas. In the class-level javadoc, they mention an option that will treat unique "hosts" as the combination of hostname and port, instead of just hostname.

          I bet we could start up a test with two nodemanagers and create some RangeInputSplits with host:port on them instead of just hostname. It's a big of a contrived test since we'd have to inject the port, but maybe it's better than nothing?

          Show
          Josh Elser added a comment - I was looking at MiniYARNCluster for some ideas. In the class-level javadoc, they mention an option that will treat unique "hosts" as the combination of hostname and port, instead of just hostname. I bet we could start up a test with two nodemanagers and create some RangeInputSplits with host:port on them instead of just hostname. It's a big of a contrived test since we'd have to inject the port, but maybe it's better than nothing?
          Hide
          Keith Turner added a comment -

          I already verified that 1.6.0 continuous verify mappers run locally

          Right. I was thinking it would be nice to verify again for release candidates or automate it in some way. ugh, this really needs to be automated.

          Show
          Keith Turner added a comment - I already verified that 1.6.0 continuous verify mappers run locally Right. I was thinking it would be nice to verify again for release candidates or automate it in some way. ugh, this really needs to be automated.
          Hide
          Mike Drob added a comment -

          I think there is value in doing this check during the testing suite. We need to verify that locality does not regress between now and release.

          Show
          Mike Drob added a comment - I think there is value in doing this check during the testing suite. We need to verify that locality does not regress between now and release.
          Hide
          Josh Elser added a comment -

          FWIW, I already verified that 1.6.0 continuous verify mappers run locally. This is how I validated that ACCUMULO-2036 was successful.

          I did not test offline mapreduce.

          Show
          Josh Elser added a comment - FWIW, I already verified that 1.6.0 continuous verify mappers run locally. This is how I validated that ACCUMULO-2036 was successful. I did not test offline mapreduce.

            People

            • Assignee:
              Keith Turner
              Reporter:
              Keith Turner
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development