Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5721

Query with only root fragment and no non-root fragment hangs when Drillbit to Drillbit Control Connection has network issues

    Details

    • Type: Bug
    • Status: Reviewable
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.12.0
    • Component/s: None
    • Labels:

      Description

      Recently I found an issue (Thanks to Krystal to create this scenario) related to Fragment Status reporting and would like some feedback on it.

      When a client submits a query to Foreman, then it is planned by Foreman and later fragments are scheduled to root and non-root nodes. Foreman creates a DriilbitStatusListener and FragmentStatusListener to know about the health of Drillbit node and a fragment respectively. The way root and non-root fragments are setup by Foreman are different:
      Root fragments are setup without any communication over control channel (since it is executed locally on Foreman)
      Non-root fragments are setup by sending control message (REQ_INITIALIZE_FRAGMENTS_VALUE) over wire. If there is failure in sending any such control message (like due to network hiccup's) during query setup then the query is failed and client is notified.

      Each fragment is executed on it's node with the help Fragment Executor which has an instance for FragmentStatusReporter. FragmentStatusReporter helps to update the status of a fragment to Foreman node over a control tunnel or connection using RPC message (REQ_FRAGMENT_STATUS) both for root and non-root fragments.

      Based on above when root fragment is submitted for setup then it is done locally without any RPC communication whereas when status for that fragment is reported by fragment executor that happens over control connection by sending a RPC message. But for non-root fragment setup and status update both happens using RPC message over control connection.

      Issue 1:
      What was observed is if for a simple query which has only 1 root fragment running on Foreman node then setup will work fine. But as part of status update when the fragment tries to create a control connection and fails to establish that, then the query hangs. This is because the root fragment will complete execution but will fail to update Foreman about it and Foreman think that the query is running for ever.

      Proposed Solution:
      For root fragment the setup of fragment is happening locally without RPC message, so we can do the same for status update of root fragments. This will avoid RPC communication for status update of fragments running locally on the foreman and hence will resolve issue 1.

        Issue Links

          Activity

          Hide
          shamirwasia Sorabh Hamirwasia added a comment -

          Parth Chandra - Please help to review this PR. For some reason the JIRA is not picking up the PR automatically should I close and re-open the PR ?

          Show
          shamirwasia Sorabh Hamirwasia added a comment - Parth Chandra - Please help to review this PR. For some reason the JIRA is not picking up the PR automatically should I close and re-open the PR ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on the issue:

          https://github.com/apache/drill/pull/919

          Rebased the branch to latest master and updated test's included in DRILL-5701

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on the issue: https://github.com/apache/drill/pull/919 Rebased the branch to latest master and updated test's included in DRILL-5701
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user parthchandra commented on a diff in the pull request:

          https://github.com/apache/drill/pull/919#discussion_r135937433

          — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java —
          @@ -1073,26 +1070,22 @@ public QueryId getQueryId() {
          */
          private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator)
          throws ExecutionSetupException {

          • @SuppressWarnings("resource")
            final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext,
            initiatingClient, drillbitContext.getFunctionImplementationRegistry());
          • @SuppressWarnings("resource")
          • final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext);
          • rootContext.setBuffers(buffers);
            -
          • queryManager.addFragmentStatusTracker(rootFragment, true);
            -
            final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint());
            + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel);
            final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment,
          • new FragmentStatusReporter(rootContext, tunnel),
          • rootOperator);
          • final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner);
            + statusReporter, rootOperator);
          • if (buffers.isDone()) {
            + queryManager.addFragmentStatusTracker(rootFragment, true);
            +
            + // FragmentManager is setting buffer for FragmentContext
            + if (rootContext.isBuffersDone()) {
              • End diff –

          I don't see where rootContext.setBuffers is being called to set the buffers

          Show
          githubbot ASF GitHub Bot added a comment - Github user parthchandra commented on a diff in the pull request: https://github.com/apache/drill/pull/919#discussion_r135937433 — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java — @@ -1073,26 +1070,22 @@ public QueryId getQueryId() { */ private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator) throws ExecutionSetupException { @SuppressWarnings("resource") final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext, initiatingClient, drillbitContext.getFunctionImplementationRegistry()); @SuppressWarnings("resource") final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext); rootContext.setBuffers(buffers); - queryManager.addFragmentStatusTracker(rootFragment, true); - final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint()); + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel); final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment, new FragmentStatusReporter(rootContext, tunnel), rootOperator); final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner); + statusReporter, rootOperator); if (buffers.isDone()) { + queryManager.addFragmentStatusTracker(rootFragment, true); + + // FragmentManager is setting buffer for FragmentContext + if (rootContext.isBuffersDone()) { End diff – I don't see where rootContext.setBuffers is being called to set the buffers
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user parthchandra commented on a diff in the pull request:

          https://github.com/apache/drill/pull/919#discussion_r135937508

          — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java —
          @@ -1109,7 +1102,7 @@ private void setupNonRootFragments(final Collection<PlanFragment> fragments) thr
          // nothing to do here
          return;
          }

          • /*
            + /*z
              • End diff –

          extra character

          Show
          githubbot ASF GitHub Bot added a comment - Github user parthchandra commented on a diff in the pull request: https://github.com/apache/drill/pull/919#discussion_r135937508 — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java — @@ -1109,7 +1102,7 @@ private void setupNonRootFragments(final Collection<PlanFragment> fragments) thr // nothing to do here return; } /* + /*z End diff – extra character
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on a diff in the pull request:

          https://github.com/apache/drill/pull/919#discussion_r135948567

          — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java —
          @@ -1073,26 +1070,22 @@ public QueryId getQueryId() {
          */
          private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator)
          throws ExecutionSetupException {

          • @SuppressWarnings("resource")
            final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext,
            initiatingClient, drillbitContext.getFunctionImplementationRegistry());
          • @SuppressWarnings("resource")
          • final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext);
          • rootContext.setBuffers(buffers);
            -
          • queryManager.addFragmentStatusTracker(rootFragment, true);
            -
            final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint());
            + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel);
            final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment,
          • new FragmentStatusReporter(rootContext, tunnel),
          • rootOperator);
          • final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner);
            + statusReporter, rootOperator);
          • if (buffers.isDone()) {
            + queryManager.addFragmentStatusTracker(rootFragment, true);
            +
            + // FragmentManager is setting buffer for FragmentContext
            + if (rootContext.isBuffersDone()) {
              • End diff –

          Inside AbstractFragmentManager constructor, since that's where the Incoming Buffers are created.

          https://github.com/apache/drill/pull/919/files#diff-d7418a1fd07314f268049ae9f07b7ed3

          ```
          + public AbstractFragmentManager(final PlanFragment fragment, final FragmentExecutor executor, final FragmentStatusReporter statusReporter, final FragmentRoot rootOperator) {
          + this.fragmentHandle = fragment.getHandle();
          + this.fragmentContext = executor.getContext();
          + this.buffers = new IncomingBuffers(fragment, fragmentContext);
          + this.fragmentContext.setBuffers(buffers);
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/919#discussion_r135948567 — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java — @@ -1073,26 +1070,22 @@ public QueryId getQueryId() { */ private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator) throws ExecutionSetupException { @SuppressWarnings("resource") final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext, initiatingClient, drillbitContext.getFunctionImplementationRegistry()); @SuppressWarnings("resource") final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext); rootContext.setBuffers(buffers); - queryManager.addFragmentStatusTracker(rootFragment, true); - final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint()); + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel); final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment, new FragmentStatusReporter(rootContext, tunnel), rootOperator); final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner); + statusReporter, rootOperator); if (buffers.isDone()) { + queryManager.addFragmentStatusTracker(rootFragment, true); + + // FragmentManager is setting buffer for FragmentContext + if (rootContext.isBuffersDone()) { End diff – Inside AbstractFragmentManager constructor, since that's where the Incoming Buffers are created. https://github.com/apache/drill/pull/919/files#diff-d7418a1fd07314f268049ae9f07b7ed3 ``` + public AbstractFragmentManager(final PlanFragment fragment, final FragmentExecutor executor, final FragmentStatusReporter statusReporter, final FragmentRoot rootOperator) { + this.fragmentHandle = fragment.getHandle(); + this.fragmentContext = executor.getContext(); + this.buffers = new IncomingBuffers(fragment, fragmentContext); + this.fragmentContext.setBuffers(buffers); ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on a diff in the pull request:

          https://github.com/apache/drill/pull/919#discussion_r135949946

          — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java —
          @@ -1109,7 +1102,7 @@ private void setupNonRootFragments(final Collection<PlanFragment> fragments) thr
          // nothing to do here
          return;
          }

          • /*
            + /*z
              • End diff –

          It was part for refactor commit only and was later removed in next commit.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/919#discussion_r135949946 — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java — @@ -1109,7 +1102,7 @@ private void setupNonRootFragments(final Collection<PlanFragment> fragments) thr // nothing to do here return; } /* + /*z End diff – It was part for refactor commit only and was later removed in next commit.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user parthchandra commented on a diff in the pull request:

          https://github.com/apache/drill/pull/919#discussion_r136155556

          — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java —
          @@ -1073,26 +1070,22 @@ public QueryId getQueryId() {
          */
          private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator)
          throws ExecutionSetupException {

          • @SuppressWarnings("resource")
            final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext,
            initiatingClient, drillbitContext.getFunctionImplementationRegistry());
          • @SuppressWarnings("resource")
          • final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext);
          • rootContext.setBuffers(buffers);
            -
          • queryManager.addFragmentStatusTracker(rootFragment, true);
            -
            final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint());
            + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel);
            final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment,
          • new FragmentStatusReporter(rootContext, tunnel),
          • rootOperator);
          • final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner);
            + statusReporter, rootOperator);
          • if (buffers.isDone()) {
            + queryManager.addFragmentStatusTracker(rootFragment, true);
            +
            + // FragmentManager is setting buffer for FragmentContext
            + if (rootContext.isBuffersDone()) {
              • End diff –

          Yes but from the looks of it the fragment manager hasn't been created yet. In fact it seems it is created in the else part of the condition you're checking.

          Show
          githubbot ASF GitHub Bot added a comment - Github user parthchandra commented on a diff in the pull request: https://github.com/apache/drill/pull/919#discussion_r136155556 — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java — @@ -1073,26 +1070,22 @@ public QueryId getQueryId() { */ private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator) throws ExecutionSetupException { @SuppressWarnings("resource") final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext, initiatingClient, drillbitContext.getFunctionImplementationRegistry()); @SuppressWarnings("resource") final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext); rootContext.setBuffers(buffers); - queryManager.addFragmentStatusTracker(rootFragment, true); - final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint()); + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel); final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment, new FragmentStatusReporter(rootContext, tunnel), rootOperator); final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner); + statusReporter, rootOperator); if (buffers.isDone()) { + queryManager.addFragmentStatusTracker(rootFragment, true); + + // FragmentManager is setting buffer for FragmentContext + if (rootContext.isBuffersDone()) { End diff – Yes but from the looks of it the fragment manager hasn't been created yet. In fact it seems it is created in the else part of the condition you're checking.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on a diff in the pull request:

          https://github.com/apache/drill/pull/919#discussion_r136157876

          — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java —
          @@ -1073,26 +1070,22 @@ public QueryId getQueryId() {
          */
          private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator)
          throws ExecutionSetupException {

          • @SuppressWarnings("resource")
            final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext,
            initiatingClient, drillbitContext.getFunctionImplementationRegistry());
          • @SuppressWarnings("resource")
          • final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext);
          • rootContext.setBuffers(buffers);
            -
          • queryManager.addFragmentStatusTracker(rootFragment, true);
            -
            final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint());
            + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel);
            final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment,
          • new FragmentStatusReporter(rootContext, tunnel),
          • rootOperator);
          • final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner);
            + statusReporter, rootOperator);
          • if (buffers.isDone()) {
            + queryManager.addFragmentStatusTracker(rootFragment, true);
            +
            + // FragmentManager is setting buffer for FragmentContext
            + if (rootContext.isBuffersDone()) {
              • End diff –

          Sorry about the confusion. It was moved out of else in second commit.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on a diff in the pull request: https://github.com/apache/drill/pull/919#discussion_r136157876 — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java — @@ -1073,26 +1070,22 @@ public QueryId getQueryId() { */ private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator) throws ExecutionSetupException { @SuppressWarnings("resource") final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext, initiatingClient, drillbitContext.getFunctionImplementationRegistry()); @SuppressWarnings("resource") final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext); rootContext.setBuffers(buffers); - queryManager.addFragmentStatusTracker(rootFragment, true); - final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint()); + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel); final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment, new FragmentStatusReporter(rootContext, tunnel), rootOperator); final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner); + statusReporter, rootOperator); if (buffers.isDone()) { + queryManager.addFragmentStatusTracker(rootFragment, true); + + // FragmentManager is setting buffer for FragmentContext + if (rootContext.isBuffersDone()) { End diff – Sorry about the confusion. It was moved out of else in second commit.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user parthchandra commented on a diff in the pull request:

          https://github.com/apache/drill/pull/919#discussion_r136161134

          — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java —
          @@ -1073,26 +1070,22 @@ public QueryId getQueryId() {
          */
          private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator)
          throws ExecutionSetupException {

          • @SuppressWarnings("resource")
            final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext,
            initiatingClient, drillbitContext.getFunctionImplementationRegistry());
          • @SuppressWarnings("resource")
          • final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext);
          • rootContext.setBuffers(buffers);
            -
          • queryManager.addFragmentStatusTracker(rootFragment, true);
            -
            final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint());
            + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel);
            final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment,
          • new FragmentStatusReporter(rootContext, tunnel),
          • rootOperator);
          • final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner);
            + statusReporter, rootOperator);
          • if (buffers.isDone()) {
            + queryManager.addFragmentStatusTracker(rootFragment, true);
            +
            + // FragmentManager is setting buffer for FragmentContext
            + if (rootContext.isBuffersDone()) {
              • End diff –

          Yes I see it now. Sorry I missed it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user parthchandra commented on a diff in the pull request: https://github.com/apache/drill/pull/919#discussion_r136161134 — Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java — @@ -1073,26 +1070,22 @@ public QueryId getQueryId() { */ private void setupRootFragment(final PlanFragment rootFragment, final FragmentRoot rootOperator) throws ExecutionSetupException { @SuppressWarnings("resource") final FragmentContext rootContext = new FragmentContext(drillbitContext, rootFragment, queryContext, initiatingClient, drillbitContext.getFunctionImplementationRegistry()); @SuppressWarnings("resource") final IncomingBuffers buffers = new IncomingBuffers(rootFragment, rootContext); rootContext.setBuffers(buffers); - queryManager.addFragmentStatusTracker(rootFragment, true); - final ControlTunnel tunnel = drillbitContext.getController().getTunnel(queryContext.getCurrentEndpoint()); + final FragmentStatusReporter statusReporter = new FragmentStatusReporter(rootContext, tunnel); final FragmentExecutor rootRunner = new FragmentExecutor(rootContext, rootFragment, new FragmentStatusReporter(rootContext, tunnel), rootOperator); final RootFragmentManager fragmentManager = new RootFragmentManager(rootFragment.getHandle(), buffers, rootRunner); + statusReporter, rootOperator); if (buffers.isDone()) { + queryManager.addFragmentStatusTracker(rootFragment, true); + + // FragmentManager is setting buffer for FragmentContext + if (rootContext.isBuffersDone()) { End diff – Yes I see it now. Sorry I missed it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user parthchandra commented on the issue:

          https://github.com/apache/drill/pull/919

          +1 LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user parthchandra commented on the issue: https://github.com/apache/drill/pull/919 +1 LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on the issue:

          https://github.com/apache/drill/pull/919

          @parthchandra - thanks for the review!

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on the issue: https://github.com/apache/drill/pull/919 @parthchandra - thanks for the review!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers commented on the issue:

          https://github.com/apache/drill/pull/919

          @sohami, can you clean up the stray commits and ask @parthchandra to again review the later commits?

          Show
          githubbot ASF GitHub Bot added a comment - Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/919 @sohami, can you clean up the stray commits and ask @parthchandra to again review the later commits?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on the issue:

          https://github.com/apache/drill/pull/919

          @paul-rogers - Not sure which commit you are referring to. All the commits are reviewed by @parthchandra. I just rebased commits to latest master after last week pull request since @amansinha100 was observing some random functional test failure. After rebase I ran those tests in isolation for multiple iterations and those were passing. Let me know if you want me to combine all these 3 commits.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on the issue: https://github.com/apache/drill/pull/919 @paul-rogers - Not sure which commit you are referring to. All the commits are reviewed by @parthchandra. I just rebased commits to latest master after last week pull request since @amansinha100 was observing some random functional test failure. After rebase I ran those tests in isolation for multiple iterations and those were passing. Let me know if you want me to combine all these 3 commits.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers commented on the issue:

          https://github.com/apache/drill/pull/919

          OK, thanks. Got confused by the commit messages that appeared after the +1. Maybe in the future, when we rebase, just add a comment saying that's what happened.

          Show
          githubbot ASF GitHub Bot added a comment - Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/919 OK, thanks. Got confused by the commit messages that appeared after the +1. Maybe in the future, when we rebase, just add a comment saying that's what happened.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers commented on the issue:

          https://github.com/apache/drill/pull/919

          Tried to commit this one to master. The functional test run failed with 20+ "random failures." That is too many to comfortably allow. So, @sohami, please rerun this, determine if the 20+ failures are, indeed, random or due to this change, and post a note here when the PR is ready for another attempt at commit into master.

          Show
          githubbot ASF GitHub Bot added a comment - Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/919 Tried to commit this one to master. The functional test run failed with 20+ "random failures." That is too many to comfortably allow. So, @sohami, please rerun this, determine if the 20+ failures are, indeed, random or due to this change, and post a note here when the PR is ready for another attempt at commit into master.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on the issue:

          https://github.com/apache/drill/pull/919

          @paul-rogers - Last week @amansinha100 also reported 5 Random failures happening with this commit. But when ran in isolation for multiple iterations those tests were passing all the time. Can you please share the list of tests failing for you with this commit ? I can try to run a sample of those tests in isolation and see if those are really random.

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on the issue: https://github.com/apache/drill/pull/919 @paul-rogers - Last week @amansinha100 also reported 5 Random failures happening with this commit. But when ran in isolation for multiple iterations those tests were passing all the time. Can you please share the list of tests failing for you with this commit ? I can try to run a sample of those tests in isolation and see if those are really random.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user sohami commented on the issue:

          https://github.com/apache/drill/pull/919

          I rebased this PR to latest master and [re-ran functional test](http://10.10.104.91:8080/job/Functional/2447/console) and seeing only 2 random failures now. As per @agirish recommendation I have created JIRA's to track these random issues. DRILL-5788(https://issues.apache.org/jira/browse/DRILL-5788) & DRILL-5789(https://issues.apache.org/jira/browse/DRILL-5789).

          Show
          githubbot ASF GitHub Bot added a comment - Github user sohami commented on the issue: https://github.com/apache/drill/pull/919 I rebased this PR to latest master and [re-ran functional test] ( http://10.10.104.91:8080/job/Functional/2447/console ) and seeing only 2 random failures now. As per @agirish recommendation I have created JIRA's to track these random issues. DRILL-5788 ( https://issues.apache.org/jira/browse/DRILL-5788 ) & DRILL-5789 ( https://issues.apache.org/jira/browse/DRILL-5789 ).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user paul-rogers commented on the issue:

          https://github.com/apache/drill/pull/919

          Attempt to merge into master failed:

          ```
          CONFLICT (content): Merge conflict in exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentStatusReporter.java
          ```

          Please rebase onto latest master. While at it, please also squash commits.

          Show
          githubbot ASF GitHub Bot added a comment - Github user paul-rogers commented on the issue: https://github.com/apache/drill/pull/919 Attempt to merge into master failed: ``` CONFLICT (content): Merge conflict in exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentStatusReporter.java ``` Please rebase onto latest master. While at it, please also squash commits.

            People

            • Assignee:
              shamirwasia Sorabh Hamirwasia
              Reporter:
              shamirwasia Sorabh Hamirwasia
              Reviewer:
              Parth Chandra
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development