Accumulo
  1. Accumulo
  2. ACCUMULO-2481

Monitor should still work if HDFS + Zookeeper go away

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.4.4, 1.4.5, 1.5.1, 1.6.0
    • Fix Version/s: 1.8.0
    • Component/s: monitor
    • Labels:
      None
    • Environment:

      1.4.5-SNAPSHOT on CDH4.5.0.

      Non-HA cluster with 2 workers and a single Zookeeper server.

      Description

      While preparing to update a test cluster, I accidentally attempted to refresh some monitor pages after I had shut down the underlying HDFS and Zookeeper services. The page simply hung. I've since tried loading several different pages (overview, logs, master view) and all simple hang on the monitor's end. If I restart the underlying cluster, the pages resolve like normal.

      I haven't had a chance to see if this happens on later versions, nor figure out which missing component matters.

      Expected behavior would be for the monitor to still load, albeit with a bunch of alarm bells about everything being off.

        Issue Links

          Activity

          Sean Busbey created issue -
          Hide
          Josh Elser added a comment -

          Yes please!!

          Show
          Josh Elser added a comment - Yes please!!
          Hide
          Sean Busbey added a comment -

          Important note: Accumulo had not been shut down at all (I was planning to check logs for failure notices about HDFS and service stoppage after zookeeper was lost).

          Show
          Sean Busbey added a comment - Important note: Accumulo had not been shut down at all (I was planning to check logs for failure notices about HDFS and service stoppage after zookeeper was lost).
          Hide
          Josh Elser added a comment -

          Yeah, I've run into that before. The monitor tends to pull all of its data ahead of time, and just sit then when it can't get it. Ideally, it would be better to formalize the data the monitor makes into an API that could just be loaded dynamically, but, that's probably a ways off

          Show
          Josh Elser added a comment - Yeah, I've run into that before. The monitor tends to pull all of its data ahead of time, and just sit then when it can't get it. Ideally, it would be better to formalize the data the monitor makes into an API that could just be loaded dynamically, but, that's probably a ways off
          Hide
          Sean Busbey added a comment -

          For completeness, I left pages loading with a shut down hdfs + zk for 10 minutes. After I restarted hdfs and zk services, the monitor pages all loaded.

          logs showed message from most services claiming zk problems and intent to retry. Everyone recovered and appeared to function normally again after.

          Show
          Sean Busbey added a comment - For completeness, I left pages loading with a shut down hdfs + zk for 10 minutes. After I restarted hdfs and zk services, the monitor pages all loaded. logs showed message from most services claiming zk problems and intent to retry. Everyone recovered and appeared to function normally again after.
          Mike Drob made changes -
          Field Original Value New Value
          Affects Version/s 1.5.1 [ 12324399 ]
          Affects Version/s 1.6.0 [ 12322468 ]
          Mike Drob made changes -
          Link This issue is duplicated by ACCUMULO-1723 [ ACCUMULO-1723 ]
          Sean Busbey made changes -
          Fix Version/s 1.5.2 [ 12326272 ]
          Fix Version/s 1.6.1 [ 12325441 ]
          Fix Version/s 1.7.0 [ 12324607 ]
          Fix Version/s 1.4.6 [ 12326350 ]
          Hide
          Andrew George Wells added a comment -

          Issue was declared a duplicate. Also, the duplicated issue claims to be "Resolved".

          Show
          Andrew George Wells added a comment - Issue was declared a duplicate. Also, the duplicated issue claims to be "Resolved".
          Andrew George Wells made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Hide
          Andrew George Wells added a comment -

          I just descoved an annoying thing jira does... dupicate issues are given a status of resolved when that is not the case.

          Show
          Andrew George Wells added a comment - I just descoved an annoying thing jira does... dupicate issues are given a status of resolved when that is not the case.
          Andrew George Wells made changes -
          Resolution Duplicate [ 3 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Andrew George Wells made changes -
          Assignee Andrew George Wells [ agwells ]
          Hide
          Christopher Tubbs added a comment -

          Duplicate issues should link to the issue they duplicate, and be resolved as a duplicate, with no fixVersion (the fixVersion and actual resolution should be the one that was duplicated, the one where the work was actually done).

          Show
          Christopher Tubbs added a comment - Duplicate issues should link to the issue they duplicate, and be resolved as a duplicate, with no fixVersion (the fixVersion and actual resolution should be the one that was duplicated, the one where the work was actually done).
          Josh Elser made changes -
          Parent ACCUMULO-3034 [ 12731106 ]
          Issue Type Bug [ 1 ] Sub-task [ 7 ]
          Josh Elser made changes -
          Fix Version/s 1.5.3 [ 12328662 ]
          Fix Version/s 1.5.2 [ 12326272 ]
          Corey J. Nolet made changes -
          Fix Version/s 1.6.2 [ 12328644 ]
          Fix Version/s 1.6.1 [ 12325441 ]
          Corey J. Nolet made changes -
          Fix Version/s 1.6.2 [ 12328644 ]
          Corey J. Nolet made changes -
          Fix Version/s 1.6.3 [ 12329154 ]
          Christopher Tubbs made changes -
          Parent ACCUMULO-3034 [ 12731106 ]
          Issue Type Sub-task [ 7 ] Bug [ 1 ]
          Christopher Tubbs made changes -
          Link This issue relates to ACCUMULO-3034 [ ACCUMULO-3034 ]
          Josh Elser made changes -
          Fix Version/s 1.8.0 [ 12329879 ]
          Fix Version/s 1.7.1 [ 12329940 ]
          Fix Version/s 1.7.0 [ 12324607 ]
          Christopher Tubbs made changes -
          Fix Version/s 1.5.3 [ 12328662 ]
          Christopher Tubbs made changes -
          Fix Version/s 1.6.3 [ 12329154 ]
          Fix Version/s 1.7.1 [ 12329940 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          72d 16h 42m 1 Andrew George Wells 26/May/14 15:03
          Resolved Resolved Reopened Reopened
          1m 17s 1 Andrew George Wells 26/May/14 15:04

            People

            • Assignee:
              Andrew George Wells
              Reporter:
              Sean Busbey
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development