Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-1374

Sudden Death of master, gc, and tservers

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: gc, master, tserver
    • Labels:
      None
    • Environment:

      1.5, svn#1470047 & 1477382 - both in standalone instance on ec2 on ubuntu and small cluster on bare metal CentOs

      Description

      I wish I could provide more information. This has happened once on a bare metal centos cluster while running vanilla continuous ingest of svn#1470047. There was nothing reported in the logs when one of the tservers just died after the system had been up for ~1 day. The out and err files were sparse, and the master only reported that it had lost connection with the tserver at the point when the tserver just stopped logging (it was overnight, so this was not witnessed until morning).

      It recently happened again on a standalone instance on ec2 running ubuntu and svn#1477382. The instance had been running for ~7 hours. This time the gc, master, and tserver died. The gc died first, and then 2m:48s later the master died. 200ms later the tserver died. Again, there was no output in any of the out or err files for the processes. The logs also have no errors or warnings in them, just abrupt stops. The processes came up fine once restarted.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vines John Vines
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: