Accumulo
  1. Accumulo
  2. ACCUMULO-2645

tablet stuck unloading, and problem is hard to diagnose

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.4
    • Fix Version/s: 1.6.1, 1.7.0
    • Component/s: tserver
    • Labels:
    • Environment:

      very large production cluster, CDH3u5

      Description

      • master failed to balance
      • custom balancer refused to balance while migrations were in place
      • tablet server was not unloading the tablet
      • tablet server was otherwise serving tablets, providing status
      • memory dump determined that there were 21K UnloadTabletHandler objects
      • jstack showed UnloadTabletHandler in Tablet.completeClose, line 2674
      • the last print of the debug "completeClose(safeState=true, completeClose=true) occured 9 days ago
      • there was a query that had been running for 9 days

        Issue Links

          Activity

          ASF subversion and git services made changes -
          Time Spent 40m [ 2400 ] 50m [ 3000 ]
          Worklog Id 18076 [ 18076 ]
          ASF subversion and git services made changes -
          Time Spent 0.5h [ 1800 ] 40m [ 2400 ]
          Worklog Id 18072 [ 18072 ]
          Christopher Tubbs made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Christopher Tubbs made changes -
          Resolution Not a Problem [ 8 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Christopher Tubbs made changes -
          Summary tablet stuck unloading tablet stuck unloading, and problem is hard to diagnose
          ASF subversion and git services made changes -
          Time Spent 20m [ 1200 ] 0.5h [ 1800 ]
          Worklog Id 18065 [ 18065 ]
          Eric Newton made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Not a Problem [ 8 ]
          Eric Newton made changes -
          Fix Version/s 1.6.1 [ 12325441 ]
          ASF subversion and git services made changes -
          Time Spent 10m [ 600 ] 20m [ 1200 ]
          Worklog Id 18053 [ 18053 ]
          ASF subversion and git services made changes -
          Remaining Estimate 0h [ 0 ]
          Time Spent 10m [ 600 ]
          Worklog Id 18052 [ 18052 ]
          Eric Newton made changes -
          Link This issue relates to ACCUMULO-2673 [ ACCUMULO-2673 ]
          Eric Newton made changes -
          Labels newbie
          Eric Newton made changes -
          Fix Version/s 1.7.0 [ 12324607 ]
          Eric Newton made changes -
          Field Original Value New Value
          Description  * master failed to balance
           * custom balancer refused to balance while migrations were in place
           * tablet server was not unloading the tablet
           * tablet server was otherwise serving tablets, providing status
           * memory dump determined that there were 21K UnloadTabletHandler objects
           * jstack showed UnloadTabletHandler in Tablet.completeClose, line 2674
           * the last print of the debug "completeClose(safeState=true, completeClose=true) occured 9 days ago
           * there was a query that had been for 9 days

           * master failed to balance
           * custom balancer refused to balance while migrations were in place
           * tablet server was not unloading the tablet
           * tablet server was otherwise serving tablets, providing status
           * memory dump determined that there were 21K UnloadTabletHandler objects
           * jstack showed UnloadTabletHandler in Tablet.completeClose, line 2674
           * the last print of the debug "completeClose(safeState=true, completeClose=true) occured 9 days ago
           * there was a query that had been running for 9 days

          Eric Newton created issue -

            People

            • Assignee:
              Eric Newton
              Reporter:
              Eric Newton
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 50m
                50m

                  Development