Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1768

Tablet server cannot recover after run out of disk space

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.0.0
    • None
    • None

    Description

      My node has small disk. after upsert into a large table several times, it ran out of disk space. I dropped the table. all nodes dropped the tablet successfully except one. it stuck in a bad state and tablet data are not removed even after restart.
      For dropped table, it shows

      tmp_store_sales	d88f8a7eba974cba8d9c547f69168c6b	hash buckets: (0)	FAILED (TABLET_DATA_DELETED): Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first			Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first
      

      for other tables, it shows

      kudu_store_sales	f539ce77657e431aa5ce4ab7f7375f84	hash buckets: (0, 7)	FAILED (TABLET_DATA_READY): IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)			IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)
      

      attached tablet server log from this node.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            jyu@cloudera.com Juan Yu

            Dates

              Created:
              Updated:

              Issue deployment