HBase
  1. HBase
  2. HBASE-2

hlog numbers should wrap around when they reach 999

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2.0
    • Component/s: regionserver
    • Labels:
      None

      Description

      Question about log numbers -> Closing current log writer hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/log_10.0.0.3_1201900440436_60020/hlog.dat.024
      What happens when the log get to hlog.dat.999

      1. 2-v2.patch
        2 kB
        Bryan Duxbury
      2. 2.patch
        2 kB
        Bryan Duxbury

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        1d 1h 3m 1 Bryan Duxbury 06/Feb/08 00:37
        Open Open Patch Available Patch Available
        4d 2h 42m 2 Bryan Duxbury 07/Feb/08 01:19
        Patch Available Patch Available Resolved Resolved
        20h 30m 1 stack 07/Feb/08 21:49
        Resolved Resolved Closed Closed
        196d 23h 23m 1 Jim Kellerman 22/Aug/08 22:13
        Jim Kellerman made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        stack made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hide
        stack added a comment -

        Committed. Thanks for the patch Bryan (And Billy for the speculation)

        Show
        stack added a comment - Committed. Thanks for the patch Bryan (And Billy for the speculation)
        Hide
        Jim Kellerman added a comment -

        Ok, forget the leading zeros and use a long.

        Show
        Jim Kellerman added a comment - Ok, forget the leading zeros and use a long.
        Hide
        Bryan Duxbury added a comment -

        No question that a long is "overkill" for representing the number of .dat files we'll have. However, we selected System.currentTimeMillis() because it would give us a unique, increasing number every time. Storing an additional 7-10 bytes per filename seems like a reasonable tradeoff.

        As far as the question of zero-fill, it's actually pretty moot, because it will only matter when the length of the timestamps increases by a full order of magnitude. For today, that means going from around 1.2 billion seconds since epoch to 10 billion seconds since epoch, which is about 300 years from now. Even then, the problem would only manifest if two log files were created across the 10 billion rollover. Do you think that we can be ok with a 1-second danger zone 300 years from now?

        Show
        Bryan Duxbury added a comment - No question that a long is "overkill" for representing the number of .dat files we'll have. However, we selected System.currentTimeMillis() because it would give us a unique, increasing number every time. Storing an additional 7-10 bytes per filename seems like a reasonable tradeoff. As far as the question of zero-fill, it's actually pretty moot, because it will only matter when the length of the timestamps increases by a full order of magnitude. For today, that means going from around 1.2 billion seconds since epoch to 10 billion seconds since epoch, which is about 300 years from now. Even then, the problem would only manifest if two log files were created across the 10 billion rollover. Do you think that we can be ok with a 1-second danger zone 300 years from now?
        Hide
        Jim Kellerman added a comment -

        Thinking about this some more, using a long is overkill, and to truly maintain sort order, would need to be zero filled from the left.

        A simple integer counter will suffice (although it too needs to be zero filled from the left to preserve sort order). e.g.:

        The following do not sort properly:

        1
        2
        10

        whereas the following do:

        01
        02
        10

        Show
        Jim Kellerman added a comment - Thinking about this some more, using a long is overkill, and to truly maintain sort order, would need to be zero filled from the left. A simple integer counter will suffice (although it too needs to be zero filled from the left to preserve sort order). e.g.: The following do not sort properly: 1 2 10 whereas the following do: 01 02 10
        Bryan Duxbury made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Bryan Duxbury [ bryanduxbury ]
        Hide
        Bryan Duxbury added a comment -

        Build passes locally.

        Show
        Bryan Duxbury added a comment - Build passes locally.
        Bryan Duxbury made changes -
        Attachment 2-v2.patch [ 12374930 ]
        Hide
        Bryan Duxbury added a comment -

        Forgot to change computeFilename method to accommodate new longer numeric portion.

        Show
        Bryan Duxbury added a comment - Forgot to change computeFilename method to accommodate new longer numeric portion.
        Bryan Duxbury made changes -
        Attachment 2.patch [ 12374928 ]
        Hide
        Bryan Duxbury added a comment -

        Here's a shot at it.

        Show
        Bryan Duxbury added a comment - Here's a shot at it.
        Hide
        Billy Pearson added a comment -

        not sure why I did that

        Show
        Billy Pearson added a comment - not sure why I did that
        Bryan Duxbury made changes -
        Fix Version/s 0.2.0 [ 12312955 ]
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Bryan Duxbury added a comment -

        Doesn't appear to be an attachment on this issue... can't really be patch available.

        Show
        Bryan Duxbury added a comment - Doesn't appear to be an attachment on this issue... can't really be patch available.
        Billy Pearson made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Billy Pearson added a comment -

        +1 thats a good idea.

        Show
        Billy Pearson added a comment - +1 thats a good idea.
        Bryan Duxbury made changes -
        Component/s regionserver [ 12312139 ]
        Hide
        Bryan Duxbury added a comment -

        +1 to that idea.

        Show
        Bryan Duxbury added a comment - +1 to that idea.
        Hide
        Jim Kellerman added a comment -

        Should they wrap? It will make log recovery more complicated. Why not use System.currentTimeMillis() as the version number, so they always sort properly?

        Show
        Jim Kellerman added a comment - Should they wrap? It will make log recovery more complicated. Why not use System.currentTimeMillis() as the version number, so they always sort properly?
        Bryan Duxbury made changes -
        Field Original Value New Value
        Summary hlog numbers hlog numbers should wrap around when they reach 999
        Hide
        Bryan Duxbury added a comment -

        clarified title

        Show
        Bryan Duxbury added a comment - clarified title
        Billy Pearson created issue -

          People

          • Assignee:
            Bryan Duxbury
            Reporter:
            Billy Pearson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development