HBase
  1. HBase
  2. HBASE-2

hlog numbers should wrap around when they reach 999

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.2.0
    • Component/s: regionserver
    • Labels:
      None

      Description

      Question about log numbers -> Closing current log writer hdfs://10.0.0.1:9000/gfs_storage/hadoop-root/hbase/log_10.0.0.3_1201900440436_60020/hlog.dat.024
      What happens when the log get to hlog.dat.999

      1. 2-v2.patch
        2 kB
        Bryan Duxbury
      2. 2.patch
        2 kB
        Bryan Duxbury

        Activity

        Hide
        stack added a comment -

        Committed. Thanks for the patch Bryan (And Billy for the speculation)

        Show
        stack added a comment - Committed. Thanks for the patch Bryan (And Billy for the speculation)
        Hide
        Jim Kellerman added a comment -

        Ok, forget the leading zeros and use a long.

        Show
        Jim Kellerman added a comment - Ok, forget the leading zeros and use a long.
        Hide
        Bryan Duxbury added a comment -

        No question that a long is "overkill" for representing the number of .dat files we'll have. However, we selected System.currentTimeMillis() because it would give us a unique, increasing number every time. Storing an additional 7-10 bytes per filename seems like a reasonable tradeoff.

        As far as the question of zero-fill, it's actually pretty moot, because it will only matter when the length of the timestamps increases by a full order of magnitude. For today, that means going from around 1.2 billion seconds since epoch to 10 billion seconds since epoch, which is about 300 years from now. Even then, the problem would only manifest if two log files were created across the 10 billion rollover. Do you think that we can be ok with a 1-second danger zone 300 years from now?

        Show
        Bryan Duxbury added a comment - No question that a long is "overkill" for representing the number of .dat files we'll have. However, we selected System.currentTimeMillis() because it would give us a unique, increasing number every time. Storing an additional 7-10 bytes per filename seems like a reasonable tradeoff. As far as the question of zero-fill, it's actually pretty moot, because it will only matter when the length of the timestamps increases by a full order of magnitude. For today, that means going from around 1.2 billion seconds since epoch to 10 billion seconds since epoch, which is about 300 years from now. Even then, the problem would only manifest if two log files were created across the 10 billion rollover. Do you think that we can be ok with a 1-second danger zone 300 years from now?
        Hide
        Jim Kellerman added a comment -

        Thinking about this some more, using a long is overkill, and to truly maintain sort order, would need to be zero filled from the left.

        A simple integer counter will suffice (although it too needs to be zero filled from the left to preserve sort order). e.g.:

        The following do not sort properly:

        1
        2
        10

        whereas the following do:

        01
        02
        10

        Show
        Jim Kellerman added a comment - Thinking about this some more, using a long is overkill, and to truly maintain sort order, would need to be zero filled from the left. A simple integer counter will suffice (although it too needs to be zero filled from the left to preserve sort order). e.g.: The following do not sort properly: 1 2 10 whereas the following do: 01 02 10
        Hide
        Bryan Duxbury added a comment -

        Build passes locally.

        Show
        Bryan Duxbury added a comment - Build passes locally.
        Hide
        Bryan Duxbury added a comment -

        Forgot to change computeFilename method to accommodate new longer numeric portion.

        Show
        Bryan Duxbury added a comment - Forgot to change computeFilename method to accommodate new longer numeric portion.
        Hide
        Bryan Duxbury added a comment -

        Here's a shot at it.

        Show
        Bryan Duxbury added a comment - Here's a shot at it.
        Hide
        Billy Pearson added a comment -

        not sure why I did that

        Show
        Billy Pearson added a comment - not sure why I did that
        Hide
        Bryan Duxbury added a comment -

        Doesn't appear to be an attachment on this issue... can't really be patch available.

        Show
        Bryan Duxbury added a comment - Doesn't appear to be an attachment on this issue... can't really be patch available.
        Hide
        Billy Pearson added a comment -

        +1 thats a good idea.

        Show
        Billy Pearson added a comment - +1 thats a good idea.
        Hide
        Bryan Duxbury added a comment -

        +1 to that idea.

        Show
        Bryan Duxbury added a comment - +1 to that idea.
        Hide
        Jim Kellerman added a comment -

        Should they wrap? It will make log recovery more complicated. Why not use System.currentTimeMillis() as the version number, so they always sort properly?

        Show
        Jim Kellerman added a comment - Should they wrap? It will make log recovery more complicated. Why not use System.currentTimeMillis() as the version number, so they always sort properly?
        Hide
        Bryan Duxbury added a comment -

        clarified title

        Show
        Bryan Duxbury added a comment - clarified title

          People

          • Assignee:
            Bryan Duxbury
            Reporter:
            Billy Pearson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development