Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5054

Chukwa database partitions does not create/expire automatically

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Redhat EL 5.1, Java 6

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      What is new in HADOOP-5054:

      - Improved database partitioning by date.
      Show
      What is new in HADOOP-5054 : - Improved database partitioning by date.

      Description

      dbAdmin.sh should create database partitions base on time, but this is not working because the timestamp variable is not inside the loop for creating database partitions. When the script is running, it would create partitions corresponding to the execution time once only.

      The database expiration script need to handle the case where partition number is invalid.

        Activity

        Hide
        hudson Hudson added a comment -
        Show
        hudson Hudson added a comment - Integrated in Hadoop-trunk #778 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/ )
        Hide
        eyang Eric Yang added a comment -

        I just committed this. Thanks guys.

        Show
        eyang Eric Yang added a comment - I just committed this. Thanks guys.
        Hide
        asrabkin Ari Rabkin added a comment -

        Okiedoke, I'm convinced.

        Show
        asrabkin Ari Rabkin added a comment - Okiedoke, I'm convinced.
        Hide
        jboulon Jerome Boulon added a comment -

        I put a +1 even if the number of partition is hardcoded since it fixes some critical bugs.
        Adding the number of partitions to the config requires to revalidate the current logic so it's better to open a Jira request just for it if needed.

        Show
        jboulon Jerome Boulon added a comment - I put a +1 even if the number of partition is hardcoded since it fixes some critical bugs. Adding the number of partitions to the config requires to revalidate the current logic so it's better to open a Jira request just for it if needed.
        Hide
        eyang Eric Yang added a comment -

        The number of partitions to keep is hard coded for now. If more people show interest that "the number of partitions to keep" should be configurable, then we move this to config file.

        Show
        eyang Eric Yang added a comment - The number of partitions to keep is hard coded for now. If more people show interest that "the number of partitions to keep" should be configurable, then we move this to config file.
        Hide
        asrabkin Ari Rabkin added a comment -

        It looks like this patch hardcodes the number of partitions to keep. Can we push this into config? Should we?

        Show
        asrabkin Ari Rabkin added a comment - It looks like this patch hardcodes the number of partitions to keep. Can we push this into config? Should we?
        Hide
        macyang Mac Yang added a comment -

        Yes, have been running this in our deployment.

        Show
        macyang Mac Yang added a comment - Yes, have been running this in our deployment.
        Hide
        asrabkin Ari Rabkin added a comment -

        Looks okay. I'm a bit nervous about the lack of tests, though I understand that it's hard to unit-test this sort of thing. Has it been tried out in production?

        Show
        asrabkin Ari Rabkin added a comment - Looks okay. I'm a bit nervous about the lack of tests, though I understand that it's hard to unit-test this sort of thing. Has it been tried out in production?
        Hide
        eyang Eric Yang added a comment -
        • Fixed shell script with date generation.
        • Fixed data expiration for partition smaller than positive number.
        • Added debug statement for Table Creation.
        Show
        eyang Eric Yang added a comment - Fixed shell script with date generation. Fixed data expiration for partition smaller than positive number. Added debug statement for Table Creation.
        Hide
        eyang Eric Yang added a comment -

        Automatic partition expiration is also not working correctly.

        Show
        eyang Eric Yang added a comment - Automatic partition expiration is also not working correctly.
        Hide
        eyang Eric Yang added a comment -

        Moved date generation into the shell script while loop.

        Show
        eyang Eric Yang added a comment - Moved date generation into the shell script while loop.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development