Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.2.0, 0.3.0
    • Component/s: Data Processors
    • Labels:
      None
    • Environment:

      Redhat EL 5,1, Java 6

      Description

      The current mr_job table, jobconf columns contains the content of jobconf for some rows, and other rows contains the
      path of the jobconf.

      In the sequence file, jobconf is the field for location of the jobconf path. The content of the jobconf is
      JOBCONF-JSON. In database table, jobconf is the content of the jobconf. Unfortunately, the new automatic mapping will
      detect sequence file "jobconf" field is a match with "jobconf" column in the database. Hence the
      last update wins.

      To work around this problem, we need ability to undefine sequence file field to match to database column.

      Personally, I think indirect data translation mapping is the wrong thing to do. The mapping should be as straight forward as possible.

      1. CHUKWA-316.patch
        1 kB
        Eric Yang
      2. CHUKWA-316-1.patch
        1 kB
        Eric Yang

        Activity

        Hide
        eyang Eric Yang added a comment -

        The current behavior for loading sequence file to database:

        SQ DB
        K1 ------> K1 (If not defined in the dictionary, the default means load to database, if the field name matches)
        K2 ------> K1
        K3 ------> K1

        In order to avoid hidden meaning, to avoid the default rule to load, we should explicit define:

        K1 ------> ""
        K2 ------> K1
        K3 ------> K1

        Show
        eyang Eric Yang added a comment - The current behavior for loading sequence file to database: SQ DB K1 ------> K1 (If not defined in the dictionary, the default means load to database, if the field name matches) K2 ------> K1 K3 ------> K1 In order to avoid hidden meaning, to avoid the default rule to load, we should explicit define: K1 ------> "" K2 ------> K1 K3 ------> K1
        Hide
        terencekwan Terence Kwan added a comment -

        look good. +1

        Show
        terencekwan Terence Kwan added a comment - look good. +1
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Terence.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Terence.
        Hide
        eyang Eric Yang added a comment -

        When configuration is specified like this:

        <property>
        <name>metric.jobdata.jobconf</name>
        <value></value>
        <description>Explicitly undefined jobconf path to load to database</description>
        </property>

        The configuration object will filter out this key. As a workaround, use configuration like:

        <property>
        <name>metric.jobdata.jobconf</name>
        <value>_delete</value>
        <description>Explicitly undefined jobconf path to load to database</description>
        </property>

        Show
        eyang Eric Yang added a comment - When configuration is specified like this: <property> <name>metric.jobdata.jobconf</name> <value></value> <description>Explicitly undefined jobconf path to load to database</description> </property> The configuration object will filter out this key. As a workaround, use configuration like: <property> <name>metric.jobdata.jobconf</name> <value>_delete</value> <description>Explicitly undefined jobconf path to load to database</description> </property>
        Hide
        eyang Eric Yang added a comment -

        Replace empty value with _delete to remove it manually from MDL translation.

        Show
        eyang Eric Yang added a comment - Replace empty value with _delete to remove it manually from MDL translation.
        Hide
        terencekwan Terence Kwan added a comment -

        +1.

        Show
        terencekwan Terence Kwan added a comment - +1.
        Hide
        eyang Eric Yang added a comment -

        I just committed this, thanks Terence.

        Show
        eyang Eric Yang added a comment - I just committed this, thanks Terence.
        Hide
        hudson Hudson added a comment -

        Integrated in Chukwa-trunk #61 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/)
        . Fix MetricDataLoader to use "_delete" keyword for removing columns. (Eric Yang)

        Show
        hudson Hudson added a comment - Integrated in Chukwa-trunk #61 (See http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/61/ ) . Fix MetricDataLoader to use "_delete" keyword for removing columns. (Eric Yang)

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            eyang Eric Yang
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development