Accumulo
  1. Accumulo
  2. ACCUMULO-1637

Update HDFS append/sync precondition check for Hadoop 1.2

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.5.1, 1.6.0
    • Component/s: tserver
    • Labels:
      None

      Description

      Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

      This can be worked around by specifying the old durable sync property in hdfs-site.xml:

      <property>
        <name>dfs.durable.sync</name>
        <value>true</value>
      </property>
      

      I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

      Thanks to Carlos Mundi for pointing this out on user@a.a.o

      Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.

      Version DFSConfigKeys.DFS_SUPPORT_APPEND_KEY DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT "dfs.durable.sync" Specific Configuration Required
      0.20.205.0 defined false not present yes
      0.23.x defined true not present no
      1.0.x defined false not present yes
      1.1.X not present absent implicit "true" no
      1.2.X not present absent implicit "true" no
      2.0.x defined true not present no
      2.1.x defined true not present no
      2.2.0 defined true not present no

        Activity

        Josh Elser created issue -
        Keith Turner made changes -
        Field Original Value New Value
        Fix Version/s 1.6.0 [ 12322468 ]
        Josh Elser made changes -
        Assignee Josh Elser [ elserj ]
        Josh Elser made changes -
        Description Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o
        Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys
        ||Version||dfs.support.append||dfs.durable.sync||
        |0.20.205.0|default:false|not present|
        |0.23.x|default:true|not present|
        |1.0.x|default:false|not present|
        |1.1.x|not present|default:true|
        |1.1.2|not present|not present|
        |2.0.x|default:true|not present|
        |2.1.x|default:true|not present|
        |2.2.0|default:true|not present|
        Josh Elser made changes -
        Description Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys
        ||Version||dfs.support.append||dfs.durable.sync||
        |0.20.205.0|default:false|not present|
        |0.23.x|default:true|not present|
        |1.0.x|default:false|not present|
        |1.1.x|not present|default:true|
        |1.1.2|not present|not present|
        |2.0.x|default:true|not present|
        |2.1.x|default:true|not present|
        |2.2.0|default:true|not present|
        Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys
        ||Version||dfs.support.append||dfs.durable.sync||
        |0.20.205.0|default:false|not present|
        |0.23.x|default:true|not present|
        |1.0.x|default:false|not present|
        |1.1.X|not present|not present|
        |2.0.x|default:true|not present|
        |2.1.x|default:true|not present|
        |2.2.0|default:true|not present|
        Josh Elser made changes -
        Description Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys
        ||Version||dfs.support.append||dfs.durable.sync||
        |0.20.205.0|default:false|not present|
        |0.23.x|default:true|not present|
        |1.0.x|default:false|not present|
        |1.1.X|not present|not present|
        |2.0.x|default:true|not present|
        |2.1.x|default:true|not present|
        |2.2.0|default:true|not present|
        Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||
        |0.20.205.0|defined|false|not present|
        |0.23.x|defined|true|not present|
        |1.0.x|defined|uhhhh? (according to current message)|not present|
        |1.0.4|defined|false|not present|
        |1.1.X|not present|absent|not present|
        |2.0.x|defined|true|not present|
        |2.1.x|defined|true|not present|
        |2.2.0|defined|true|not present|
        Josh Elser made changes -
        Description Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||
        |0.20.205.0|defined|false|not present|
        |0.23.x|defined|true|not present|
        |1.0.x|defined|uhhhh? (according to current message)|not present|
        |1.0.4|defined|false|not present|
        |1.1.X|not present|absent|not present|
        |2.0.x|defined|true|not present|
        |2.1.x|defined|true|not present|
        |2.2.0|defined|true|not present|
        Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||
        |0.20.205.0|defined|false|not present|
        |0.23.x|defined|true|not present|
        |1.0.x|defined|uhhhh? (according to current message)|not present|
        |1.0.4|defined|false|not present|
        |1.1.X|not present|absent|implicit "true"|
        |2.0.x|defined|true|not present|
        |2.1.x|defined|true|not present|
        |2.2.0|defined|true|not present|
        Josh Elser made changes -
        Description Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||
        |0.20.205.0|defined|false|not present|
        |0.23.x|defined|true|not present|
        |1.0.x|defined|uhhhh? (according to current message)|not present|
        |1.0.4|defined|false|not present|
        |1.1.X|not present|absent|implicit "true"|
        |2.0.x|defined|true|not present|
        |2.1.x|defined|true|not present|
        |2.2.0|defined|true|not present|
        Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||Specific Configuration Required||
        |0.20.205.0|defined|false|not present|yes|
        |0.23.x|defined|true|not present|no|
        |1.0.x|defined|uhhhh? (according to current message)|not present|yes|
        |1.0.4|defined|false|not present|yes|
        |1.1.X|not present|absent|implicit "true"|no|
        |2.0.x|defined|true|not present|no|
        |2.1.x|defined|true|not present|no|
        |2.2.0|defined|true|not present|no|
        Josh Elser made changes -
        Description Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||Specific Configuration Required||
        |0.20.205.0|defined|false|not present|yes|
        |0.23.x|defined|true|not present|no|
        |1.0.x|defined|uhhhh? (according to current message)|not present|yes|
        |1.0.4|defined|false|not present|yes|
        |1.1.X|not present|absent|implicit "true"|no|
        |2.0.x|defined|true|not present|no|
        |2.1.x|defined|true|not present|no|
        |2.2.0|defined|true|not present|no|
        Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||Specific Configuration Required||
        |0.20.205.0|defined|false|not present|yes|
        |0.23.x|defined|true|not present|no|
        |1.0.x|defined|false|not present|yes|
        |1.1.X|not present|absent|implicit "true"|no|
        |2.0.x|defined|true|not present|no|
        |2.1.x|defined|true|not present|no|
        |2.2.0|defined|true|not present|no|
        Josh Elser made changes -
        Description Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||Specific Configuration Required||
        |0.20.205.0|defined|false|not present|yes|
        |0.23.x|defined|true|not present|no|
        |1.0.x|defined|false|not present|yes|
        |1.1.X|not present|absent|implicit "true"|no|
        |2.0.x|defined|true|not present|no|
        |2.1.x|defined|true|not present|no|
        |2.2.0|defined|true|not present|no|
        Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync is on by default.

        This can be worked around by specifying the old durable sync property in hdfs-site.xml:

        {noformat}
        <property>
          <name>dfs.durable.sync</name>
          <value>true</value>
        </property>
        {noformat}

        I'm not sure how to best way to address the differences between the newer and older versions of Hadoop and their differing append support.

        Thanks to Carlos Mundi for pointing this out on user@a.a.o

        Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys and from the codebase when there is no config parameter for it.
        ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||Specific Configuration Required||
        |0.20.205.0|defined|false|not present|yes|
        |0.23.x|defined|true|not present|no|
        |1.0.x|defined|false|not present|yes|
        |1.1.X|not present|absent|implicit "true"|no|
        |1.2.X|not present|absent|implicit "true"|no|
        |2.0.x|defined|true|not present|no|
        |2.1.x|defined|true|not present|no|
        |2.2.0|defined|true|not present|no|
        Josh Elser made changes -
        Josh Elser made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Josh Elser made changes -
        Status Resolved [ 5 ] Reopened [ 4 ]
        Resolution Fixed [ 1 ]
        Josh Elser made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Josh Elser made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Josh Elser made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Josh Elser
            Reporter:
            Josh Elser
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development