Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3644

OEV should recognize and deal with 0.20.20x opcode versions

    Details

    • Type: Bug Bug
    • Status: Reopened
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0.0-alpha, 3.0.0
    • Fix Version/s: None
    • Component/s: tools
    • Labels:
      None

      Description

      We have some opcode conflicts for edit logs between 0.20.20x (LV -19, -31) vs newer versions. For edit log loading, we dealt with this by forcing users to save namespace on an earlier version before upgrading. But, using a trunk OEV on an older version is useful since the OEV has had so many improvements. It would be nice to be able to specify a flag to the OEV to be able to run on older edit logs.

        Activity

        Todd Lipcon created issue -
        Hide
        Suresh Srinivas added a comment -

        This jira is not necessary. The conflict code was only in 0.20.203. Post upgrade to later releases the conflicting opcode is not used. I am closing than as Won't Fix. Reopen if you disagree.

        Show
        Suresh Srinivas added a comment - This jira is not necessary. The conflict code was only in 0.20.203. Post upgrade to later releases the conflicting opcode is not used. I am closing than as Won't Fix. Reopen if you disagree.
        Suresh Srinivas made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]
        Show
        Suresh Srinivas added a comment - BTW a comment to relevant to my previous comment - https://issues.apache.org/jira/browse/HDFS-1842?focusedCommentId=13021839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13021839
        Hide
        Todd Lipcon added a comment -

        I disagree. There are people running systems with LV -19 which has the conflicted opcodes. Currently if you run OEV on these logs, you end up getting errors because it reads delegation token ops as eg symlink ops. If we don't support OEVing a given LV, we should raise an error.

        Show
        Todd Lipcon added a comment - I disagree. There are people running systems with LV -19 which has the conflicted opcodes. Currently if you run OEV on these logs, you end up getting errors because it reads delegation token ops as eg symlink ops. If we don't support OEVing a given LV, we should raise an error.
        Todd Lipcon made changes -
        Resolution Won't Fix [ 2 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Suresh Srinivas added a comment -

        Todd, can you tell me which apache release the LV -19 is from. Saves me time, since you have already done this analysis.

        Show
        Suresh Srinivas added a comment - Todd, can you tell me which apache release the LV -19 is from. Saves me time, since you have already done this analysis.
        Hide
        Eli Collins added a comment -

        Suresh,

        hadoop-branch-1 $ grep -r LAYOUT_VERSIONS_203 src/
        src/hdfs/org/apache/hadoop/hdfs/server/common/Storage.java:  public static final int[] LAYOUT_VERSIONS_203 = {-19, -31};
        
        Show
        Eli Collins added a comment - Suresh, hadoop-branch-1 $ grep -r LAYOUT_VERSIONS_203 src/ src/hdfs/org/apache/hadoop/hdfs/server/common/Storage.java: public static final int [] LAYOUT_VERSIONS_203 = {-19, -31};
        Hide
        Suresh Srinivas added a comment -

        @Eli, not sure if you saw my previous comment:

        The conflict code was only in 0.20.203. Post upgrade to later releases the conflicting opcode is not used.

        Given that a tool that works with the opcodes seems unnecessary, since the problem is only in 0.20.203 alone. Even the editlog code does not handle these conflicts in 0.20.204. We make users save namespace to work around it.

        Show
        Suresh Srinivas added a comment - @Eli, not sure if you saw my previous comment: The conflict code was only in 0.20.203. Post upgrade to later releases the conflicting opcode is not used. Given that a tool that works with the opcodes seems unnecessary, since the problem is only in 0.20.203 alone. Even the editlog code does not handle these conflicts in 0.20.204. We make users save namespace to work around it.
        Hide
        Todd Lipcon added a comment -

        We make users save namespace to work around it

        That's the same proposal here: we error out with "Sorry, you cannot use the OEV on an edit log from this version of Hadoop." Instead of erroring out with "EofException" while trying to read a symlink target, which makes people think the log itself is corrupt.

        Show
        Todd Lipcon added a comment - We make users save namespace to work around it That's the same proposal here: we error out with "Sorry, you cannot use the OEV on an edit log from this version of Hadoop." Instead of erroring out with "EofException" while trying to read a symlink target, which makes people think the log itself is corrupt.
        Hide
        Suresh Srinivas added a comment -

        That's the same proposal here

        Sorry, that was not clear to me by the title or description. Perhaps we could change them for better clarity.

        Show
        Suresh Srinivas added a comment - That's the same proposal here Sorry, that was not clear to me by the title or description. Perhaps we could change them for better clarity.
        Hide
        Colin Patrick McCabe added a comment -

        That's the same proposal here: we error out with "Sorry, you cannot use the OEV on an edit log from this version of Hadoop." Instead of erroring out with "EofException" while trying to read a symlink target, which makes people think the log itself is corrupt.

        Seems reasonable to me.

        Show
        Colin Patrick McCabe added a comment - That's the same proposal here: we error out with "Sorry, you cannot use the OEV on an edit log from this version of Hadoop." Instead of erroring out with "EofException" while trying to read a symlink target, which makes people think the log itself is corrupt. Seems reasonable to me.
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        10h 48m 1 Suresh Srinivas 12/Jul/12 17:51
        Resolved Resolved Reopened Reopened
        1h 13m 1 Todd Lipcon 12/Jul/12 19:04

          People

          • Assignee:
            Unassigned
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:

              Development