Details
-
Type:
Improvement
-
Status: Open
-
Priority:
Blocker
-
Resolution: Unresolved
-
Affects Version/s: 3.0.0-alpha1, 3.0.0-alpha2
-
Fix Version/s: None
-
Component/s: build
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Incompatible change
Description
Standard protobuf 2.5.0 does not work properly on many platforms. (See, for example, https://gist.github.com/BennettSmith/7111094 ). In order for us to avoid crazy work arounds in the build environment and the fact that 2.5.0 is starting to slowly disappear as a standard install-able package for even Linux/x86, we need to either upgrade or self bundle or something else.
Issue Links
- is blocked by
-
HADOOP-13070
classloading isolation improvements for stricter dependencies
-
- Open
-
- is depended upon by
-
HADOOP-9991
Fix up Hadoop POMs, roll up JARs to latest versions
-
- Open
-
- is duplicated by
-
HDFS-11010
Update to Protobuf 3 in trunk
-
- Resolved
-
- is related to
-
HADOOP-14284
Shade Guava everywhere
-
- Open
-
- relates to
-
HBASE-15638
Shade protobuf
-
- Resolved
-
-
HDFS-11010
Update to Protobuf 3 in trunk
-
- Resolved
-
-
HADOOP-9845
Update protobuf to 2.5 from 2.4.x
-
- Closed
-
Show 2 more links (2 relates to)
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 6m 38s | trunk passed |
| +1 | compile | 0m 9s | trunk passed |
| +1 | mvnsite | 0m 11s | trunk passed |
| +1 | mvneclipse | 0m 9s | trunk passed |
| +1 | javadoc | 0m 9s | trunk passed |
| +1 | mvninstall | 0m 7s | the patch passed |
| +1 | compile | 0m 6s | the patch passed |
| +1 | javac | 0m 6s | the patch passed |
| +1 | mvnsite | 0m 8s | the patch passed |
| +1 | mvneclipse | 0m 7s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 6s | the patch passed |
| +1 | unit | 0m 6s | hadoop-project in the patch passed. |
| +1 | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| 9m 5s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12816966/HADOOP-13363.001.patch |
| JIRA Issue | HADOOP-13363 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
| uname | Linux 0f4f7012a335 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 932aed6 |
| Default Java | 1.8.0_91 |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/9949/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9949/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks Allen for reporting this and thanks Tsuyoshi for the initial patch. We need to update BUILDING.txt as well.
We also need to update the Dockerfile to pull in protobuf 2.6.x, which may end up being tricky.
This merits a discussion on common-dev for visibility. There was a discussion before but no conclusion iirc.
I think it is fine as long as it does not break wire compatibility and we can shade the dependency, to be kind to downstream projects.
What is going to happen when code generated by protoc 2.5 tries to run against 2.6 library? Has anyone tested this? Because thats the metric of how traumatic this is going to be
This merits a discussion on common-dev for visibility. There was a discussion before but no conclusion iirc.
Sure.
What is going to happen when code generated by protoc 2.5 tries to run against 2.6 library? Has anyone tested this? Because thats the metric of how traumatic this is going to be
We must test it before merging it, of course. IIUC, protobuf is wire compatible between 2.5 and 2.6. I found a test suite which seems useful here: https://chromium.googlesource.com/external/github.com/google/protobuf/+/master/java/compatibility_tests/README.md
We also need to check code-level compatibility.
wire-level backward compatibility is a core feature of protobuf, I'm not worried there. What I am worried about is compile-compat, as that is what broke hadoop, hbase, etc, everything using protobuf 2.4. Google's internal build process is a clean, unified build of everything, so they don't have to worry about source level compatibliity
Thank you for the clarification, Steve. I got the point. This kind of "updating dependency work" is related to the classpath isolation work(HADOOP-13070), so I'd like to start discussion on ML.
Did a ML discussion happen?
pb3.1.0 is out. It runs in a 2.5.0 compatibility mode by default. Has some facility for saving on data copying that might be of interest in the NN. If upgrading, you need to run the newer protoc. Newer lib can't read the protos made by older protoc (IIRC). Newer protoc, in my experience, has no problem digesting pb 2.5.0 .proto files. The generated files are a little different, not consumable by the old protobuf lib.
Would this be a problem? Old clients can talk to the new servers because of wire compatible. Is anyone consuming hadoop protos directly other than hadoop? Are hadoop proto files considered InterfaceAudience.Private or InterfaceAudience.Public? If the former, I could work on a patch for 3.0.0 (It'd be big but boring). Does Hadoop have Protobuf in its API anywhere (I can take a look but being lazy asking here first).
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 7m 7s | trunk passed |
| +1 | compile | 0m 8s | trunk passed |
| +1 | mvnsite | 0m 11s | trunk passed |
| +1 | mvneclipse | 0m 9s | trunk passed |
| +1 | javadoc | 0m 9s | trunk passed |
| +1 | mvninstall | 0m 7s | the patch passed |
| +1 | compile | 0m 6s | the patch passed |
| +1 | javac | 0m 6s | the patch passed |
| +1 | mvnsite | 0m 9s | the patch passed |
| +1 | mvneclipse | 0m 6s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 7s | the patch passed |
| +1 | unit | 0m 6s | hadoop-project in the patch passed. |
| +1 | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| 9m 35s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13363 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12816966/HADOOP-13363.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
| uname | Linux 714e389ffa51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c62ae71 |
| Default Java | 1.8.0_101 |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/10826/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10826/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 22s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 16s | Maven dependency ordering for branch |
| +1 | mvninstall | 14m 29s | trunk passed |
| +1 | compile | 17m 30s | trunk passed |
| +1 | checkstyle | 1m 57s | trunk passed |
| +1 | mvnsite | 1m 14s | trunk passed |
| +1 | mvneclipse | 0m 36s | trunk passed |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| +1 | findbugs | 1m 52s | trunk passed |
| +1 | javadoc | 1m 6s | trunk passed |
| 0 | mvndep | 0m 19s | Maven dependency ordering for patch |
| -1 | mvninstall | 0m 9s | hadoop-hdfs in the patch failed. |
| -1 | compile | 0m 24s | root in the patch failed. |
| -1 | javac | 0m 24s | root in the patch failed. |
| +1 | checkstyle | 1m 48s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| -1 | mvnsite | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | mvneclipse | 0m 11s | hadoop-hdfs in the patch failed. |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 13s | hadoop-hdfs in the patch failed. |
| -1 | javadoc | 0m 13s | hadoop-hdfs in the patch failed. |
| +1 | unit | 0m 12s | hadoop-project in the patch passed. |
| -1 | unit | 0m 13s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 68m 34s |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 16s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 | mvninstall | 12m 46s | trunk passed |
| +1 | compile | 16m 4s | trunk passed |
| +1 | checkstyle | 1m 54s | trunk passed |
| +1 | mvnsite | 1m 16s | trunk passed |
| +1 | mvneclipse | 0m 37s | trunk passed |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| +1 | findbugs | 1m 53s | trunk passed |
| +1 | javadoc | 1m 5s | trunk passed |
| 0 | mvndep | 0m 19s | Maven dependency ordering for patch |
| -1 | mvninstall | 0m 9s | hadoop-hdfs in the patch failed. |
| -1 | compile | 0m 25s | root in the patch failed. |
| -1 | javac | 0m 25s | root in the patch failed. |
| +1 | checkstyle | 1m 50s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| -1 | mvnsite | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | mvneclipse | 0m 11s | hadoop-hdfs in the patch failed. |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | javadoc | 0m 12s | hadoop-hdfs in the patch failed. |
| +1 | unit | 0m 13s | hadoop-project in the patch passed. |
| -1 | unit | 0m 12s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 65m 23s |
This message was automatically generated.
Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha3-SNAPSHOT:protoc (compile-protoc)
i.e., dev-support/docker/Dockerfile needs to be updated to use the same version of protobuf that's in the maven pom.xml file.
(It might also be time to move to Xenial from Trusty. I think I've got a working Dockerfile around here that does that too.)
Attaching a patch to update Dockerfile to download protobuf 3.2.0.
Thanks Allen! Updated.
@stack unfortunately, not yet. I've also checked discussion on HDFS-11010. Based on the discussion on the jira and here, I would like to start next argument.
A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/11938/console in case of problems.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 16m 41s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 17s | Maven dependency ordering for branch |
| -1 | mvninstall | 0m 32s | root in trunk failed. |
| -1 | compile | 0m 28s | root in trunk failed. |
| +1 | checkstyle | 1m 56s | trunk passed |
| -1 | mvnsite | 0m 11s | hadoop-hdfs in trunk failed. |
| -1 | mvneclipse | 0m 9s | hadoop-hdfs in trunk failed. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 11s | hadoop-hdfs in trunk failed. |
| -1 | javadoc | 0m 11s | hadoop-hdfs in trunk failed. |
| 0 | mvndep | 0m 22s | Maven dependency ordering for patch |
| -1 | mvninstall | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | compile | 0m 28s | root in the patch failed. |
| -1 | javac | 0m 28s | root in the patch failed. |
| +1 | checkstyle | 1m 57s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| -1 | mvnsite | 0m 15s | hadoop-hdfs in the patch failed. |
| -1 | mvneclipse | 0m 15s | hadoop-hdfs in the patch failed. |
| +1 | shellcheck | 0m 10s | There were no new shellcheck issues. |
| +1 | shelldocs | 0m 9s | There were no new shelldocs issues. |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 2s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 16s | hadoop-hdfs in the patch failed. |
| -1 | javadoc | 0m 17s | hadoop-hdfs in the patch failed. |
| +1 | unit | 0m 16s | hadoop-project in the patch passed. |
| -1 | unit | 0m 18s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| 51m 29s |
This message was automatically generated.
Trying to fix a problem by lack of execution permission.
A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/11939/console in case of problems.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 14m 22s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 15s | Maven dependency ordering for branch |
| -1 | mvninstall | 0m 27s | root in trunk failed. |
| -1 | compile | 0m 24s | root in trunk failed. |
| +1 | checkstyle | 1m 49s | trunk passed |
| -1 | mvnsite | 0m 10s | hadoop-hdfs in trunk failed. |
| -1 | mvneclipse | 0m 10s | hadoop-hdfs in trunk failed. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 12s | hadoop-hdfs in trunk failed. |
| -1 | javadoc | 0m 13s | hadoop-hdfs in trunk failed. |
| 0 | mvndep | 0m 19s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 5s | the patch passed |
| +1 | compile | 17m 35s | the patch passed |
| -1 | javac | 17m 35s | root generated 1232 new + 0 unchanged - 0 fixed = 1232 total (was 0) |
| +1 | checkstyle | 2m 5s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| +1 | mvnsite | 1m 37s | the patch passed |
| +1 | mvneclipse | 0m 58s | the patch passed |
| -1 | shellcheck | 0m 16s | The patch generated 632 new + 104 unchanged - 0 fixed = 736 total (was 104) |
| -0 | shelldocs | 0m 21s | The patch generated 282 new + 104 unchanged - 0 fixed = 386 total (was 104) |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| +1 | findbugs | 2m 13s | the patch passed |
| -1 | javadoc | 1m 12s | hadoop-hdfs-project_hadoop-hdfs generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) |
| +1 | unit | 0m 29s | hadoop-project in the patch passed. |
| -1 | unit | 54m 5s | hadoop-hdfs in the patch failed. |
| -1 | asflicense | 0m 51s | The patch generated 1 ASF License warnings. |
| 128m 8s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| hadoop.hdfs.TestHDFSFileSystemContract | |
| hadoop.hdfs.TestErasureCodingPolicyWithSnapshot | |
| hadoop.hdfs.TestEncryptedTransfer | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport | |
| hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks | |
| hadoop.hdfs.TestBlockStoragePolicy | |
| hadoop.hdfs.TestRollingUpgrade | |
| hadoop.hdfs.web.TestWebHDFSForHA | |
| hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | |
| hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | |
| hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStream | |
| hadoop.hdfs.server.blockmanagement.TestPendingReconstruction | |
| hadoop.hdfs.TestClientProtocolForPipelineRecovery | |
| hadoop.hdfs.server.namenode.TestFSEditLogLoader | |
| hadoop.fs.contract.hdfs.TestHDFSContractSeek | |
| hadoop.hdfs.TestPread | |
| hadoop.fs.viewfs.TestViewFsFileStatusHdfs | |
| hadoop.hdfs.TestWriteConfigurationToDFS | |
| hadoop.cli.TestAclCLIWithPosixAclInheritance | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | |
| hadoop.hdfs.TestParallelShortCircuitLegacyRead | |
| hadoop.hdfs.TestFileCreationClient | |
| hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes | |
| hadoop.hdfs.TestLeaseRecoveryStriped | |
| hadoop.hdfs.server.diskbalancer.TestDiskBalancer | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation | |
| hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy | |
| hadoop.hdfs.crypto.TestHdfsCryptoStreams | |
| hadoop.hdfs.server.datanode.TestTransferRbw | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks | |
| hadoop.hdfs.TestWriteBlockGetsBlockLengthHint | |
| hadoop.hdfs.server.namenode.TestAddBlock | |
| hadoop.hdfs.TestDatanodeDeath | |
| hadoop.fs.TestSymlinkHdfsDisable | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | |
| hadoop.hdfs.TestDFSInotifyEventInputStream | |
| hadoop.hdfs.TestDecommission | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary | |
| hadoop.hdfs.server.namenode.TestCommitBlockWithInvalidGenStamp | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics | |
| hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead | |
| hadoop.hdfs.TestWriteReadStripedFile | |
| hadoop.hdfs.server.mover.TestMover | |
| hadoop.hdfs.TestDFSRemove | |
| hadoop.hdfs.TestAclsEndToEnd | |
| hadoop.fs.loadGenerator.TestLoadGenerator | |
| hadoop.hdfs.TestFileCorruption | |
| hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication | |
| hadoop.hdfs.server.datanode.TestDataNodeTcpNoDelay | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter | |
| hadoop.hdfs.TestFileAppend4 | |
| hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC | |
| hadoop.hdfs.server.namenode.ha.TestDNFencing | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength | |
| hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport | |
| hadoop.security.TestPermission | |
| hadoop.hdfs.TestDFSShell | |
| hadoop.hdfs.server.blockmanagement.TestNodeCount | |
| hadoop.hdfs.server.namenode.TestDeleteRace | |
| hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication | |
| hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade | |
| hadoop.hdfs.web.TestWebHDFSXAttr | |
| hadoop.hdfs.TestDFSClientFailover | |
| hadoop.hdfs.TestModTime | |
| hadoop.hdfs.TestDistributedFileSystem | |
| hadoop.hdfs.web.TestWebHDFS | |
| hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot | |
| hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles | |
| hadoop.fs.TestWebHdfsFileContextMainOperations | |
| hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot | |
| hadoop.hdfs.TestFileCreationDelete | |
| hadoop.hdfs.TestDFSUpgrade | |
| hadoop.hdfs.TestParallelRead | |
| hadoop.hdfs.TestTrashWithEncryptionZones | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 | |
| hadoop.security.TestPermissionSymlinks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | |
| hadoop.hdfs.server.blockmanagement.TestBlockManager | |
| hadoop.hdfs.server.datanode.TestTriggerBlockReport | |
| hadoop.hdfs.TestAbandonBlock | |
| hadoop.hdfs.server.mover.TestStorageMover | |
| hadoop.hdfs.server.datanode.TestDataNodeMetrics | |
| hadoop.hdfs.TestDatanodeLayoutUpgrade | |
| hadoop.hdfs.TestDataStream | |
| hadoop.hdfs.TestParallelShortCircuitReadUnCached | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion | |
| hadoop.hdfs.TestWriteRead | |
| hadoop.hdfs.server.namenode.TestStripedINodeFile | |
| hadoop.hdfs.tools.TestDebugAdmin | |
| hadoop.hdfs.server.datanode.TestDeleteBlockPool | |
| hadoop.hdfs.TestParallelShortCircuitRead | |
| hadoop.hdfs.TestDFSXORStripedInputStream | |
| hadoop.hdfs.TestRenameWhileOpen | |
| hadoop.hdfs.TestInjectionForSimulatedStorage | |
| hadoop.hdfs.server.datanode.TestCachingStrategy | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | |
| hadoop.hdfs.TestExternalBlockReader | |
| hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | |
| hadoop.hdfs.TestHFlush | |
| hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate | |
| hadoop.fs.contract.hdfs.TestHDFSContractOpen | |
| hadoop.hdfs.web.TestFSMainOperationsWebHdfs | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap | |
| hadoop.fs.TestFcHdfsPermission | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 | |
| hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | |
| hadoop.hdfs.TestFileChecksum | |
| hadoop.hdfs.TestClose | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.TestAppendSnapshotTruncate | |
| hadoop.hdfs.client.impl.TestBlockReaderFactory | |
| hadoop.hdfs.TestDFSRollback | |
| hadoop.hdfs.server.datanode.TestBlockScanner | |
| hadoop.hdfs.TestLeaseRecovery2 | |
| hadoop.hdfs.TestSnapshotCommands | |
| hadoop.hdfs.server.datanode.TestBatchIbr | |
| hadoop.fs.viewfs.TestViewFileSystemWithTruncate | |
| hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.server.datanode.TestReadOnlySharedStorage | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.TestLeaseRecovery | |
| hadoop.hdfs.TestRestartDFS | |
| hadoop.fs.contract.hdfs.TestHDFSContractMkdir | |
| hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy | |
| hadoop.hdfs.server.namenode.TestNamenodeRetryCache | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| hadoop.hdfs.TestDFSFinalize | |
| hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages | |
| hadoop.hdfs.TestMiniDFSCluster | |
| hadoop.hdfs.TestAppendDifferentChecksum | |
| hadoop.hdfs.server.datanode.TestDataNodeMXBean | |
| hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes | |
| hadoop.hdfs.TestDFSStripedOutputStream | |
| hadoop.hdfs.security.token.block.TestBlockToken | |
| hadoop.hdfs.TestDFSStorageStateRecovery | |
| hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement | |
| hadoop.hdfs.server.blockmanagement.TestSequentialBlockId | |
| hadoop.fs.TestUrlStreamHandler | |
| hadoop.hdfs.TestReplaceDatanodeOnFailure | |
| hadoop.hdfs.server.namenode.TestNestedEncryptionZones | |
| hadoop.hdfs.server.namenode.TestHostsFiles | |
| hadoop.hdfs.TestDFSPermission | |
| hadoop.hdfs.server.datanode.TestDiskError | |
| hadoop.hdfs.TestReadStripedFileWithDecoding | |
| hadoop.fs.TestSymlinkHdfsFileSystem | |
| hadoop.hdfs.TestListFilesInDFS | |
| hadoop.hdfs.server.namenode.TestParallelImageWrite | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs | |
| hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 | |
| hadoop.hdfs.server.namenode.TestAuditLogger | |
| hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot | |
| hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | |
| hadoop.hdfs.server.namenode.TestMetaSave | |
| hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks | |
| hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot | |
| hadoop.hdfs.TestDFSRename | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl | |
| hadoop.hdfs.TestParallelUnixDomainRead | |
| hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy | |
| hadoop.fs.viewfs.TestViewFsAtHdfsRoot | |
| hadoop.fs.TestSWebHdfsFileContextMainOperations | |
| hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | |
| hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation | |
| hadoop.hdfs.server.namenode.ha.TestHASafeMode | |
| hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand | |
| hadoop.hdfs.server.namenode.ha.TestStandbyIsHot | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory | |
| hadoop.hdfs.TestBlockMissingException | |
| hadoop.hdfs.tools.TestDFSAdmin | |
| hadoop.hdfs.client.impl.TestClientBlockVerification | |
| hadoop.hdfs.TestDataTransferProtocol | |
| hadoop.hdfs.TestDFSClientRetries | |
| hadoop.hdfs.TestQuota | |
| hadoop.hdfs.server.namenode.TestNameNodeMXBean | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing | |
| hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN | |
| hadoop.fs.contract.hdfs.TestHDFSContractDelete | |
| hadoop.hdfs.TestMissingBlocksAlert | |
| hadoop.hdfs.TestFSOutputSummer | |
| hadoop.hdfs.server.namenode.TestQuotaByStorageType | |
| hadoop.hdfs.server.namenode.TestAddStripedBlocks | |
| hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock | |
| hadoop.hdfs.client.impl.TestBlockReaderLocal | |
| hadoop.hdfs.server.namenode.TestFileContextXAttr | |
| hadoop.hdfs.server.namenode.TestReconstructStripedBlocks | |
| hadoop.hdfs.server.namenode.TestFileTruncate | |
| hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR | |
| hadoop.hdfs.TestFileConcurrentReader | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 | |
| hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| hadoop.hdfs.TestFileAppend2 | |
| hadoop.hdfs.server.namenode.TestNameNodeXAttr | |
| hadoop.fs.contract.hdfs.TestHDFSContractConcat | |
| hadoop.fs.TestEnhancedByteBufferAccess | |
| hadoop.hdfs.server.namenode.TestProcessCorruptBlocks | |
| hadoop.hdfs.web.TestWebHdfsFileSystemContract | |
| hadoop.hdfs.TestMultiThreadedHflush | |
| hadoop.hdfs.TestEncryptionZones | |
| hadoop.hdfs.security.TestDelegationToken | |
| hadoop.hdfs.TestConnCache | |
| hadoop.hdfs.server.namenode.TestFSImage | |
| hadoop.hdfs.server.namenode.TestHDFSConcat | |
| hadoop.hdfs.server.datanode.TestHSync | |
| hadoop.hdfs.TestRead | |
| hadoop.hdfs.server.namenode.TestSnapshotPathINodes | |
| hadoop.hdfs.TestSetTimes | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.TestDisableConnCache | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename | |
| hadoop.hdfs.TestDFSOutputStream | |
| hadoop.hdfs.TestSmallBlock | |
| hadoop.fs.TestFcHdfsSetUMask | |
| hadoop.hdfs.TestFsShellPermission | |
| hadoop.hdfs.TestEncryptionZonesWithHA | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 | |
| hadoop.hdfs.server.namenode.TestBackupNode | |
| hadoop.tracing.TestTracing | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedInputStream | |
| hadoop.fs.contract.hdfs.TestHDFSContractRename | |
| hadoop.hdfs.server.namenode.ha.TestHAStateTransitions | |
| hadoop.hdfs.TestGetFileChecksum | |
| hadoop.hdfs.TestApplyingStoragePolicy | |
| hadoop.hdfs.server.datanode.TestBlockRecovery | |
| hadoop.hdfs.TestFileAppend | |
| hadoop.hdfs.TestDFSClientExcludedNodes | |
| hadoop.hdfs.TestLocalDFS | |
| hadoop.hdfs.server.balancer.TestBalancer | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl | |
| hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup | |
| hadoop.hdfs.TestHdfsAdmin | |
| hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | |
| hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure | |
| hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus | |
| hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics | |
| hadoop.hdfs.server.namenode.TestListCorruptFileBlocks | |
| hadoop.hdfs.TestDFSMkdirs | |
| hadoop.hdfs.TestEncryptionZonesWithKMS | |
| hadoop.fs.TestSymlinkHdfsFileContext | |
| hadoop.hdfs.TestDatanodeReport | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | |
| hadoop.hdfs.TestDFSXORStripedOutputStream | |
| hadoop.hdfs.server.datanode.TestDataNodeFaultInjector | |
| hadoop.hdfs.TestFileStatus | |
| hadoop.hdfs.TestSafeMode | |
| hadoop.hdfs.server.datanode.TestIncrementalBrVariations | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | |
| hadoop.hdfs.TestHDFSTrash | |
| hadoop.hdfs.server.namenode.ha.TestQuotasWithHA | |
| hadoop.hdfs.server.namenode.TestFSImageWithSnapshot | |
| hadoop.hdfs.server.namenode.TestFsck | |
| hadoop.hdfs.TestLease | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer | |
| hadoop.hdfs.TestLargeBlock | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs | |
| hadoop.hdfs.server.namenode.ha.TestHAAppend | |
| hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId | |
| hadoop.hdfs.server.datanode.TestBlockReplacement | |
| hadoop.hdfs.TestClientReportBadBlock | |
| hadoop.hdfs.protocolPB.TestPBHelper | |
| hadoop.hdfs.TestFileCreation | |
| hadoop.tracing.TestTracingShortCircuitLocalRead | |
| hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots | |
| hadoop.hdfs.TestSafeModeWithStripedFile | |
| hadoop.hdfs.TestMaintenanceState | |
| hadoop.hdfs.TestSetrepDecreasing | |
| hadoop.hdfs.tools.TestStoragePolicyCommands | |
| hadoop.hdfs.server.namenode.TestBlockUnderConstruction | |
| hadoop.hdfs.security.TestDelegationTokenForProxyUser | |
| hadoop.hdfs.server.namenode.ha.TestHAMetrics | |
| hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | |
| hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.fs.shell.TestHdfsTextCommand | |
| hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs | |
| hadoop.cli.TestAclCLI | |
| hadoop.hdfs.TestSecureEncryptionZoneWithKMS | |
| hadoop.hdfs.TestBlocksScheduledCounter | |
| hadoop.hdfs.server.namenode.TestFileLimit | |
| hadoop.hdfs.server.namenode.TestFSDirectory | |
| hadoop.hdfs.TestReservedRawPaths | |
| hadoop.tools.TestJMXGet | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 | |
| hadoop.hdfs.TestDFSInputStream | |
| hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | |
| hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 | |
| hadoop.hdfs.TestDFSStripedInputStream | |
| hadoop.fs.TestFcHdfsCreateMkdir | |
| hadoop.hdfs.client.impl.TestBlockReaderRemote | |
| hadoop.hdfs.TestFileAppendRestart | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 | |
| hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer | |
| hadoop.cli.TestHDFSCLI | |
| hadoop.hdfs.server.namenode.TestAuditLogs | |
| hadoop.fs.viewfs.TestViewFileSystemHdfs | |
| hadoop.fs.TestHDFSFileContextMainOperations | |
| hadoop.hdfs.TestFileLengthOnClusterRestart | |
| hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality | |
| hadoop.hdfs.TestListFilesInFileContext | |
| hadoop.hdfs.server.namenode.TestCheckpoint | |
| hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold | |
| hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots | |
| hadoop.hdfs.TestCrcCorruption | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 | |
| hadoop.hdfs.TestErasureCodeBenchmarkThroughput | |
| hadoop.hdfs.server.datanode.TestFsDatasetCache | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 | |
| hadoop.hdfs.TestReadWhileWriting | |
| hadoop.hdfs.TestFileAppend3 | |
| hadoop.hdfs.server.namenode.TestNNThroughputBenchmark | |
| hadoop.hdfs.TestSeekBug | |
| hadoop.hdfs.TestReplication | |
| hadoop.hdfs.TestGetBlocks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 | |
| hadoop.hdfs.TestDFSStartupVersions | |
| hadoop.fs.contract.hdfs.TestHDFSContractCreate | |
| hadoop.hdfs.TestDFSUpgradeFromImage | |
| hadoop.hdfs.TestDecommissionWithStriped | |
| hadoop.hdfs.web.TestHttpsFileSystem | |
| hadoop.hdfs.server.namenode.TestStartup | |
| hadoop.hdfs.TestParallelShortCircuitReadNoChecksum | |
| hadoop.hdfs.server.namenode.TestCacheDirectives | |
| hadoop.hdfs.TestDFSXORStripedOutputStreamWithFailure | |
| hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd | |
| hadoop.hdfs.TestReconstructStripedFile | |
| hadoop.hdfs.TestPipelines | |
| hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot | |
| hadoop.hdfs.server.namenode.TestINodeFile | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeMetrics | |
| hadoop.hdfs.TestSetrepIncreasing | |
| hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA | |
| hadoop.hdfs.shortcircuit.TestShortCircuitCache | |
| hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles | |
| hadoop.fs.contract.hdfs.TestHDFSContractAppend | |
| hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes | |
| hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshot | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart | |
| hadoop.hdfs.server.namenode.TestLargeDirectoryDelete | |
| hadoop.hdfs.TestTrashWithSecureEncryptionZones | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS | |
| hadoop.fs.TestUnbuffer | |
| hadoop.hdfs.TestFSInputChecker | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement | |
| hadoop.fs.permission.TestStickyBit | |
| hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness | |
| hadoop.fs.viewfs.TestViewFsDefaultValue | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | |
| hadoop.fs.viewfs.TestViewFsHdfs | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration | |
| hadoop.hdfs.server.namenode.TestNameEditsConfigs | |
| hadoop.hdfs.TestDataTransferKeepalive | |
| hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes | |
| Timed out junit tests | org.apache.hadoop.hdfs.TestPersistBlocks |
This message was automatically generated.
protoc fails to generate files? Let me survey for a while.
I see there's been motion on this JIRA, did we ever have the ML discussion? Is source compatibility handled by the shaded client? Did we test wire compatibility? Chris Douglas brought up some incompatibilities on HDFS-11010 that I don't think were resolved.
I also asked this on HDFS-11010, but what do we gain from upgrading to a 3.x release of PB?
I see two big gains going to at least a newer version:
- protoc 2.5.0 doesn't work out of the box on a variety of chipsets and platforms (It was an issue raised on the mailing list a while back, so it is impacting users.)
- Ease of contribution: protobuf 2.5.0 is no longer the default protoc that is getting installed on new OS deployments.
The first point, IMO, is fairly important. The OOB compilation experience for non-x86, non-Linux is pretty awful and this would be a huge improvement.
HBase uses pb3 internally because you can avoid having to copy all bytes making a Message ("zero-copy serialization") at least on output (working on same for input) and because it has support for ByteBuffers. pb3 in pb2 mode – the default – is wire compatible in our tests.
Andrew, I've started discussion on ML: http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201703.mbox/%3CCAAD07OLiwnCH2%3D1gw79vz9nUWX-VOMy8KtGt2pj8_Ln32oeDnw%40mail.gmail.com%3E
I summarized opinions by developers discussed on HADOOP-13363 and HDFS-11010. I'm waiting for the opinions by other developers.
Thanks Stack and Allen for sharing the information. The experience and information you shared makes us get motivated to update protobuf's version, I think.
protoc 2.4 is actually becoming hard to find: homebrew doesn't serve it up any more, making setting up new dev systems hard.
At the same time, not only does proto3 appear to have wire changes. I have no faith whatsoever in point releases of protobuf being compatible with previous protoc-generated coded code. To be fair, Avro has a similar policy. They just appear to recognise that the "just rebuild everything" strategy doesn't work for us. That is: the ASF big data stack is distributed development system, whereas the google build is centralised.
If PB can be shaded, then moving up would be good, and, with the shading, let us update at our own rate.
I would like to try proto v2 mode on protoc 3.2 by using syntax = "proto2": maybe this is what Stack mentioned as compatibility mode. Let me know if I'm wrong.
Based on my read, proto2 is the default parsing mode, we need to explicitly opt-in to proto3. This is nice, since we won't have to modify our existing proto files if we upgrade PB.
I think the mode also only affects parsing of the .proto file. proto3 adds new types like maps and unions, which aren't available in proto2 mode. I think enums are also different.
Tsuyoshi Ozawa Yes sir. Default mode is proto2. You have to explicitly ask for proto3. When protoc runs, it emits a WARNING that the .proto does not have an explicit version designation but it is just a WARNING that can be quelled by explicitly stating protoc2 (we did not bother doing this in our project – not yet at least).
I am guessing the wire compatibility will cover this, but want to make sure. We are using protobuf to serialize hdfs fsimage out to disk. The on-disk data format should be compatible.
Kihwal Lee thanks for bringing this up. I believe this scenario will be covered by our existing fsimage upgrade tests, which run on checked-in fsimages.
Hi Tsuyoshi Ozawa, what's the schedule for getting this committed? I'd like alpha3 to be our last alpha, and this is not a change I want to sneak in between alpha3 and beta1. The current planned date for alpha3 is May 15th.
Also to be very explicit, I consider shading PB everywhere to be a prerequisite for upgrading to PB3.
Upgrading to blocker and setting target version for tracking purposes.
Kihwal Lee Good point.
What we thinking for a shading approach?
Shading runs post package and pulls in the relocated classes into the jar that references them.
I see hadoop common, hdfs, hdfs-client, mr-client, yarn-common, yarn-client, yarn-server all making reference to pb (I didn't look at src/test). We thinking we'd dupe instances of pb jars up into each of these artifacts.
Would it make sense having a shaded-3rdparty-libs jar that had relocated protobuf and other libs we want to shade? Downside is all pb references would have to be changed to reference the relocated classes (org.apache.hadoop.com.google.protobuf.*).
Would hadoop3 ship a pb2.5 at all?
Andrew Wang I thought this should be in alpha3, personally. After discussion on ML, we confirmed that proto2 syntax is compatible with protobuf spec v2. We have enough confidence to update protobuf version, I think.
Would it make sense having a shaded-3rdparty-libs jar that had relocated protobuf and other libs we want to shade?
It makes sense to me, but IIUC, this is one kind of optimization to reduce jar size. IMHO, we can make this change later. What do you think?
Maybe I have not enough time to finish this until May 15th, to finish Guava related works until May 15th.
stack could you turn over this issue?
You can ask shading to only bundle required classes which should cut down on some of the duplicates and while I know we're not averse to carrying around a bit of fat in these parts, it does seem like we should try and avoid repeating pb classes seven times (counting above listed artifacts), at least.
How you thinking of shading guava?
stack could you turn over this issue?
I'm a bit stuck at mo for spare cycles....
stack Thanks for your comment. After playing with Guava's problem for a while, I'm thinking that it's enough to shade protobuf/guava in hadoop-common because other components depend on common. What do you think? I think this workaround works well.
So, this would be squashing the above suggested 3rdparty-lib module instead into hadoop-common? It has some pluses. Here's a few downsides:
- Everywhere EXCEPT in hadoop-common we'd have to change com.google.protobuf.* to instead point to the relocated classes bundled up in hadoop-common. This would be easy to do but would probably confuse developers (com.google.protobuf when in hadoop-common but org.apache.hadoop.shaded.com.google.protobuf.* everywhere else).
- In your IDE, working on hadoop-common, you'd be fine but when working on a downstream module – say hdfs – you'd need to depend on a built hadoop-common artifact (unless we included src code for relocated classes).
Pity there is not a pre-build. We make up an artifact of all relocated classes and push it to the repo. Then the main build would depend on it wth all hadoop references amended to refer to the relocated classes.
Make sense, thanks for clarification. It sounds awful.
Let me try to create shaded-3rdparty-libs. I will try to shade both Guava and protobuf there.
I will be back here after HADOOP-14284.
From CHANGES.txt: https://github.com/google/protobuf/blob/master/CHANGES.txt