Details
Description
Standard protobuf 2.5.0 does not work properly on many platforms. (See, for example, https://gist.github.com/BennettSmith/7111094 ). In order for us to avoid crazy work arounds in the build environment and the fact that 2.5.0 is starting to slowly disappear as a standard install-able package for even Linux/x86, we need to either upgrade or self bundle or something else.
Issue Links
- is blocked by
-
HADOOP-13070
classloading isolation improvements for stricter dependencies
-
- Open
-
- is depended upon by
-
HADOOP-9991
Fix up Hadoop POMs, roll up JARs to latest versions
-
- Open
-
- is duplicated by
-
HDFS-11010
Update to Protobuf 3 in trunk
-
- Resolved
-
- is related to
-
HADOOP-14284
Shade Guava everywhere
-
- Resolved
-
- relates to
-
HBASE-15638
Shade protobuf
-
- Resolved
-
-
HDFS-11010
Update to Protobuf 3 in trunk
-
- Resolved
-
-
HADOOP-9845
Update protobuf to 2.5 from 2.4.x
-
- Closed
-
-
REEF-1882
Verify whether REEF works with Protocol Buffers 3.x
-
- Open
-
Show 3 more links (3 relates to)
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
We need wire-compatibility no matter when we upgrade, whether it's from 2.x to 3.0, or 2.x to 3.1.
How can we ensure it? My first thought to ensure this is to run cluster with protobuf 3.x cluster jar and clients with protobuf 2.x jar.
By the way, the current latest version of protobuf seems to be v3.4.0.
https://github.com/google/protobuf/releases
Also, the motivation I've heard for upgrading is to make it easier for devs to build Hadoop since it's hard to find a copy of PB 2.5.0.
That's one issue, but it's not the only one. See the description of this very jira. 
I consider shading a prerequisite for the PB upgrade. If we shade, there aren't any API/ABI compatibility issues. We need wire-compatibility no matter when we upgrade, whether it's from 2.x to 3.0, or 2.x to 3.1.
Also, the motivation I've heard for upgrading is to make it easier for devs to build Hadoop since it's hard to find a copy of PB 2.5.0. If that's the problem, there are other solutions that do not have any downstream user impact. For instance, hosting the PB 2.5.0 binary somewhere, or providing it in a docker image.
I still remember when we bumped the protobuf version shortly before 2.2.0 GA, and the downstream pain it caused due to ABI conflicts
We certainly want to avoid that, but does anyone have cycles to fit this between 3.0.0-beta1 and GA? Bumping the PB version in a minor release would be regrettable, particularly since we're making such a big jump. Taking the hit in 3.0.0 and addressing incompatibility in a minor, even patch release might be easier.
Users expect upgrading to a major version to require some additional testing. Bumping this in a minor version might cause more downstream headaches, even if careful shading would avoid more actual incompatibilities. Most of our use of PB is behind interfaces, so most users shouldn't notice...
Yes, basically. I still remember when we bumped the protobuf version shortly before 2.2.0 GA, and the downstream pain it caused due to ABI conflicts.
We have the shaded-client now, but no one is using it yet, and there are also all the downstreams incorrectly depending on hadoop-hdfs rather than a client artifact. HBase was able to upgrade safely through shading PB. We also need a safe transition for our downstreams.
Where did this stall? On concerns that clients will need to rebuild because they link against the protobuf 2.5 jar, but not the 3.x jar?
I'm dropping this out of beta1 since it seems unlikely to happen in time. Given that we have client shading, we can revisit this later in a minor release.
I will be back here after HADOOP-14284.
Make sense, thanks for clarification. It sounds awful.
Let me try to create shaded-3rdparty-libs. I will try to shade both Guava and protobuf there.
So, this would be squashing the above suggested 3rdparty-lib module instead into hadoop-common? It has some pluses. Here's a few downsides:
- Everywhere EXCEPT in hadoop-common we'd have to change com.google.protobuf.* to instead point to the relocated classes bundled up in hadoop-common. This would be easy to do but would probably confuse developers (com.google.protobuf when in hadoop-common but org.apache.hadoop.shaded.com.google.protobuf.* everywhere else).
- In your IDE, working on hadoop-common, you'd be fine but when working on a downstream module – say hdfs – you'd need to depend on a built hadoop-common artifact (unless we included src code for relocated classes).
Pity there is not a pre-build. We make up an artifact of all relocated classes and push it to the repo. Then the main build would depend on it wth all hadoop references amended to refer to the relocated classes.
stack Thanks for your comment. After playing with Guava's problem for a while, I'm thinking that it's enough to shade protobuf/guava in hadoop-common because other components depend on common. What do you think? I think this workaround works well.
You can ask shading to only bundle required classes which should cut down on some of the duplicates and while I know we're not averse to carrying around a bit of fat in these parts, it does seem like we should try and avoid repeating pb classes seven times (counting above listed artifacts), at least.
How you thinking of shading guava?
stack could you turn over this issue?
I'm a bit stuck at mo for spare cycles....
Maybe I have not enough time to finish this until May 15th, to finish Guava related works until May 15th.
stack could you turn over this issue?
Andrew Wang I thought this should be in alpha3, personally. After discussion on ML, we confirmed that proto2 syntax is compatible with protobuf spec v2. We have enough confidence to update protobuf version, I think.
Would it make sense having a shaded-3rdparty-libs jar that had relocated protobuf and other libs we want to shade?
It makes sense to me, but IIUC, this is one kind of optimization to reduce jar size. IMHO, we can make this change later. What do you think?
Kihwal Lee Good point.
What we thinking for a shading approach?
Shading runs post package and pulls in the relocated classes into the jar that references them.
I see hadoop common, hdfs, hdfs-client, mr-client, yarn-common, yarn-client, yarn-server all making reference to pb (I didn't look at src/test). We thinking we'd dupe instances of pb jars up into each of these artifacts.
Would it make sense having a shaded-3rdparty-libs jar that had relocated protobuf and other libs we want to shade? Downside is all pb references would have to be changed to reference the relocated classes (org.apache.hadoop.com.google.protobuf.*).
Would hadoop3 ship a pb2.5 at all?
Upgrading to blocker and setting target version for tracking purposes.
Kihwal Lee thanks for bringing this up. I believe this scenario will be covered by our existing fsimage upgrade tests, which run on checked-in fsimages.
Hi Tsuyoshi Ozawa, what's the schedule for getting this committed? I'd like alpha3 to be our last alpha, and this is not a change I want to sneak in between alpha3 and beta1. The current planned date for alpha3 is May 15th.
Also to be very explicit, I consider shading PB everywhere to be a prerequisite for upgrading to PB3.
I am guessing the wire compatibility will cover this, but want to make sure. We are using protobuf to serialize hdfs fsimage out to disk. The on-disk data format should be compatible.
Tsuyoshi Ozawa Yes sir. Default mode is proto2. You have to explicitly ask for proto3. When protoc runs, it emits a WARNING that the .proto does not have an explicit version designation but it is just a WARNING that can be quelled by explicitly stating protoc2 (we did not bother doing this in our project – not yet at least).
Based on my read, proto2 is the default parsing mode, we need to explicitly opt-in to proto3. This is nice, since we won't have to modify our existing proto files if we upgrade PB.
I think the mode also only affects parsing of the .proto file. proto3 adds new types like maps and unions, which aren't available in proto2 mode. I think enums are also different.
I would like to try proto v2 mode on protoc 3.2 by using syntax = "proto2": maybe this is what Stack mentioned as compatibility mode. Let me know if I'm wrong.
protoc 2.4 is actually becoming hard to find: homebrew doesn't serve it up any more, making setting up new dev systems hard.
At the same time, not only does proto3 appear to have wire changes. I have no faith whatsoever in point releases of protobuf being compatible with previous protoc-generated coded code. To be fair, Avro has a similar policy. They just appear to recognise that the "just rebuild everything" strategy doesn't work for us. That is: the ASF big data stack is distributed development system, whereas the google build is centralised.
If PB can be shaded, then moving up would be good, and, with the shading, let us update at our own rate.
Andrew, I've started discussion on ML: http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201703.mbox/%3CCAAD07OLiwnCH2%3D1gw79vz9nUWX-VOMy8KtGt2pj8_Ln32oeDnw%40mail.gmail.com%3E
I summarized opinions by developers discussed on HADOOP-13363 and HDFS-11010. I'm waiting for the opinions by other developers.
Thanks Stack and Allen for sharing the information. The experience and information you shared makes us get motivated to update protobuf's version, I think.
HBase uses pb3 internally because you can avoid having to copy all bytes making a Message ("zero-copy serialization") at least on output (working on same for input) and because it has support for ByteBuffers. pb3 in pb2 mode – the default – is wire compatible in our tests.
I see two big gains going to at least a newer version:
- protoc 2.5.0 doesn't work out of the box on a variety of chipsets and platforms (It was an issue raised on the mailing list a while back, so it is impacting users.)
- Ease of contribution: protobuf 2.5.0 is no longer the default protoc that is getting installed on new OS deployments.
The first point, IMO, is fairly important. The OOB compilation experience for non-x86, non-Linux is pretty awful and this would be a huge improvement.
I see there's been motion on this JIRA, did we ever have the ML discussion? Is source compatibility handled by the shaded client? Did we test wire compatibility? Chris Douglas brought up some incompatibilities on HDFS-11010 that I don't think were resolved.
I also asked this on HDFS-11010, but what do we gain from upgrading to a 3.x release of PB?
protoc fails to generate files? Let me survey for a while.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 14m 22s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 15s | Maven dependency ordering for branch |
| -1 | mvninstall | 0m 27s | root in trunk failed. |
| -1 | compile | 0m 24s | root in trunk failed. |
| +1 | checkstyle | 1m 49s | trunk passed |
| -1 | mvnsite | 0m 10s | hadoop-hdfs in trunk failed. |
| -1 | mvneclipse | 0m 10s | hadoop-hdfs in trunk failed. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 12s | hadoop-hdfs in trunk failed. |
| -1 | javadoc | 0m 13s | hadoop-hdfs in trunk failed. |
| 0 | mvndep | 0m 19s | Maven dependency ordering for patch |
| +1 | mvninstall | 1m 5s | the patch passed |
| +1 | compile | 17m 35s | the patch passed |
| -1 | javac | 17m 35s | root generated 1232 new + 0 unchanged - 0 fixed = 1232 total (was 0) |
| +1 | checkstyle | 2m 5s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| +1 | mvnsite | 1m 37s | the patch passed |
| +1 | mvneclipse | 0m 58s | the patch passed |
| -1 | shellcheck | 0m 16s | The patch generated 632 new + 104 unchanged - 0 fixed = 736 total (was 104) |
| -0 | shelldocs | 0m 21s | The patch generated 282 new + 104 unchanged - 0 fixed = 386 total (was 104) |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| +1 | findbugs | 2m 13s | the patch passed |
| -1 | javadoc | 1m 12s | hadoop-hdfs-project_hadoop-hdfs generated 9 new + 0 unchanged - 0 fixed = 9 total (was 0) |
| +1 | unit | 0m 29s | hadoop-project in the patch passed. |
| -1 | unit | 54m 5s | hadoop-hdfs in the patch failed. |
| -1 | asflicense | 0m 51s | The patch generated 1 ASF License warnings. |
| 128m 8s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer |
| hadoop.hdfs.TestHDFSFileSystemContract | |
| hadoop.hdfs.TestErasureCodingPolicyWithSnapshot | |
| hadoop.hdfs.TestEncryptedTransfer | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport | |
| hadoop.hdfs.server.namenode.TestQuotaWithStripedBlocks | |
| hadoop.hdfs.TestBlockStoragePolicy | |
| hadoop.hdfs.TestRollingUpgrade | |
| hadoop.hdfs.web.TestWebHDFSForHA | |
| hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | |
| hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | |
| hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStream | |
| hadoop.hdfs.server.blockmanagement.TestPendingReconstruction | |
| hadoop.hdfs.TestClientProtocolForPipelineRecovery | |
| hadoop.hdfs.server.namenode.TestFSEditLogLoader | |
| hadoop.fs.contract.hdfs.TestHDFSContractSeek | |
| hadoop.hdfs.TestPread | |
| hadoop.fs.viewfs.TestViewFsFileStatusHdfs | |
| hadoop.hdfs.TestWriteConfigurationToDFS | |
| hadoop.cli.TestAclCLIWithPosixAclInheritance | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | |
| hadoop.hdfs.TestParallelShortCircuitLegacyRead | |
| hadoop.hdfs.TestFileCreationClient | |
| hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes | |
| hadoop.hdfs.TestLeaseRecoveryStriped | |
| hadoop.hdfs.server.diskbalancer.TestDiskBalancer | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation | |
| hadoop.hdfs.server.namenode.TestUpgradeDomainBlockPlacementPolicy | |
| hadoop.hdfs.crypto.TestHdfsCryptoStreams | |
| hadoop.hdfs.server.datanode.TestTransferRbw | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks | |
| hadoop.hdfs.TestWriteBlockGetsBlockLengthHint | |
| hadoop.hdfs.server.namenode.TestAddBlock | |
| hadoop.hdfs.TestDatanodeDeath | |
| hadoop.fs.TestSymlinkHdfsDisable | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | |
| hadoop.hdfs.TestDFSInotifyEventInputStream | |
| hadoop.hdfs.TestDecommission | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary | |
| hadoop.hdfs.server.namenode.TestCommitBlockWithInvalidGenStamp | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotMetrics | |
| hadoop.hdfs.shortcircuit.TestShortCircuitLocalRead | |
| hadoop.hdfs.TestWriteReadStripedFile | |
| hadoop.hdfs.server.mover.TestMover | |
| hadoop.hdfs.TestDFSRemove | |
| hadoop.hdfs.TestAclsEndToEnd | |
| hadoop.fs.loadGenerator.TestLoadGenerator | |
| hadoop.hdfs.TestFileCorruption | |
| hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication | |
| hadoop.hdfs.server.datanode.TestDataNodeTcpNoDelay | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter | |
| hadoop.hdfs.TestFileAppend4 | |
| hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC | |
| hadoop.hdfs.server.namenode.ha.TestDNFencing | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength | |
| hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport | |
| hadoop.security.TestPermission | |
| hadoop.hdfs.TestDFSShell | |
| hadoop.hdfs.server.blockmanagement.TestNodeCount | |
| hadoop.hdfs.server.namenode.TestDeleteRace | |
| hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication | |
| hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade | |
| hadoop.hdfs.web.TestWebHDFSXAttr | |
| hadoop.hdfs.TestDFSClientFailover | |
| hadoop.hdfs.TestModTime | |
| hadoop.hdfs.TestDistributedFileSystem | |
| hadoop.hdfs.web.TestWebHDFS | |
| hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot | |
| hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles | |
| hadoop.fs.TestWebHdfsFileContextMainOperations | |
| hadoop.hdfs.server.namenode.snapshot.TestFileContextSnapshot | |
| hadoop.hdfs.TestFileCreationDelete | |
| hadoop.hdfs.TestDFSUpgrade | |
| hadoop.hdfs.TestParallelRead | |
| hadoop.hdfs.TestTrashWithEncryptionZones | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 | |
| hadoop.security.TestPermissionSymlinks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | |
| hadoop.hdfs.server.blockmanagement.TestBlockManager | |
| hadoop.hdfs.server.datanode.TestTriggerBlockReport | |
| hadoop.hdfs.TestAbandonBlock | |
| hadoop.hdfs.server.mover.TestStorageMover | |
| hadoop.hdfs.server.datanode.TestDataNodeMetrics | |
| hadoop.hdfs.TestDatanodeLayoutUpgrade | |
| hadoop.hdfs.TestDataStream | |
| hadoop.hdfs.TestParallelShortCircuitReadUnCached | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion | |
| hadoop.hdfs.TestWriteRead | |
| hadoop.hdfs.server.namenode.TestStripedINodeFile | |
| hadoop.hdfs.tools.TestDebugAdmin | |
| hadoop.hdfs.server.datanode.TestDeleteBlockPool | |
| hadoop.hdfs.TestParallelShortCircuitRead | |
| hadoop.hdfs.TestDFSXORStripedInputStream | |
| hadoop.hdfs.TestRenameWhileOpen | |
| hadoop.hdfs.TestInjectionForSimulatedStorage | |
| hadoop.hdfs.server.datanode.TestCachingStrategy | |
| hadoop.hdfs.server.namenode.TestDecommissioningStatus | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | |
| hadoop.hdfs.TestExternalBlockReader | |
| hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA | |
| hadoop.hdfs.TestHFlush | |
| hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate | |
| hadoop.fs.contract.hdfs.TestHDFSContractOpen | |
| hadoop.hdfs.web.TestFSMainOperationsWebHdfs | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap | |
| hadoop.fs.TestFcHdfsPermission | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 | |
| hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | |
| hadoop.hdfs.TestFileChecksum | |
| hadoop.hdfs.TestClose | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.TestAppendSnapshotTruncate | |
| hadoop.hdfs.client.impl.TestBlockReaderFactory | |
| hadoop.hdfs.TestDFSRollback | |
| hadoop.hdfs.server.datanode.TestBlockScanner | |
| hadoop.hdfs.TestLeaseRecovery2 | |
| hadoop.hdfs.TestSnapshotCommands | |
| hadoop.hdfs.server.datanode.TestBatchIbr | |
| hadoop.fs.viewfs.TestViewFileSystemWithTruncate | |
| hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.server.datanode.TestReadOnlySharedStorage | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure | |
| hadoop.hdfs.TestLeaseRecovery | |
| hadoop.hdfs.TestRestartDFS | |
| hadoop.fs.contract.hdfs.TestHDFSContractMkdir | |
| hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy | |
| hadoop.hdfs.server.namenode.TestNamenodeRetryCache | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| hadoop.hdfs.TestDFSFinalize | |
| hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages | |
| hadoop.hdfs.TestMiniDFSCluster | |
| hadoop.hdfs.TestAppendDifferentChecksum | |
| hadoop.hdfs.server.datanode.TestDataNodeMXBean | |
| hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes | |
| hadoop.hdfs.TestDFSStripedOutputStream | |
| hadoop.hdfs.security.token.block.TestBlockToken | |
| hadoop.hdfs.TestDFSStorageStateRecovery | |
| hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement | |
| hadoop.hdfs.server.blockmanagement.TestSequentialBlockId | |
| hadoop.fs.TestUrlStreamHandler | |
| hadoop.hdfs.TestReplaceDatanodeOnFailure | |
| hadoop.hdfs.server.namenode.TestNestedEncryptionZones | |
| hadoop.hdfs.server.namenode.TestHostsFiles | |
| hadoop.hdfs.TestDFSPermission | |
| hadoop.hdfs.server.datanode.TestDiskError | |
| hadoop.hdfs.TestReadStripedFileWithDecoding | |
| hadoop.fs.TestSymlinkHdfsFileSystem | |
| hadoop.hdfs.TestListFilesInDFS | |
| hadoop.hdfs.server.namenode.TestParallelImageWrite | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs | |
| hadoop.hdfs.server.namenode.ha.TestFailoverWithBlockTokensEnabled | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120 | |
| hadoop.hdfs.server.namenode.TestAuditLogger | |
| hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot | |
| hadoop.hdfs.server.namenode.TestNamenodeCapacityReport | |
| hadoop.hdfs.server.namenode.TestMetaSave | |
| hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks | |
| hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot | |
| hadoop.hdfs.TestDFSRename | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl | |
| hadoop.hdfs.TestParallelUnixDomainRead | |
| hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy | |
| hadoop.fs.viewfs.TestViewFsAtHdfsRoot | |
| hadoop.fs.TestSWebHdfsFileContextMainOperations | |
| hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | |
| hadoop.hdfs.server.datanode.TestFsDatasetCacheRevocation | |
| hadoop.hdfs.server.namenode.ha.TestHASafeMode | |
| hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand | |
| hadoop.hdfs.server.namenode.ha.TestStandbyIsHot | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory | |
| hadoop.hdfs.TestBlockMissingException | |
| hadoop.hdfs.tools.TestDFSAdmin | |
| hadoop.hdfs.client.impl.TestClientBlockVerification | |
| hadoop.hdfs.TestDataTransferProtocol | |
| hadoop.hdfs.TestDFSClientRetries | |
| hadoop.hdfs.TestQuota | |
| hadoop.hdfs.server.namenode.TestNameNodeMXBean | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing | |
| hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN | |
| hadoop.fs.contract.hdfs.TestHDFSContractDelete | |
| hadoop.hdfs.TestMissingBlocksAlert | |
| hadoop.hdfs.TestFSOutputSummer | |
| hadoop.hdfs.server.namenode.TestQuotaByStorageType | |
| hadoop.hdfs.server.namenode.TestAddStripedBlocks | |
| hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock | |
| hadoop.hdfs.client.impl.TestBlockReaderLocal | |
| hadoop.hdfs.server.namenode.TestFileContextXAttr | |
| hadoop.hdfs.server.namenode.TestReconstructStripedBlocks | |
| hadoop.hdfs.server.namenode.TestFileTruncate | |
| hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR | |
| hadoop.hdfs.TestFileConcurrentReader | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 | |
| hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| hadoop.hdfs.TestFileAppend2 | |
| hadoop.hdfs.server.namenode.TestNameNodeXAttr | |
| hadoop.fs.contract.hdfs.TestHDFSContractConcat | |
| hadoop.fs.TestEnhancedByteBufferAccess | |
| hadoop.hdfs.server.namenode.TestProcessCorruptBlocks | |
| hadoop.hdfs.web.TestWebHdfsFileSystemContract | |
| hadoop.hdfs.TestMultiThreadedHflush | |
| hadoop.hdfs.TestEncryptionZones | |
| hadoop.hdfs.security.TestDelegationToken | |
| hadoop.hdfs.TestConnCache | |
| hadoop.hdfs.server.namenode.TestFSImage | |
| hadoop.hdfs.server.namenode.TestHDFSConcat | |
| hadoop.hdfs.server.datanode.TestHSync | |
| hadoop.hdfs.TestRead | |
| hadoop.hdfs.server.namenode.TestSnapshotPathINodes | |
| hadoop.hdfs.TestSetTimes | |
| hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
| hadoop.hdfs.TestDisableConnCache | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotRename | |
| hadoop.hdfs.TestDFSOutputStream | |
| hadoop.hdfs.TestSmallBlock | |
| hadoop.fs.TestFcHdfsSetUMask | |
| hadoop.hdfs.TestFsShellPermission | |
| hadoop.hdfs.TestEncryptionZonesWithHA | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 | |
| hadoop.hdfs.server.namenode.TestBackupNode | |
| hadoop.tracing.TestTracing | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedInputStream | |
| hadoop.fs.contract.hdfs.TestHDFSContractRename | |
| hadoop.hdfs.server.namenode.ha.TestHAStateTransitions | |
| hadoop.hdfs.TestGetFileChecksum | |
| hadoop.hdfs.TestApplyingStoragePolicy | |
| hadoop.hdfs.server.datanode.TestBlockRecovery | |
| hadoop.hdfs.TestFileAppend | |
| hadoop.hdfs.TestDFSClientExcludedNodes | |
| hadoop.hdfs.TestLocalDFS | |
| hadoop.hdfs.server.balancer.TestBalancer | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForAcl | |
| hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup | |
| hadoop.hdfs.TestHdfsAdmin | |
| hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | |
| hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshotNameWithInvalidCharacters | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure | |
| hadoop.fs.contract.hdfs.TestHDFSContractGetFileStatus | |
| hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics | |
| hadoop.hdfs.server.namenode.TestListCorruptFileBlocks | |
| hadoop.hdfs.TestDFSMkdirs | |
| hadoop.hdfs.TestEncryptionZonesWithKMS | |
| hadoop.fs.TestSymlinkHdfsFileContext | |
| hadoop.hdfs.TestDatanodeReport | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | |
| hadoop.hdfs.TestDFSXORStripedOutputStream | |
| hadoop.hdfs.server.datanode.TestDataNodeFaultInjector | |
| hadoop.hdfs.TestFileStatus | |
| hadoop.hdfs.TestSafeMode | |
| hadoop.hdfs.server.datanode.TestIncrementalBrVariations | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | |
| hadoop.hdfs.TestHDFSTrash | |
| hadoop.hdfs.server.namenode.ha.TestQuotasWithHA | |
| hadoop.hdfs.server.namenode.TestFSImageWithSnapshot | |
| hadoop.hdfs.server.namenode.TestFsck | |
| hadoop.hdfs.TestLease | |
| hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer | |
| hadoop.hdfs.TestLargeBlock | |
| hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs | |
| hadoop.hdfs.server.namenode.ha.TestHAAppend | |
| hadoop.hdfs.server.blockmanagement.TestSequentialBlockGroupId | |
| hadoop.hdfs.server.datanode.TestBlockReplacement | |
| hadoop.hdfs.TestClientReportBadBlock | |
| hadoop.hdfs.protocolPB.TestPBHelper | |
| hadoop.hdfs.TestFileCreation | |
| hadoop.tracing.TestTracingShortCircuitLocalRead | |
| hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots | |
| hadoop.hdfs.TestSafeModeWithStripedFile | |
| hadoop.hdfs.TestMaintenanceState | |
| hadoop.hdfs.TestSetrepDecreasing | |
| hadoop.hdfs.tools.TestStoragePolicyCommands | |
| hadoop.hdfs.server.namenode.TestBlockUnderConstruction | |
| hadoop.hdfs.security.TestDelegationTokenForProxyUser | |
| hadoop.hdfs.server.namenode.ha.TestHAMetrics | |
| hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | |
| hadoop.hdfs.web.TestWebHdfsWithRestCsrfPreventionFilter | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.fs.shell.TestHdfsTextCommand | |
| hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs | |
| hadoop.cli.TestAclCLI | |
| hadoop.hdfs.TestSecureEncryptionZoneWithKMS | |
| hadoop.hdfs.TestBlocksScheduledCounter | |
| hadoop.hdfs.server.namenode.TestFileLimit | |
| hadoop.hdfs.server.namenode.TestFSDirectory | |
| hadoop.hdfs.TestReservedRawPaths | |
| hadoop.tools.TestJMXGet | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 | |
| hadoop.hdfs.TestDFSInputStream | |
| hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | |
| hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 | |
| hadoop.hdfs.TestDFSStripedInputStream | |
| hadoop.fs.TestFcHdfsCreateMkdir | |
| hadoop.hdfs.client.impl.TestBlockReaderRemote | |
| hadoop.hdfs.TestFileAppendRestart | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 | |
| hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer | |
| hadoop.cli.TestHDFSCLI | |
| hadoop.hdfs.server.namenode.TestAuditLogs | |
| hadoop.fs.viewfs.TestViewFileSystemHdfs | |
| hadoop.fs.TestHDFSFileContextMainOperations | |
| hadoop.hdfs.TestFileLengthOnClusterRestart | |
| hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality | |
| hadoop.hdfs.TestListFilesInFileContext | |
| hadoop.hdfs.server.namenode.TestCheckpoint | |
| hadoop.hdfs.server.datanode.TestDnRespectsBlockReportSplitThreshold | |
| hadoop.hdfs.server.namenode.snapshot.TestUpdatePipelineWithSnapshots | |
| hadoop.hdfs.TestCrcCorruption | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210 | |
| hadoop.hdfs.TestErasureCodeBenchmarkThroughput | |
| hadoop.hdfs.server.datanode.TestFsDatasetCache | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestInterDatanodeProtocol | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040 | |
| hadoop.hdfs.TestReadWhileWriting | |
| hadoop.hdfs.TestFileAppend3 | |
| hadoop.hdfs.server.namenode.TestNNThroughputBenchmark | |
| hadoop.hdfs.TestSeekBug | |
| hadoop.hdfs.TestReplication | |
| hadoop.hdfs.TestGetBlocks | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 | |
| hadoop.hdfs.TestDFSStartupVersions | |
| hadoop.fs.contract.hdfs.TestHDFSContractCreate | |
| hadoop.hdfs.TestDFSUpgradeFromImage | |
| hadoop.hdfs.TestDecommissionWithStriped | |
| hadoop.hdfs.web.TestHttpsFileSystem | |
| hadoop.hdfs.server.namenode.TestStartup | |
| hadoop.hdfs.TestParallelShortCircuitReadNoChecksum | |
| hadoop.hdfs.server.namenode.TestCacheDirectives | |
| hadoop.hdfs.TestDFSXORStripedOutputStreamWithFailure | |
| hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd | |
| hadoop.hdfs.TestReconstructStripedFile | |
| hadoop.hdfs.TestPipelines | |
| hadoop.hdfs.server.namenode.snapshot.TestGetContentSummaryWithSnapshot | |
| hadoop.hdfs.server.namenode.TestINodeFile | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeMetrics | |
| hadoop.hdfs.TestSetrepIncreasing | |
| hadoop.hdfs.server.namenode.ha.TestHarFileSystemWithHA | |
| hadoop.hdfs.shortcircuit.TestShortCircuitCache | |
| hadoop.hdfs.server.namenode.metrics.TestNNMetricFilesInGetListingOps | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles | |
| hadoop.fs.contract.hdfs.TestHDFSContractAppend | |
| hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes | |
| hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot | |
| hadoop.hdfs.server.namenode.snapshot.TestSnapshot | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestDatanodeRestart | |
| hadoop.hdfs.server.namenode.TestLargeDirectoryDelete | |
| hadoop.hdfs.TestTrashWithSecureEncryptionZones | |
| hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS | |
| hadoop.fs.TestUnbuffer | |
| hadoop.hdfs.TestFSInputChecker | |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement | |
| hadoop.fs.permission.TestStickyBit | |
| hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness | |
| hadoop.fs.viewfs.TestViewFsDefaultValue | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | |
| hadoop.fs.viewfs.TestViewFsHdfs | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration | |
| hadoop.hdfs.server.namenode.TestNameEditsConfigs | |
| hadoop.hdfs.TestDataTransferKeepalive | |
| hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes | |
| Timed out junit tests | org.apache.hadoop.hdfs.TestPersistBlocks |
This message was automatically generated.
A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/11939/console in case of problems.
Trying to fix a problem by lack of execution permission.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 16m 41s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 17s | Maven dependency ordering for branch |
| -1 | mvninstall | 0m 32s | root in trunk failed. |
| -1 | compile | 0m 28s | root in trunk failed. |
| +1 | checkstyle | 1m 56s | trunk passed |
| -1 | mvnsite | 0m 11s | hadoop-hdfs in trunk failed. |
| -1 | mvneclipse | 0m 9s | hadoop-hdfs in trunk failed. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 11s | hadoop-hdfs in trunk failed. |
| -1 | javadoc | 0m 11s | hadoop-hdfs in trunk failed. |
| 0 | mvndep | 0m 22s | Maven dependency ordering for patch |
| -1 | mvninstall | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | compile | 0m 28s | root in the patch failed. |
| -1 | javac | 0m 28s | root in the patch failed. |
| +1 | checkstyle | 1m 57s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| -1 | mvnsite | 0m 15s | hadoop-hdfs in the patch failed. |
| -1 | mvneclipse | 0m 15s | hadoop-hdfs in the patch failed. |
| +1 | shellcheck | 0m 10s | There were no new shellcheck issues. |
| +1 | shelldocs | 0m 9s | There were no new shelldocs issues. |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 2s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 16s | hadoop-hdfs in the patch failed. |
| -1 | javadoc | 0m 17s | hadoop-hdfs in the patch failed. |
| +1 | unit | 0m 16s | hadoop-project in the patch passed. |
| -1 | unit | 0m 18s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
| 51m 29s |
This message was automatically generated.
A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/11938/console in case of problems.
Thanks Allen! Updated.
@stack unfortunately, not yet. I've also checked discussion on HDFS-11010. Based on the discussion on the jira and here, I would like to start next argument.
Attaching a patch to update Dockerfile to download protobuf 3.2.0.
(It might also be time to move to Xenial from Trusty. I think I've got a working Dockerfile around here that does that too.)
Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-alpha3-SNAPSHOT:protoc (compile-protoc)
i.e., dev-support/docker/Dockerfile needs to be updated to use the same version of protobuf that's in the maven pom.xml file.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 16s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 14s | Maven dependency ordering for branch |
| +1 | mvninstall | 12m 46s | trunk passed |
| +1 | compile | 16m 4s | trunk passed |
| +1 | checkstyle | 1m 54s | trunk passed |
| +1 | mvnsite | 1m 16s | trunk passed |
| +1 | mvneclipse | 0m 37s | trunk passed |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| +1 | findbugs | 1m 53s | trunk passed |
| +1 | javadoc | 1m 5s | trunk passed |
| 0 | mvndep | 0m 19s | Maven dependency ordering for patch |
| -1 | mvninstall | 0m 9s | hadoop-hdfs in the patch failed. |
| -1 | compile | 0m 25s | root in the patch failed. |
| -1 | javac | 0m 25s | root in the patch failed. |
| +1 | checkstyle | 1m 50s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| -1 | mvnsite | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | mvneclipse | 0m 11s | hadoop-hdfs in the patch failed. |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | javadoc | 0m 12s | hadoop-hdfs in the patch failed. |
| +1 | unit | 0m 13s | hadoop-project in the patch passed. |
| -1 | unit | 0m 12s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 65m 23s |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 22s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| 0 | mvndep | 0m 16s | Maven dependency ordering for branch |
| +1 | mvninstall | 14m 29s | trunk passed |
| +1 | compile | 17m 30s | trunk passed |
| +1 | checkstyle | 1m 57s | trunk passed |
| +1 | mvnsite | 1m 14s | trunk passed |
| +1 | mvneclipse | 0m 36s | trunk passed |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| +1 | findbugs | 1m 52s | trunk passed |
| +1 | javadoc | 1m 6s | trunk passed |
| 0 | mvndep | 0m 19s | Maven dependency ordering for patch |
| -1 | mvninstall | 0m 9s | hadoop-hdfs in the patch failed. |
| -1 | compile | 0m 24s | root in the patch failed. |
| -1 | javac | 0m 24s | root in the patch failed. |
| +1 | checkstyle | 1m 48s | root: The patch generated 0 new + 27 unchanged - 1 fixed = 27 total (was 28) |
| -1 | mvnsite | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 | mvneclipse | 0m 11s | hadoop-hdfs in the patch failed. |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| 0 | findbugs | 0m 0s | Skipped patched modules with no Java source: hadoop-project |
| -1 | findbugs | 0m 13s | hadoop-hdfs in the patch failed. |
| -1 | javadoc | 0m 13s | hadoop-hdfs in the patch failed. |
| +1 | unit | 0m 12s | hadoop-project in the patch passed. |
| -1 | unit | 0m 13s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 68m 34s |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 7m 7s | trunk passed |
| +1 | compile | 0m 8s | trunk passed |
| +1 | mvnsite | 0m 11s | trunk passed |
| +1 | mvneclipse | 0m 9s | trunk passed |
| +1 | javadoc | 0m 9s | trunk passed |
| +1 | mvninstall | 0m 7s | the patch passed |
| +1 | compile | 0m 6s | the patch passed |
| +1 | javac | 0m 6s | the patch passed |
| +1 | mvnsite | 0m 9s | the patch passed |
| +1 | mvneclipse | 0m 6s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 7s | the patch passed |
| +1 | unit | 0m 6s | hadoop-project in the patch passed. |
| +1 | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| 9m 35s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HADOOP-13363 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12816966/HADOOP-13363.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
| uname | Linux 714e389ffa51 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c62ae71 |
| Default Java | 1.8.0_101 |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/10826/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10826/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Did a ML discussion happen?
pb3.1.0 is out. It runs in a 2.5.0 compatibility mode by default. Has some facility for saving on data copying that might be of interest in the NN. If upgrading, you need to run the newer protoc. Newer lib can't read the protos made by older protoc (IIRC). Newer protoc, in my experience, has no problem digesting pb 2.5.0 .proto files. The generated files are a little different, not consumable by the old protobuf lib.
Would this be a problem? Old clients can talk to the new servers because of wire compatible. Is anyone consuming hadoop protos directly other than hadoop? Are hadoop proto files considered InterfaceAudience.Private or InterfaceAudience.Public? If the former, I could work on a patch for 3.0.0 (It'd be big but boring). Does Hadoop have Protobuf in its API anywhere (I can take a look but being lazy asking here first).
Thank you for the clarification, Steve. I got the point. This kind of "updating dependency work" is related to the classpath isolation work(HADOOP-13070), so I'd like to start discussion on ML.
wire-level backward compatibility is a core feature of protobuf, I'm not worried there. What I am worried about is compile-compat, as that is what broke hadoop, hbase, etc, everything using protobuf 2.4. Google's internal build process is a clean, unified build of everything, so they don't have to worry about source level compatibliity
This merits a discussion on common-dev for visibility. There was a discussion before but no conclusion iirc.
Sure.
What is going to happen when code generated by protoc 2.5 tries to run against 2.6 library? Has anyone tested this? Because thats the metric of how traumatic this is going to be
We must test it before merging it, of course. IIUC, protobuf is wire compatible between 2.5 and 2.6. I found a test suite which seems useful here: https://chromium.googlesource.com/external/github.com/google/protobuf/+/master/java/compatibility_tests/README.md
We also need to check code-level compatibility.
What is going to happen when code generated by protoc 2.5 tries to run against 2.6 library? Has anyone tested this? Because thats the metric of how traumatic this is going to be
This merits a discussion on common-dev for visibility. There was a discussion before but no conclusion iirc.
I think it is fine as long as it does not break wire compatibility and we can shade the dependency, to be kind to downstream projects.
We also need to update the Dockerfile to pull in protobuf 2.6.x, which may end up being tricky.
Thanks Allen for reporting this and thanks Tsuyoshi for the initial patch. We need to update BUILDING.txt as well.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 6m 38s | trunk passed |
| +1 | compile | 0m 9s | trunk passed |
| +1 | mvnsite | 0m 11s | trunk passed |
| +1 | mvneclipse | 0m 9s | trunk passed |
| +1 | javadoc | 0m 9s | trunk passed |
| +1 | mvninstall | 0m 7s | the patch passed |
| +1 | compile | 0m 6s | the patch passed |
| +1 | javac | 0m 6s | the patch passed |
| +1 | mvnsite | 0m 8s | the patch passed |
| +1 | mvneclipse | 0m 7s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 | javadoc | 0m 6s | the patch passed |
| +1 | unit | 0m 6s | hadoop-project in the patch passed. |
| +1 | asflicense | 0m 15s | The patch does not generate ASF License warnings. |
| 9m 5s |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12816966/HADOOP-13363.001.patch |
| JIRA Issue | HADOOP-13363 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml |
| uname | Linux 0f4f7012a335 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 932aed6 |
| Default Java | 1.8.0_91 |
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/9949/testReport/ |
| modules | C: hadoop-project U: hadoop-project |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9949/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
From CHANGES.txt: https://github.com/google/protobuf/blob/master/CHANGES.txt
2.6.1
Java
- Updated DynamicMessage.setField() to handle repeated enum values
correctly.- Fixed a bug that caused NullPointerException to be thrown when
converting manually constructed FileDescriptorProto to
FileDescriptor.
2.6.0
General
General
- Added oneofs(unions) feature. Fields in the same oneof will share
memory and at most one field can be set at the same time. Use the
oneof keyword to define a oneof like:message SampleMessage { oneof test_oneof { string name = 4; YourMessage sub_message = 9; } }- Files, services, enums, messages, methods and enum values can be marked
as deprecated now.- Added Support for list values, including lists of messages, when
parsing text-formatted protos in C++ and Java.
For example: foo: [1, 2, 3]
...
Java- writeTo() method in ByteString can now write a substring to an
output stream. Added endWith() method for ByteString.- ByteString and ByteBuffer are now supported in CodedInputStream
and CodedOutputStream.- java_generate_equals_and_hash can now be used with the LITE_RUNTIME.
I'd presume upgrade but run in pb2 mode, the default. That has been wire compat in my limited experience. We could do a bit of testing I suppose.