HDDS-15387. Selectively deserialize KeyInfo/DirectoryInfo fields for snapshot diff#10603
Draft
SaketaChalamchala wants to merge 3 commits into
Draft
HDDS-15387. Selectively deserialize KeyInfo/DirectoryInfo fields for snapshot diff#10603SaketaChalamchala wants to merge 3 commits into
SaketaChalamchala wants to merge 3 commits into
Conversation
…lds for diff generation from Key/DirectoryInfo.
SaketaChalamchala
added
the
snapshot
adoroszlai
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces SnapshotDiffValueParser, a low-level Protobuf parser intended to support more efficient snapshot diff report generation by selectively deserializing only needed KeyInfo / DirectoryInfo fields and computing lightweight “compare signatures”.
Changes:
- Added
SnapshotDiffValueParserto parse required ID/name fields and compute compare signatures directly from persisted Protobuf bytes. - Added unit tests covering required-field parsing, updateID inclusion toggling, and signature sensitivity to metadata vs. volatile timestamps.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffValueParser.java | New selective Protobuf parsing + signature computation for KeyInfo/DirectoryInfo. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestSnapshotDiffValueParser.java | New unit tests validating parsing behavior and some signature properties. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| package org.apache.hadoop.ozone.om.snapshot; | ||
|
|
||
| import static org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.CommitKeyRequest.HSYNC_FIELD_NUMBER; |
Comment on lines
+101
to
+105
| case KeyInfo.METADATA_FIELD_NUMBER: | ||
| MetadataEntry metadataEntry = parseMetadataEntry(input.readBytes().toByteArray()); | ||
| if (metadataEntry != null) { | ||
| updateDigestWithRawBytes(digest, fieldNumber, metadataEntry.getDigest()); | ||
| if (metadataEntry.hasHsync()) { |
Comment on lines
+177
to
+181
| case DirectoryInfo.METADATA_FIELD_NUMBER: | ||
| MetadataEntry metadataEntry = parseMetadataEntry(input.readBytes().toByteArray()); | ||
| if (metadataEntry != null) { | ||
| updateDigestWithRawBytes(digest, fieldNumber, metadataEntry.getDigest()); | ||
| } |
Comment on lines
+52
to
+57
| @Test | ||
| void testKeyInfoParserSignatureAndUpdateId() throws Exception { | ||
| OmKeyInfo keyInfo = createKeyInfo(100L, 200L, 1024L, createChecksum((byte) 1), | ||
| createMetadata("meta", "one"), createTags("tag", "one"), createAcls(), | ||
| Collections.singletonList(createKeyLocationGroup(1L))); | ||
| byte[] rawData = OmKeyInfo.getCodec().toPersistedFormat(keyInfo); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Developed with the help of Cursor AI.
This PR is intended as the foundation for a more efficient snapshot diff (HDDS-9154).
This PR adds a
SnapshotDiffValueParserwith methods to deserialize required fields only from KeyInfo and DirectoryInfo for snapshot diff report generation.The list of fields that are deserialized:
From KeyInfo
fileTable/keyTableentries under consideration :keyName,objectID,parentIDupdateIDis deserialized optionally only for full snapshot diffs.updateID/sequenceNumbergating fields required to build a compare signature(used to compare entries from the snapshots participating in the diff):dataSize,fileChecksum, latest block locationmetadata,acls,tagsFrom DirectoryInfo
fileTable/keyTableentries under consideration :name,objectID,parentIDupdateIDis deserialized optionally only for full snapshot diffs.updateID/sequenceNumbergating fields required to build a compare signature:metadata,acls,What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15387
How was this patch tested?
Unit Test