Fix expireSnapshots fails with IllegalArgumentException on object-store-backed tables#4259
Open
felixhzhu wants to merge 1 commit into
Open
Fix expireSnapshots fails with IllegalArgumentException on object-store-backed tables#4259felixhzhu wants to merge 1 commit into
felixhzhu wants to merge 1 commit into
Conversation
…xception on object-store-backed tables When the underlying FileIO is an object store (S3FileIO/OSSFileIO/GCSFileIO), deleteEmptyDirectory() calls asFileSystemIO() which throws IllegalArgumentException because supportFileSystemOperations() returns false. Fix: 1) Add early-return in TableFileUtil.deleteEmptyDirectory() when !supportFileSystemOperations() 2) Guard the parent directory cleanup loop in RollingFileCleaner with supportFileSystemOperations() check 3) Move buffer clear() into finally block to prevent re-submission on failure fix apache#4237
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix
expireSnapshotstask failing withIllegalArgumentException: nullwhen theunderlying FileIO is an object store (S3FileIO / OSSFileIO / GCSFileIO / ResolvingFileIO).
Root Cause
RollingFileCleaner.doCleanFiles()unconditionally callsTableFileUtil.deleteEmptyDirectory(),which calls
io.asFileSystemIO(). For object-store FileIO implementations,supportFileSystemOperations()returnsfalse, causingPreconditions.checkArgument()to throw
IllegalArgumentException: null."Delete empty parent directory" is only meaningful for HDFS/Hadoop FS semantics — object
stores have no real directory concept.
Fix
TableFileUtil.deleteEmptyDirectory(): Add early-return when!io.supportFileSystemOperations()RollingFileCleaner.doCleanFiles():supportFileSystemOperations()checkparentDirectories.clear()andcollectedFiles.clear()intofinallyblockNo schema / config / API signature changes. HDFS tables are completely unaffected.
How was this patch tested?
fix #4237