HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix by Gargi-jais11 · Pull Request #10586 · apache/ozone

Gargi-jais11 · 2026-06-23T06:03:36Z

What changes were proposed in this pull request?

Ozone S3 Gateway fails s3-tests ListObjects cases that AWS S3 passes:

test_bucket_list_encoding_basic — With EncodingType=url, keys and CommonPrefixes containing spaces are encoded with + (Java URLEncoder form encoding) instead of AWS-style %20.
Example: prefix quux ab/ is returned as quux+ab/ but should be quux%20ab/.
test_bucket_list_delimiter_empty — When Delimiter='' is sent, listing behavior is correct (all keys returned, no CommonPrefixes), but the response incorrectly includes a Delimiter field. AWS omits Delimiter from the XML when the client passes an empty delimiter.
test_bucket_list_prefix_unreadable — ListObjects with Prefix='\x0a' (newline) should echo the prefix in the response and return empty Contents/CommonPrefixes. Ozone may not preserve or echo the control-character prefix correctly.

https://ozone.s3.peterxcli.dev/#latest-run-section

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15600

How was this patch tested?

Before fix:

encoding type:

bash-5.1$ aws s3api list-objects   --bucket buck1   --encoding-type url   --delimiter /   --endpoint-url http://s3g:9878/
{
    "Contents": [
        {
            "Key": "asdf%2Bb",
            "LastModified": "2026-06-19T09:51:39.569Z",
            "ETag": "\"f75b8179e4bbe7e2b4a074dcef62de95\"",
            "Size": 8,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "testuser",
                "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
            }
        }
    ],
    "CommonPrefixes": [
        {
            "Prefix": "foo/"
        },
        {
            "Prefix": "foo%2B1/"
        },
        {
            "Prefix": "quux+ab/".                       <----------------- wrong output
        }
    ],
    "RequestCharged": null,
    "Prefix": ""
}

empty delimiter:

bash-5.1$ aws --debug s3api list-objects \
  --bucket buck1 \
  --delimiter "" \
  --endpoint-url http://s3g:9878 2>&1 \
  | grep -oE '<Prefix>[^<]*</Prefix>|<Delimiter>[^<]*</Delimiter>|<KeyCount>[^<]*</KeyCount>'
<Prefix></Prefix>
<KeyCount>0</KeyCount>
<Delimiter></Delimiter>.           <---------- should not be present

3 echoed prefix always url-encoded:

bash-5.1$ aws s3api list-objects \
  --bucket buck1 \
  --prefix $'\n' \
  --endpoint-url http://s3g:9878/
{
    "RequestCharged": null,
    "Prefix": "%0A"               <------------------ wrong behaviour
}

After fix:

encoding-type:

bash-5.1$ aws s3api list-objects   --bucket buck1   --encoding-type url   --delimiter /   --endpoint-url http://s3g:9878/
{
    "Contents": [
        {
            "Key": "asdf%2Bb",
            "LastModified": "2026-06-19T09:08:58.933Z",
            "ETag": "\"f75b8179e4bbe7e2b4a074dcef62de95\"",
            "Size": 8,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "testuser",
                "ID": "bb2bd7ca4a327f84e6cd3979f8fa3828a50a08893c1b68f9d6715352c8d07b93"
            }
        }
    ],
    "CommonPrefixes": [
        {
            "Prefix": "foo/"
        },
        {
            "Prefix": "foo%2B1/"
        },
        {
            "Prefix": "quux%20ab/"                <------------------------correct output
        }
    ],
    "RequestCharged": null,
    "Prefix": null
}

empty delimeter :

bash-5.1$ aws --debug s3api list-objects \
  --bucket buck1 \
  --delimiter "" \
  --endpoint-url http://s3g:9878 2>&1 \
  | grep -oE '<Prefix>[^<]*</Prefix>|<Delimiter>[^<]*</Delimiter>|<KeyCount>[^<]*</KeyCount>'
  
<KeyCount>0</KeyCount>.                 <-------------------- no empty delimiter present

echoed prefix should not be url encoded:

bash-5.1$ aws s3api list-objects \
  --bucket buck1 \
  --prefix $'\n' \
  --endpoint-url http://s3g:9878/
{
    "RequestCharged": null,
    "Prefix": "\n"
}

…rol-character prefix

adoroszlai

Thanks @Gargi-jais11 for the patch.

chungen0126

Thanks @Gargi-jais11 for working on this. Overall, looks good to me. Just a nit.

Gargi-jais11 · 2026-06-30T07:00:54Z

@ivandika3 please review this patch.

Russole

Thanks @Gargi-jais11 for the patch. I’ve left a few review comments.

Russole · 2026-06-30T15:38:25Z

+    if (prefixSpecified) {
+      response.setPrefix(EncodingTypeObject.createNullable(prefix, null));
+    }


Prefix should also be encoded when encoding-type=url is specified.

AWS docs say EncodingType applies to Delimiter, Prefix, Key, and StartAfter:
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#AmazonS3-ListObjectsV2-response-EncodingType

Local test:

aws s3api list-objects \ --bucket buck1 \ --prefix $'\n' \ --encoding-type url \ --endpoint-url http://localhost:9878/

Actual:

"Prefix": "\n"

Expected:

"Prefix": "%0A"

Looks like Prefix is created with a null encoding type here. Should this use encodingType instead?

Fix ListObjects response for encoding-type, empty delimiter, and cont…

cf2f8b2

…rol-character prefix

adoroszlai added the s3 S3 Gateway label Jun 23, 2026

adoroszlai reviewed Jun 23, 2026

View reviewed changes

Comment thread hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3Utils.java

Gargi-jais11 marked this pull request as ready for review June 23, 2026 08:29

Merge branch 'master' of github.com:apache/ozone into HDDS-15600

45fdef1

Gargi-jais11 requested review from ivandika3 and peterxcli June 29, 2026 07:53

Gargi-jais11 marked this pull request as draft June 29, 2026 08:11

fix checkstyle

1f720b5

Gargi-jais11 marked this pull request as ready for review June 29, 2026 11:57

chungen0126 approved these changes Jun 30, 2026

View reviewed changes

address review comments

2852242

Russole requested changes Jun 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix#10586

HDDS-15600. Fix ListObjects response for encoding-type, empty delimiter, and control-character prefix#10586
Gargi-jais11 wants to merge 4 commits into
apache:masterfrom
Gargi-jais11:HDDS-15600

Gargi-jais11 commented Jun 23, 2026 •

edited

Loading

Uh oh!

adoroszlai left a comment

Uh oh!

Uh oh!

chungen0126 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gargi-jais11 commented Jun 30, 2026

Uh oh!

Russole left a comment

Uh oh!

Russole Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Gargi-jais11 commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chungen0126 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gargi-jais11 commented Jun 30, 2026

Uh oh!

Russole left a comment

Choose a reason for hiding this comment

Uh oh!

Russole Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Gargi-jais11 commented Jun 23, 2026 •

edited

Loading