csaf_downloader - memory leak#743
Open
adulau wants to merge 2 commits into
Open
Conversation
…nloader-for-memory-leaks Limit downloader memory retention
Member
Contributor
|
@adulau Thank you for the contribution. We are currently addressing the issues around the memory leaks in other branches and PRs. We need to consolidate them first before we can merge this PR. |
Author
|
You're welcome. It could be related but we only seen the issue on Do you think this could be merged in the next release? It's becoming a major issue for the large CSAF feeds. Thank you very much. |
Member
Yes, we plan memory consumption improvements for the next release and actively work on it.
If you have some specific testcases for us (best in a new issues if it does not fit #718) we can consider them during the development and in testing before the release. The one from the BSI for instance? |
Author
|
I think a good test case is the one from RedHat. It's pretty large. |
Member
Yes, certainly is. BTW, you know that many CSAF files from Redhat are invalid (last time I've looked), https://redhat.atlassian.net/browse/SECDATA-1167 |
Author
|
Even from the download perspective, we have many failing for RedHat: It's hard to do it right ;-) |
Member
Quite a few larger CSAF providers will throttle someone who requests many things either too fast or in parallel. Redhat maybe one of them. ;-) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improve CSAF downloader resource management by limiting memory retention and closing HTTP response bodies on failed requests. Advisory data is now buffered locally for each download, preventing long-running workers from retaining the largest previously processed advisory in memory. This reduces memory pressure during large or continuous feed synchronisations and improves cleanup of network resources on error paths.