Bound the buffered read to the range the server declared - #39
Open
Gares95 wants to merge 1 commit into
Open
Conversation
PartialBuffer represents a declared byte range of a remote file, but on the non-stream path it called buffer.read() with no argument, consuming whatever the server chose to send before the declared size was consulted. How much got buffered was decided by the server rather than by the range requested: a client asking for 100 bytes buffered 20 MB when the server streamed that much, which defeats the purpose of fetching ranges at all. Read at most `size` bytes instead, in a loop that tolerates short reads. A single read() call is not enough: a socket-backed response can return fewer bytes than requested while more are still coming, so reading once would silently truncate. The data is written straight into the result buffer so no intermediate copy of the whole range is held. The existing tests all use BytesIO, which never short-reads, so these cases need their own tests. RemoteFetcher.fetch also turned the server's Content-Range straight into that size without checking it. A header whose end precedes its start, such as "bytes 100-50/1000", produced a negative size, and PartialBuffer.read(0) then computed a negative length, which for a file object means read everything. Malformed values such as "bytes abc-def/1000", "bytes */1000" or an empty header raised a bare ValueError out of the library. Both are now RemoteZipError. Adds tests that a server sending more than it declared does not enlarge the buffer, that a server sending less does not hang or raise, that short reads are handled without truncation, and that invalid or malformed Content-Range values are rejected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
PartialBufferrepresents a declared byte range of a remote file, but on thenon-stream path it reads the whole response body:
buffer.read()with no argument consumes whatever the server chooses to send,and it happens before
sizeis consulted at all. So how much gets buffered isdecided by the server rather than by the range requested. Asking for a 100 byte
range from a server that streams 20 MB buffers all 20 MB:
That defeats the point of using range requests, and for anything that runs
remotezip against a URL it does not control it is a way to exhaust memory
remotely.
There is a second route to the same place.
fetchturns the server'sContent-Rangestraight into the size without checking it:A header whose end precedes its start gives a negative size:
PartialBuffer.read(0)then computes a negative length, andfile.read(negative)means read everything for a Python file object. End to end, a client that asked
for 100 bytes received the entire 5000 byte body.
Separately, a header that cannot be parsed leaks a bare
ValueErrorout of thelibrary:
'bytes abc-def/1000','bytes -500-100/1000','bytes /1000'and anempty value all do this today. So does
'bytes */1000', which is theRFC 7233 unsatisfied-range form, so this is not only about hostile input: a
server can send that legitimately.
The change
sizebytes inPartialBuffer, in a loop that tolerates shortreads. A single
read(size)call is not sufficient: a socket-backedresponse can return fewer bytes than requested while more are still coming, so
reading once would silently truncate. Worth stating because the existing tests
all use
BytesIO, which never short-reads, so that failure mode is invisibleto them. The data is written straight into the result buffer, so no
intermediate copy of the whole range is held.
RemoteZipErrorfor aContent-Rangethat ends before it starts orcarries no end, and for one that cannot be parsed at all.
The suffix form that
parse_range_headerreturns forbytes -123is leftalone, since that is a request form rather than a response header, and its
existing test still passes.
bytes 0-99/*, an unknown total length, remainsaccepted and has a test to keep it that way.
What this does not fix
Worth being explicit: this bounds buffering by the size the server declares,
not by the size the client requested. A server that declares a very large
range and then streams it will still be buffered in full:
bytes 0-99/1000bytes 0-4999999/5000000Closing that would mean rejecting or clamping a response range that does not
match the requested one. I have not done it here because the right policy is a
judgement call: some servers legitimately return a different range than asked
for, and
RemoteIO.seekderives_file_sizefrom the declared size, soclamping naively would desynchronise it. Happy to follow up if you have a
preference.
Tests
Four added. The first two fail against the current behaviour:
Content-Rangeis rejected.The other two guard the new code rather than the old bug, and pass before the
change as well, since reading to EOF never truncated:
Validation
python test_remotezip.pygoes from 19 tests to 23, all passing, with noexisting test modified.