Skip to content

Bound the buffered read to the range the server declared - #39

Open
Gares95 wants to merge 1 commit into
gtsystem:masterfrom
Gares95:fix/bound-buffer-to-declared-range
Open

Bound the buffered read to the range the server declared#39
Gares95 wants to merge 1 commit into
gtsystem:masterfrom
Gares95:fix/bound-buffer-to-declared-range

Conversation

@Gares95

@Gares95 Gares95 commented Aug 23, 2026

Copy link
Copy Markdown

Description

PartialBuffer represents a declared byte range of a remote file, but on the
non-stream path it reads the whole response body:

self.buffer = buffer if stream else io.BytesIO(buffer.read())

buffer.read() with no argument consumes whatever the server chooses to send,
and it happens before size is consulted at all. So how much gets buffered is
decided by the server rather than by the range requested. Asking for a 100 byte
range from a server that streams 20 MB buffers all 20 MB:

server sends range requested bytes buffered
100 100 100
1,000,000 100 1,000,000
20,000,000 100 20,000,000

That defeats the point of using range requests, and for anything that runs
remotezip against a URL it does not control it is a way to exhaust memory
remotely.

There is a second route to the same place. fetch turns the server's
Content-Range straight into the size without checking it:

range_min, range_max = self.parse_range_header(range_header)
return PartialBuffer(res, range_min, range_max - range_min + 1, stream)

A header whose end precedes its start gives a negative size:

>>> RemoteFetcher.parse_range_header('bytes 100-50/1000')
(100, 50)          # size becomes 50 - 100 + 1 = -49

PartialBuffer.read(0) then computes a negative length, and file.read(negative)
means read everything for a Python file object. End to end, a client that asked
for 100 bytes received the entire 5000 byte body.

Separately, a header that cannot be parsed leaks a bare ValueError out of the
library: 'bytes abc-def/1000', 'bytes -500-100/1000', 'bytes /1000' and an
empty value all do this today. So does 'bytes */1000', which is the
RFC 7233 unsatisfied-range form, so this is not only about hostile input: a
server can send that legitimately.

The change

  • Read at most size bytes in PartialBuffer, in a loop that tolerates short
    reads. A single read(size) call is not sufficient: a socket-backed
    response can return fewer bytes than requested while more are still coming, so
    reading once would silently truncate. Worth stating because the existing tests
    all use BytesIO, which never short-reads, so that failure mode is invisible
    to them. The data is written straight into the result buffer, so no
    intermediate copy of the whole range is held.
  • Raise RemoteZipError for a Content-Range that ends before it starts or
    carries no end, and for one that cannot be parsed at all.

The suffix form that parse_range_header returns for bytes -123 is left
alone, since that is a request form rather than a response header, and its
existing test still passes. bytes 0-99/*, an unknown total length, remains
accepted and has a test to keep it that way.

What this does not fix

Worth being explicit: this bounds buffering by the size the server declares,
not by the size the client requested. A server that declares a very large
range and then streams it will still be buffered in full:

server declares server sends buffered
bytes 0-99/1000 5,000,000 100
bytes 0-4999999/5000000 5,000,000 5,000,000

Closing that would mean rejecting or clamping a response range that does not
match the requested one. I have not done it here because the right policy is a
judgement call: some servers legitimately return a different range than asked
for, and RemoteIO.seek derives _file_size from the declared size, so
clamping naively would desynchronise it. Happy to follow up if you have a
preference.

Tests

Four added. The first two fail against the current behaviour:

  • a server sending more than it declared does not enlarge the buffer;
  • an invalid or malformed Content-Range is rejected.

The other two guard the new code rather than the old bug, and pass before the
change as well, since reading to EOF never truncated:

  • short reads from the stream are handled without truncation;
  • a server sending less than it declared does not hang or raise.

Validation

python test_remotezip.py goes from 19 tests to 23, all passing, with no
existing test modified.

PartialBuffer represents a declared byte range of a remote file, but on the
non-stream path it called buffer.read() with no argument, consuming whatever the
server chose to send before the declared size was consulted. How much got
buffered was decided by the server rather than by the range requested: a client
asking for 100 bytes buffered 20 MB when the server streamed that much, which
defeats the purpose of fetching ranges at all.

Read at most `size` bytes instead, in a loop that tolerates short reads. A
single read() call is not enough: a socket-backed response can return fewer
bytes than requested while more are still coming, so reading once would silently
truncate. The data is written straight into the result buffer so no intermediate
copy of the whole range is held. The existing tests all use BytesIO, which never
short-reads, so these cases need their own tests.

RemoteFetcher.fetch also turned the server's Content-Range straight into that
size without checking it. A header whose end precedes its start, such as
"bytes 100-50/1000", produced a negative size, and PartialBuffer.read(0) then
computed a negative length, which for a file object means read everything.
Malformed values such as "bytes abc-def/1000", "bytes */1000" or an empty header
raised a bare ValueError out of the library. Both are now RemoteZipError.

Adds tests that a server sending more than it declared does not enlarge the
buffer, that a server sending less does not hang or raise, that short reads are
handled without truncation, and that invalid or malformed Content-Range values
are rejected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant