Reduce memory allocations for RI and POT generators by skatkov · Pull Request #1788 · ruby/rdoc

skatkov · 2026-08-26T12:08:57Z

Each documented method currently retains an array of RDoc::Parser::RubyColorizer::ColoredToken objects and their fragmented source strings.

This data is unnecessary for:

RI, whose serialized method representation excludes token_stream
POT, which extracts translatable documentation comments
Coverage reports, which generate no method-source pages

Large generated SDKs can contain millions of tokens. Avoiding those allocations substantially reduces parsing time and memory without changing generated output.

Implementation

Generators can declare whether they require method source:

class RDoc::Generator::RI
  def self.store_method_source? = false
end

class RDoc::Generator::POT
  def self.store_method_source? = false
end

The Ruby parser checks that capability before syntax highlighting a method body.

Generators without the capability default to true, preserving compatibility with existing third-party generators.

This also allows third-party generators (like rdoc-markdown) to skip this costly process if they don't have need for method.token_stream or method.markup_code.

If store_method_source? was disabled, but token_stream was still accessed, an error will be raised.

Benchmark

Here is a code that was used for benchmark this fix:
https://github.com/skatkov/rdoc-store-method-source-benchmark

RDoc 8.0.0 and google-api-client 0.53.0 on Ruby 4.0.6:

Variant	Parse RSS	Peak RSS	Colored tokens	Live heap slots	Time
Baseline	1,069.4 MiB	1,508.4 MiB	3,900,930	10,031,484	61.72s
Optimized	463.6 MiB	730.0 MiB	0	2,229,446	51.62s

Parse RSS reduction: 56.6%
Peak RSS reduction: 51.6%
Generated RI output: identical

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

RDoc currently syntax-highlights and retains every Ruby method body during parsing, regardless of whether the selected generator renders method source. This PR adds a generator capability for controlling method-source collection. RI and POT opt out, while Darkfish, Aliki, and unknown third-party generators retain the existing behavior. This reduces roughly 50% of allocations for generators that opt-out.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings August 26, 2026 12:08

skatkov requested a deployment to fork-preview-protection August 26, 2026 12:09 — with GitHub Actions Waiting