diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ead4340..577c0c4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -3,11 +3,9 @@ name: CI # Installs the bundle, builds the gem and runs the default test suite on the # Ruby versions this project validates continuously. # -# The supported range and the validated set are two different statements. The -# code is written for Ruby 3.3 through 4.0, which is what required_ruby_version -# says; the matrix below runs the ends of that range and the release in the -# middle, rather than every intermediate release. See doc/POLICY.md section 11 -# and doc/REQUIREMENTS.md section 20. +# The supported Ruby requirement lives in automatic.gemspec. The matrix +# below is the authoritative set validated continuously; see +# doc/POLICY.md section 11 and doc/REQUIREMENTS.md section 20. # # No credential is configured here and nothing reaches an external service: the # suite must need neither (doc/POLICY.md Invariant 6). Examples tagged :network @@ -15,8 +13,8 @@ name: CI # installed, so no plugin's own gem is a condition of this workflow passing. # # This is the required check, and it is deliberately the minimal configuration: -# what it proves on every commit is that the framework needs nothing but its -# four runtime dependencies. The all-plugins configuration is a separate, +# what it proves on every commit is that the framework needs only the +# runtime dependencies declared in automatic.gemspec. The all-plugins configuration is a separate, # non-required workflow, plugins.yml. on: @@ -41,9 +39,10 @@ jobs: strategy: fail-fast: false matrix: - # The continuously validated versions. Keep this in step with the - # supported environment section of README.md; the floor here and - # required_ruby_version in automatic.gemspec are the same version. + # The continuously validated versions. This matrix is authoritative for + # that set; automatic.gemspec is authoritative for the supported Ruby + # requirement. Documentation refers to these sources rather than acting as + # a second matrix. ruby: ['3.3', '3.4', '4.0'] steps: diff --git a/.github/workflows/plugins.yml b/.github/workflows/plugins.yml index be463bb..6d43873 100644 --- a/.github/workflows/plugins.yml +++ b/.github/workflows/plugins.yml @@ -1,8 +1,8 @@ name: Optional plugin dependencies # Installs the Gemfile's optional `plugins` group and runs the same suite, so -# that the second documented way of setting up a checkout is checked as well as -# described: the group resolves, and the plugin specs it brings in pass. +# that the documented all-optional-plugin checkout setup is checked as well +# as described: the group resolves, and the plugin specs it brings in pass. # # This is not the required check. ci.yml is, and it installs none of this: what # has to hold on every commit is that the framework runs on its own runtime diff --git a/Gemfile b/Gemfile index 517cd77..8f9202c 100644 --- a/Gemfile +++ b/Gemfile @@ -3,8 +3,8 @@ # which this file evaluates. Only the optional, plugin-specific gems are listed # here. See doc/POLICY.md section 9. # -# `bundle install` with no configuration installs the framework's four runtime -# dependencies and the development ones, and nothing below: a checkout is set +# `bundle install` with no configuration installs the framework's runtime +# dependencies declared in automatic.gemspec and the development ones, and nothing below: a checkout is set # up to run and to test the framework, not to run every plugin. source 'https://rubygems.org' @@ -16,9 +16,10 @@ gemspec # and a Recipe that does not use the plugin does not need them. # # Every group here is optional, so nothing below is installed by default and -# neither the default test suite nor required CI depends on any of it. Each gem -# is in two groups: `plugins`, which is all of them at once, and one named -# after what it is for, which is one of them on its own. Both are Bundler +# neither the default test suite nor required CI depends on any of it. +# Gems in the aggregate optional-plugin set belong to `plugins` and to a +# purpose-specific group. The group declarations below are the source of +# truth for that membership. Both are Bundler # groups and both are selected the same way, in the checkout's own .bundle # directory, which is not committed: # diff --git a/README.md b/README.md index b7a4a6c..58152e4 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Fluentd, or writes them to a database. Write your own plugin and it composes with all the others. **[Follow the Quick Start](doc/QUICKSTART.md)** to install the gem, write a -Recipe that watches four public index pages, install what that Recipe needs, and +Recipe that watches public index pages, install what that Recipe needs, and produce Markdown from the articles they list. --- @@ -98,8 +98,10 @@ exactly three things: over, - **a loader** that finds a step by name, so that a Recipe can name it. -Everything else is a plugin. The framework is under seven hundred lines of Ruby -and is meant to stay that size. +Everything else is a plugin. The framework is intentionally kept small; its +responsibilities and architectural boundary are defined in +[`doc/BASIC_DESIGN.md`](doc/BASIC_DESIGN.md) and +[`doc/POLICY.md`](doc/POLICY.md). It is one person's tooling, run unattended from `cron`, against their own accounts and their own files. It is not a service, and there is no notion of a @@ -115,8 +117,8 @@ a plugin set every part of which still has somewhere to talk to. See - **Recipes in YAML.** A job is a file, not a program. No Ruby is written to wire a pipeline together. -- **Plugins across seven categories.** Subscribe, custom feed, filter, store, - provide, notify and publish plugins compose through the same pipeline contract. +- **Plugins organized by role.** Subscribe, custom feed, filter, store, provide, + notify and publish plugins compose through the same pipeline contract. - **Markdown out of the box.** `PublishMarkdown` writes the result as a plain Markdown document, to a file or to standard output, with no service and no credential behind it. It is the natural end of a new Recipe. @@ -130,9 +132,9 @@ a plugin set every part of which still has somewhere to talk to. See - **Retry and interval** on everything that reaches the network, configured per plugin in the Recipe. - **A small installation.** A gem needed by one plugin is not a dependency of - the framework: `gem install automatic` brings four pure-Ruby gems and the - command, and installs neither an HTML parser nor a database — let alone an - AWS SDK. + the framework: `gem install automatic` brings the framework's runtime + dependencies declared in `automatic.gemspec` and the command, and installs + neither an HTML parser nor a database — let alone an AWS SDK. - **No museum.** Every plugin is classified, with its reason, in [`doc/PLUGINS.md`](doc/PLUGINS.md). Nothing dead is stubbed into looking alive, and an integration whose service has gone is removed rather than @@ -184,7 +186,8 @@ The full account is [`doc/BASIC_DESIGN.md`](doc/BASIC_DESIGN.md). ## 4. Supported environment -- **Ruby 3.3 through 4.0.** CI validates 3.3, 3.4 and 4.0. +- **Ruby 3.3 through 4.0.** The continuously validated versions are the matrix + in [`.github/workflows/ci.yml`](.github/workflows/ci.yml). - A Unix-like system. GNU/Linux and macOS are what it is used on. Windows is not supported. - A compiler only if you install an optional plugin gem that builds from source @@ -199,10 +202,11 @@ Two statements, and they are not the same one: - **Supported range.** The code is written for Ruby 3.3 through 4.0, using APIs the whole range shares. `required_ruby_version` is `>= 3.3.0` and has no upper bound, so a Ruby newer than the matrix is permitted rather than refused. -- **Continuously validated versions.** CI runs the ends of the range and the - release in the middle — 3.3, 3.4 and 4.0 — rather than every intermediate - release. A version's absence from the matrix means it is not verified on every - commit; it does not mean it is expected to fail. +- **Continuously validated versions.** The authoritative set is the matrix in + [`.github/workflows/ci.yml`](.github/workflows/ci.yml), which represents the + supported range rather than enumerating every intermediate release. A + version's absence from the matrix means it is not verified on every commit; + it does not mean it is expected to fail. ## 5. Installation @@ -213,7 +217,8 @@ gem install automatic automatic --version ``` -That installs the framework, the command and four pure-Ruby dependencies. +That installs the framework, the command and the runtime dependencies +declared in `automatic.gemspec`. A gem that only one plugin needs is not among them: install it when you use that plugin, with `gem install nokogiri` or `gem install activerecord sqlite3`. [`doc/DEPLOYMENT.md`](doc/DEPLOYMENT.md) lists which plugin needs which. @@ -221,8 +226,7 @@ that plugin, with `gem install nokogiri` or `gem install activerecord sqlite3`. ### From a checkout Use a checkout to try the current development version, change the source, -develop a plugin or verify changes before a release. There are three ways to -set one up; start with the first. +develop a plugin or verify changes before a release. The supported checkout setups are shown below; start with the minimal setup. ```sh git clone https://github.com/id774/automaticruby.git @@ -262,8 +266,9 @@ a Recipe taken step by step through choosing its groups are in ## 6. Quick start The complete first-run guide is [`doc/QUICKSTART.md`](doc/QUICKSTART.md): it -writes one Recipe that reads four public index pages, installs what that Recipe -needs, and leaves the new articles in a Markdown document. +writes one Recipe that reads the public index pages listed by that guide, +installs what that Recipe needs, and leaves the new articles in a Markdown +document. ```sh automatic scaffold @@ -280,7 +285,9 @@ nothing but the framework and what `gem install automatic` brought. `feed2console.yml` beside it is the same pipeline printing to the terminal. Read the file, `grep` it, put it in a repository, or hand it to whatever reads text next. A Recipe that names a plugin with an optional gem installs that gem first, -which is step 4 of the Quick Start and the habit worth learning early. +which is covered by the "Install what the Recipe needs" section of +[`doc/QUICKSTART.md`](doc/QUICKSTART.md) and is the habit worth learning +early. To check the framework without any network, write this instead: @@ -333,7 +340,7 @@ plugins: # required - `global.log.level` is the only framework setting. `global.timezone` and `global.cache` appear in old Recipes and are read by nothing. -Two conventions worth knowing before writing one: +Conventions worth knowing before writing one: - **Put a store plugin in front of anything with an effect.** `StorePermalink` records what has been seen and passes on only what has not. It is what makes a @@ -392,25 +399,21 @@ like a shipped plugin replaces it. ### Which plugins still work -Every shipped plugin is classified in -[`doc/PLUGINS.md`](doc/PLUGINS.md) section 6, with its settings and the reason -for its status: +The complete shipped-plugin catalogue and the current status of each plugin +are maintained in +[`doc/PLUGINS.md`](doc/PLUGINS.md#6-the-plugins). +The status vocabulary and its meaning are defined in +[Reading the catalogue](doc/PLUGINS.md#5-reading-the-catalogue), and removed +integrations and the reasons for their removal are recorded in +[Plugins that were removed](doc/PLUGINS.md#8-plugins-that-were-removed). -| Status | Meaning | -| --- | --- | -| **Supported** | Works on the supported Rubies with current dependencies | -| **Supported (external)** | Works, but needs something you provide: a service, a command, a credential, a data file | -| **Needs rework** | The service exists; this plugin speaks a replaced interface | - -Eleven plugins were removed in this release rather than kept as history: each -talked to a service that has shut down, or through an API that has been -withdrawn with no replacement. They are listed with their reasons in -[`doc/PLUGINS.md`](doc/PLUGINS.md) section 8, and Git history holds the code. -A Recipe naming one of them now fails at load, before anything runs. +This README intentionally does not repeat current plugin totals, per-status +totals, or the current membership of a status. Adding, removing, or +reclassifying a plugin changes the canonical catalogue rather than a +synchronized summary here. -`PublishHatenaBookmark` is currently classified as **Needs rework**; restoring -it to the service's current interface is self-contained work and a good first -contribution. +A plugin classified as **Needs rework** in the catalogue is a +self-contained contribution candidate. No plugin here is stubbed, mocked or simulated to make a test pass. Where a plugin's gem is not installed its spec is skipped and says which gem is @@ -449,8 +452,8 @@ automatic --version | `opmlparser ` | Print the feed URLs in an OPML file. | | `log ` | Emit one line in the framework's log format. | -The middle five answer "will this work as a Recipe input?" before you write the -Recipe: +The discovery and parsing subcommands help answer "will this work as a Recipe +input?" before you write the Recipe: ```sh automatic autodiscovery https://example.com/ @@ -474,8 +477,9 @@ There is no configuration file besides the Recipe. Every setting a job needs is in the Recipe that defines the job, which is what makes a Recipe portable between machines. -**Framework settings** — one, `global.log.level`, with the values `info`, -`warn`, `error` and `none`. +**Framework setting:** `global.log.level`, with the values `info`, `warn`, +`error` and `none`. Its Recipe-level contract is specified in +[`doc/PLUGINS.md`](doc/PLUGINS.md#24-global). **Plugin settings** — the `config` mapping of a plugin entry, passed to that plugin and read by nothing else. Established names: `retry` for an attempt @@ -576,8 +580,8 @@ bundle exec rake bundle exec bin/automatic -c config/feed2console.yml ``` -Contributions are welcome — a new plugin, or reviving the one that needs -rework, most of all. +Contributions are welcome — a new plugin, or restoring a plugin currently +classified as **Needs rework** in the canonical catalogue, most of all. 1. Fork the repository. 2. Write the change, with a spec that reaches no network. @@ -590,7 +594,7 @@ judged by: the direction of dependency, where a new capability belongs, how dependencies are added, how the documents divide, and how a version history entry is written. -Two rules worth knowing before you start: +Rules worth knowing before you start: - **A gem needed by one plugin is not a dependency of the framework.** Require it at the top of the plugin's own file. diff --git a/automatic.gemspec b/automatic.gemspec index 2df8ea7..863d734 100644 --- a/automatic.gemspec +++ b/automatic.gemspec @@ -87,8 +87,9 @@ Gem::Specification.new do |spec| spec.extra_rdoc_files = ['README.md', 'doc/LICENSE.md'] # Runtime dependencies: what the framework in lib/ requires, and nothing - # else. Requiring `automatic`, loading a Recipe, loading a plugin, running a - # pipeline and the CLI's own work are what these four are for. + # else. Requiring `automatic`, loading a Recipe, loading a plugin, + # running a pipeline and the CLI's own work are what these dependencies are + # for. # # A gem needed by a plugin is NOT declared here, however useful that plugin # is. It is required inside the plugin's own file and installed by the diff --git a/doc/AI_TUTORIAL.md b/doc/AI_TUTORIAL.md index 1e26155..34b0b8f 100644 --- a/doc/AI_TUTORIAL.md +++ b/doc/AI_TUTORIAL.md @@ -41,7 +41,7 @@ FilterSakuraAI ask one question about that text PublishMarkdown write the answer to a document ``` -Seven plugins, each doing one thing, each handing its result to the next. The +Each plugin does one thing and hands its result to the next. The Recipe that expresses it is in section 4, and nothing in it is a special case: every entry is a plugin the framework loads by name, and the order of the list is the order of the work. @@ -133,7 +133,7 @@ gem install activerecord sqlite3 gem install sanitize ``` -In a checkout the same three groups are selected together: +In a checkout, select the groups required by those plugins together: ```sh bundle config set --local with "html store sanitize" @@ -153,22 +153,27 @@ link, its date and whatever text the pipeline could get. No AI service has been contacted, no credential exists yet, and the Recipe is complete in itself: a person who only ever wanted this can stop here and put it in `cron`. -**Expect `FilterFullFeed` to find nothing for these three sites, and read the -log to see it.** The shipped `assets/siteinfo/items_all.json` is a snapshot of -the LDRFullFeed database whose newest entries are from 2013, and none of its URL -patterns matches these sites: +**Read the `FilterFullFeed` log rather than assuming bundled siteinfo +coverage.** The shipped siteinfo file is a snapshot. If a record matches an +article URL, `FilterFullFeed` fetches the page and applies that record's +XPath. If no record matches, the existing item is preserved and the log says: ```text -Fulltext SITEINFO not found: https://go.dev/blog/pkgsite-api +Fulltext SITEINFO not found: https://example.com/article ``` +A Recipe that relies on full article bodies should supply siteinfo for the +sites it relies on under `~/.automatic/assets/siteinfo/`, rather than treating +the bundled snapshot as a guarantee of coverage. + Where no record matches, the plugin leaves the item exactly as it arrived, and the run continues. That is the behaviour to rely on and also the reason the document above holds a title, a link and a date per article and no body: an index page read without a `description_selector` carries no description for `FilterFullFeed` to have improved on. -Getting text into these items is a choice between two places, and both are +Text can enter these items from the index page or from the article page, +and both are configuration rather than code: - **From the index page.** Where the listing prints a summary, an @@ -241,7 +246,7 @@ Now the AI filter goes between `FilterJoin` and `PublishMarkdown`: - module: FilterSakuraAI config: token: YOUR_SAKURA_AI_TOKEN - model: gpt-oss-120b + model: prompt: | 以下の記事群について、個別記事の要約を羅列するのではなく、 全体を一つのダイジェストとして日本語で要約してください。 @@ -291,7 +296,7 @@ plugins: - module: FilterSakuraAI config: token: YOUR_SAKURA_AI_TOKEN - model: gpt-oss-120b + model: prompt: | 以下の記事群について、個別記事の要約を羅列するのではなく、 全体を一つのダイジェストとして日本語で要約してください。 @@ -341,7 +346,7 @@ Move the AI filter to before `FilterJoin` and change nothing else: - module: FilterSakuraAI config: token: YOUR_SAKURA_AI_TOKEN - model: gpt-oss-120b + model: prompt: | 以下の記事を日本語で三行に要約してください。 retry: 2 @@ -374,16 +379,16 @@ and that no plugin had to be changed or configured to express it. ## 6. Change the service by changing one line -The four AI filters — `FilterOpenAI`, `FilterClaude`, `FilterGemini` and -`FilterSakuraAI` — are one per service rather than one plugin with a `provider` -setting. Each replaces an item's description with what its service answers, so +Automatic Ruby uses separate service-specific AI filters rather than one +plugin with a `provider` setting. Their current catalogue and settings are +maintained in [`PLUGINS.md`](PLUGINS.md) section 6.3. Each replaces an item's description with what its service answers, so in a Recipe they are interchangeable at the same position: ```yaml - module: FilterOpenAI config: token: YOUR_OPENAI_API_KEY - model: gpt-5.6 + model: prompt: | 以下の記事群を一つのダイジェストとして日本語で要約してください。 retry: 2 @@ -394,7 +399,7 @@ in a Recipe they are interchangeable at the same position: - module: FilterClaude config: token: YOUR_ANTHROPIC_API_KEY - model: claude-opus-5 + model: prompt: | 以下の記事群を一つのダイジェストとして日本語で要約してください。 max_tokens: 2048 @@ -406,7 +411,7 @@ in a Recipe they are interchangeable at the same position: - module: FilterGemini config: token: YOUR_GEMINI_API_KEY - model: gemini-3.5-flash + model: prompt: | 以下の記事群を一つのダイジェストとして日本語で要約してください。 retry: 2 @@ -420,13 +425,14 @@ you are using rather than from this document, and see [`PLUGINS.md`](PLUGINS.md) section 6.3 for each plugin's endpoint, authentication and answer handling. -That a Recipe names the service on its face is the reason for four plugins. A +A Recipe naming the service on its face is the reason for keeping +service-specific plugins. A line reading `FilterClaude` says where the text is going, which is a question worth being able to answer by reading the Recipe. ## 7. Change the prompt, and the Recipe does another job -None of the four filters is a summarizer. The prompt is the instruction and the +None of the AI filters is a summarizer. The prompt is the instruction and the item's description is the text it applies to, so summarizing, translating, extracting and classifying are the same plugin with different words in one setting. Keeping the Recipe of section 4 and replacing only the `prompt`: @@ -457,15 +463,17 @@ still worth remembering what the input is: pages fetched from the open web. A prompt that states what to do when the text does not contain what was asked for is more robust than one that assumes it does. -**Long input has a limit that belongs to the model, not to the framework.** A -joined text of forty full articles can exceed what a model accepts, and the -service answers with an error that ends the run. Fewer sites, a smaller -`fetch_items`, a stricter `include`, or the per-article arrangement of section 5 -are the ways to stay under it with the plugins that ship today. Splitting one -long text into pieces, transforming each and asking a final question about the -results is the natural next arrangement conceptually, and it is **not** -something this repository provides: there is no chunking plugin, and a Recipe -cannot express it today. +**Long input has a limit that belongs to the model, not to the framework.** +A joined text can exceed what a model accepts, and the service then returns an +error that ends the run. `FilterLimit` can cap how many source items proceed, +and `FilterBatch` can group source items into fixed-size item batches before +an AI filter so that one request need not contain the whole pipeline. Their +exact behavior is specified in [`PLUGINS.md`](PLUGINS.md) section 6.3. + +`FilterBatch` does not split one already-joined text by character or token +length, and Automatic Ruby does not perform recursive summarization or a +final synthesis automatically. A Recipe that needs those operations still +needs a different pipeline design or a purpose-built component. ## 8. Why it is built this way diff --git a/doc/DEPLOYMENT.md b/doc/DEPLOYMENT.md index c8319af..08e4a4c 100644 --- a/doc/DEPLOYMENT.md +++ b/doc/DEPLOYMENT.md @@ -28,15 +28,16 @@ nothing to stop. - A Unix-like system. GNU/Linux and macOS are what this is used on; Windows is not supported. -- **Ruby 3.3 through 4.0.** Check with `ruby -v`. CI validates 3.3, 3.4 and 4.0; - a version between them is supported and is simply not checked on every commit, - and a Ruby newer than 4.0 is permitted rather than refused. See - [`REQUIREMENTS.md`](REQUIREMENTS.md) section 20. +- **Ruby 3.3 through 4.0.** Check with `ruby -v`. The continuously validated + versions are the matrix in [`.github/workflows/ci.yml`](../.github/workflows/ci.yml); + a supported version absent from that matrix is simply not checked on every + commit, and a Ruby newer than the matrix is permitted rather than refused. + See [`REQUIREMENTS.md`](REQUIREMENTS.md) section 20. - Optional gems for particular plugins, listed in the table under "Optional plugin dependencies" below. None is needed to install Automatic Ruby, or to run a Recipe that does not use the plugin; a Recipe that does - name one installs it as a step of its own, which is what the Quick Start's - step 4 is. + name one installs it as a step of its own, which is what the "Install what + the Recipe needs" section of [`QUICKSTART.md`](QUICKSTART.md) demonstrates. - A build environment may be needed for one of those optional gems — `nokogiri` and `sqlite3` build a native extension where no binary package matches your platform. The framework's own dependencies are pure Ruby, so the normal @@ -52,9 +53,9 @@ gem install automatic automatic --version ``` -That installs the framework, the `automatic` command and four pure-Ruby -runtime dependencies: `activesupport`, `hashie`, `rexml` and `rss`. That is the -whole of it. No HTML parser, no database, no service client — a gem needed by +That installs the framework, the `automatic` command and the runtime +dependencies declared in `automatic.gemspec`. That gemspec is the source of +truth for the framework's runtime dependency set. No HTML parser, no database, no service client — a gem needed by one plugin is installed by the operator who uses that plugin, so installing Automatic Ruby does not install what your Recipes do not use. @@ -71,7 +72,7 @@ which, and is the list to check before adding anything. ### From a checkout For working on the framework, or for running a version that is not released. -There are three ways to set one up, and the first is the one to start with. +The supported checkout setups are shown below; start with the minimal setup. **Minimal — the framework and its test suite.** What you want for running the checkout, and for developing the framework itself: @@ -99,8 +100,10 @@ bundle install bundle exec rake ``` -That adds `activerecord`, `sqlite3`, `nokogiri`, `sanitize` and `feedbag`, -and their specs then run as part of the ordinary suite. The setting +That installs the gems declared in the `plugins` group of `Gemfile`, and +their specs then run as part of the ordinary suite. The +[Optional plugin dependencies](#optional-plugin-dependencies) table maps +those dependencies to the plugins and tools that use them. The setting is written to the checkout's own `.bundle/config`, which is not committed; `bundle config unset --local with` returns the checkout to the minimum, and `bundle install` afterwards. @@ -115,7 +118,7 @@ bundle install ``` Several at once are space-separated: `bundle config set --local with "store -html"`, and a Recipe using plugins from two groups needs exactly that. The group +html"`, and a Recipe using plugins from multiple groups needs exactly that. The group names are in the table below, and "Working out what a Recipe needs, in a checkout" takes one Recipe through choosing them. @@ -162,8 +165,8 @@ full path runs without it. automatic -c ~/.automatic/config/example/feed2console.yml ``` -That fetches one feed and prints its items. Two things can go wrong and both are -worth telling apart: +That fetches one feed and prints its items. Distinguish an unreachable feed +from an optional-dependency failure: - A message about the feed being unreachable means the network or that particular feed, not the installation. @@ -230,12 +233,12 @@ plugins: interval: 3 ``` -That Recipe needs three optional gems, because of the plugins it names rather -than because of the framework: `nokogiri` for `FilterImageSource`, and -`activerecord` and `sqlite3` for `StorePermalink`. See "Optional plugin -dependencies" below. +That Recipe needs `nokogiri` for `FilterImageSource`, and `activerecord` plus +`sqlite3` for `StorePermalink`, because of the plugins it names rather than +because of the framework. The canonical mapping is the +[Optional plugin dependencies](#optional-plugin-dependencies) table. -Three things in that Recipe are the operational advice of this document: +The operational advice illustrated by that Recipe is: - **`StorePermalink` before the plugin with the effect.** It records what has been seen and passes on only what has not, which is what makes the Recipe safe @@ -318,8 +321,7 @@ mkdir -p ~/notes && chmod 700 ~/notes **Keeping the document clean.** With no `file`, the document goes to standard output — and so does the log ([`REQUIREMENTS.md`](REQUIREMENTS.md) section 14), -so a redirect that collects one collects the other. Two ways out, and the Recipe -chooses: +so a redirect that collects one collects the other. Either disable log output in the Recipe or give `PublishMarkdown` a `file`: ```yaml global: @@ -446,8 +448,10 @@ Recipe is stated there in terms you can act on. ## Optional plugin dependencies -This table is the list. Which plugin needs which gem, how to install it, and -whether the plugin still works are all here, and nothing else repeats it. +This table is the operator-facing source of truth for which plugin or tool +needs which optional dependency and how to install it. Current plugin support +status is maintained separately in +[`PLUGINS.md`](PLUGINS.md#6-the-plugins). None of these gems is installed by `gem install automatic` or by a default `bundle install`. Install one only if you use the plugin. @@ -456,26 +460,27 @@ None of these gems is installed by `gem install automatic` or by a default --local with ` and `bundle install`, because `bundle exec` sees only the bundle. `plugins` is every group in the first block at once. -| Plugin | Needs | Installed gem | Checkout group | Status | -| --- | --- | --- | --- | --- | -| `StorePermalink`, `StoreFullText`, `StoreDigest` | `activerecord`, `sqlite3` | `gem install activerecord sqlite3` | `store` | Supported | -| `FilterImageSource`, `FilterDescriptionLink`, `SubscriptionLink`, `SubscriptionTumblr`, `CustomFeedWeb` | `nokogiri` | `gem install nokogiri` | `html` | Supported (`SubscriptionTumblr` external) | -| `PublishMarkdown` | `nokogiri`, for HTML bodies only | `gem install nokogiri` | `html` | Supported; runs without it | -| `FilterSanitize` | `sanitize` | `gem install sanitize` | `sanitize` | Supported | -| `autodiscovery` and `inspect` subcommands | `feedbag` | `gem install feedbag` | `autodiscovery` | Supported | -| `FilterFullFeed` | `nokogiri`, and a siteinfo file | `gem install nokogiri` | `html` | Supported (external) | -| `CustomFeedSVNLog` | the `svn` command; no gem | — | — | Supported (external) | -| `ProvideFluentd`, `PublishFluentd` | `fluent-logger`, and a Fluentd instance | `gem install fluent-logger` | `fluentd` | Supported (external) | -| `PublishMemcached` | `dalli`, and a memcached server | `gem install dalli` | `memcached` | Supported (external) | -| `PublishAmazonS3`, `StoreFile` S3 path | `aws-sdk-s3`, and a bucket | `gem install aws-sdk-s3` | `s3` | Supported (external) | -| `PublishInstapaper` | an Instapaper account; no gem | — | — | Supported (external) | -| `PublishEject` | the `eject` or `drutil` command | — | — | Supported (external) | -| `NotifyIkachan` | an `ikachan` gateway you run | — | — | Supported (external) | -| `FilterOpenAI`, `FilterClaude`, `FilterGemini`, `FilterSakuraAI` | an account and an API token with that one service; no gem | — | — | Supported (external) | -| `PublishHatenaBookmark` | the current Hatena API, which it does not speak | — | — | Needs rework | - -The `plugins` group is the first five rows: the optional gems of the plugins -whose specs need nothing but the gem. The gems below it are in their own groups +| Plugin | Needs | Installed gem | Checkout group | +| --- | --- | --- | --- | +| `StorePermalink`, `StoreFullText`, `StoreDigest` | `activerecord`, `sqlite3` | `gem install activerecord sqlite3` | `store` | +| `FilterImageSource`, `FilterDescriptionLink`, `SubscriptionLink`, `SubscriptionTumblr`, `CustomFeedWeb` | `nokogiri` | `gem install nokogiri` | `html` | +| `PublishMarkdown` | `nokogiri`, for HTML bodies only | `gem install nokogiri` | `html` | +| `FilterSanitize` | `sanitize` | `gem install sanitize` | `sanitize` | +| `autodiscovery` and `inspect` subcommands | `feedbag` | `gem install feedbag` | `autodiscovery` | +| `FilterFullFeed` | `nokogiri`, and a siteinfo file | `gem install nokogiri` | `html` | +| `CustomFeedSVNLog` | the `svn` command; no gem | — | — | +| `ProvideFluentd`, `PublishFluentd` | `fluent-logger`, and a Fluentd instance | `gem install fluent-logger` | `fluentd` | +| `PublishMemcached` | `dalli`, and a memcached server | `gem install dalli` | `memcached` | +| `PublishAmazonS3`, `StoreFile` S3 path | `aws-sdk-s3`, and a bucket | `gem install aws-sdk-s3` | `s3` | +| `PublishInstapaper` | an Instapaper account; no gem | — | — | +| `PublishEject` | the `eject` or `drutil` command | — | — | +| `NotifyIkachan` | an `ikachan` gateway you run | — | — | +| `FilterOpenAI`, `FilterClaude`, `FilterGemini`, `FilterSakuraAI` | an account and an API token with that one service; no gem | — | — | +| `PublishHatenaBookmark` | the current Hatena API, which it does not speak | — | — | + +Membership of the aggregate `plugins` group is defined in `Gemfile`. It +contains the optional gems whose corresponding specs need no external +service; the table above maps those dependencies to their consumers. The gems below it are in their own groups only, because each of those plugins also needs a service, a bucket or a command, and installing a gem alone would not make the plugin — or its spec — work. @@ -546,7 +551,7 @@ plugins: [`QUICKSTART.md`](QUICKSTART.md) runs that Recipe end to end, with the sites it watches and what each plugin does. What follows is the part a checkout does -differently: turning those three plugins into a bundle that can run them. +differently: turning the Recipe's plugins into a bundle that can run them. **1. List the plugins the Recipe names.** They are the `module` lines, in order: `CustomFeedWeb`, `StoreDigest`, `PublishMarkdown`. @@ -693,7 +698,7 @@ bundle exec ruby -ractive_record -e 'puts ActiveRecord::VERSION::STRING' bundle exec ruby -rsqlite3 -e 'puts SQLite3::VERSION' ``` -Those three `require` the libraries the way the plugins do — note +Those commands `require` the libraries the way the plugins do — note `active_record` for the `activerecord` gem — under the same bundle the Recipe will run under. `gem list` answers a different question and is the one to distrust here: it lists what RubyGems has, which in a checkout is neither what @@ -701,7 +706,8 @@ the plugins will load nor what a missing-gem message is about. ### All of the optional gems, or only the ones a Recipe names -Two ways to select groups, for two purposes: +Choose the aggregate `plugins` group for plugin development, or +purpose-specific groups for the Recipe being run: ```sh bundle config set --local with plugins # all of them, at once diff --git a/doc/PLUGINS.md b/doc/PLUGINS.md index bce8dca..af93206 100644 --- a/doc/PLUGINS.md +++ b/doc/PLUGINS.md @@ -158,8 +158,8 @@ The value is stored and is read correctly, because plugins read their settings by string key rather than as a property. This is noise, not breakage, and a Recipe using such a key needs no change. -Two shipped plugins have such a name, from before this was understood, and they -keep it: `FilterSort`'s `sort` and `PublishMemcached`'s `key`. Renaming them +The existing collisions retained for compatibility are `FilterSort`'s `sort` +and `PublishMemcached`'s `key`. Renaming them would break every Recipe using them, which is not a trade worth making for a warning. **A new plugin should not introduce one**: prefer `max_length` to `max`, `item_count` to `count`, `cache_key` to `key`. @@ -265,7 +265,7 @@ found. ### 3.3 Discovery and precedence -Two search roots, in this order: +Search roots, in this order: 1. `~/.automatic/plugins//.rb` 2. `/plugins//.rb` @@ -276,7 +276,7 @@ plugin's behaviour without editing the installation. Creating a new category is creating a directory. `~/.automatic/plugins/mine/` plus a class named `MineSomething` works with no change to the framework, though -staying inside the seven categories is preferred, because their names tell a +staying inside the established categories listed in section 3.2 is preferred, because their names tell a reader where in a pipeline the plugin belongs. Loading is lazy: the loader registers an `autoload`, so the file is read when @@ -558,11 +558,11 @@ Section 6 lists every plugin shipped in the gem. Each carries a status: | **Supported (external)** | The plugin is current, but it needs something the operator provides — a running service, an installed command, a credential, a data file. | | **Needs rework** | The service and the capability still exist, but this plugin speaks an interface that has been replaced. It will not work as written, and restoring it is a self-contained piece of work. | -There is no fourth row. There used to be one, holding plugins whose service had -shut down, and the plugins that were in it have been removed rather than kept: -see section 8. +No status is used for an integration whose service or capability is +permanently unavailable. Such plugins are removed rather than retained; see +section 8. -Two rules govern this table, and they are the reason it exists at all: +The rules that govern this table are the reason it exists at all: - **Nothing is faked.** A plugin is not stubbed, mocked or simulated to make a test pass or a catalogue entry look better. Where a plugin's gem is absent @@ -584,15 +584,13 @@ its spec as part of the ordinary suite, which is also what the separate plugin is not demoted for needing a gem, and is not promoted by a test that CI never executes. -**This classification is a snapshot taken in August 2026,** based on the -published status of each service and on what each plugin's code actually calls. -The statuses that depend on an outside service can change without any commit -here. Where a status was reached from published information rather than from a +**Statuses involving an outside service are snapshots of external state.** +They can change without any commit here. Where a status was reached from published information rather than from a live check, the entry says so. To verify one yourself, run its Recipe from `test/integration` by hand; those are not part of CI and never will be. -Restoring the one **Needs rework** plugin is a self-contained piece of work and -a good first contribution. +Restoring a plugin classified as **Needs rework** is a self-contained piece +of work and a good first contribution. --- @@ -1075,34 +1073,27 @@ fetch attempt whether it succeeded or failed. A non-positive or non-numeric Needs `nokogiri`: `gem install nokogiri`, or the `html` group in a checkout. -The shipped `assets/siteinfo/items_all.json` is a snapshot of the LDRFullFeed -database taken from `wedata.net`, which no longer operates, so the file cannot -be refreshed from its origin and its newest entries are from 2013. The plugin -works; how well it works depends on whether the sites you read are in that -snapshot and still laid out the same way. Supplying your own file in -`~/.automatic/assets/siteinfo/` is the way to keep it useful. - -Three things follow from the database being that old, and the plugin now -accounts for each: - -- **A link matches under either scheme.** 3,448 of the 3,504 usable records - anchor on a scheme and all but twenty of those say `^http://`. The sites they - name have since moved to HTTPS, which is what a feed hands over, so matching - the link as it stands matched almost nothing and the filter quietly did - nothing at all. A record describes a site's layout, not how it is - transported, so the link is tried under both. Only the match is rewritten; - the page is fetched from the link the feed gave. -- **A record that selects nothing leaves the summary alone.** A site redesigned - since its XPath was written selects no nodes, and putting that empty result - into the item replaced a perfectly good summary with an empty description. - The item keeps what it arrived with, and the miss is logged at `warn` with - the XPath that missed. -- **The page's own encoding is believed before the record's.** The page is - parsed from the stream, so a charset in a `meta` tag is read even when the - response declared none. A record's `enc` is the fallback for a page that - declares nothing anywhere — 1,186 records carry one, mostly EUC-JP and - Shift_JIS — and an `enc` naming an encoding Ruby does not have is ignored - rather than raised. What comes out is UTF-8 either way. +The shipped `assets/siteinfo/items_all.json` is a bundled LDRFullFeed-compatible +siteinfo snapshot. Its coverage is a snapshot rather than a guarantee that a +site is present or still uses the recorded layout. The bundled file may be +replaced from a newer upstream snapshot, as recorded in +[`VERSIONS`](VERSIONS); an operator can also supply a siteinfo file under +`~/.automatic/assets/siteinfo/`. + +The plugin's behavior does not depend on the bundled snapshot having a +particular record count, date, scheme distribution, or encoding distribution: + +- **A link matches under either HTTP scheme.** A stored URL pattern can name + `http://` while a current feed supplies `https://`, or the reverse. Matching + therefore tries both schemes while fetching the page from the original item + link. +- **A record that selects nothing leaves the summary alone.** If the recorded + XPath no longer selects content, the existing description is preserved and + the miss is logged at `warn`. +- **The page's own encoding is believed before the record's.** Response or + document declarations take precedence; the record's `enc` is only a fallback, + and an encoding Ruby does not recognize is ignored rather than raised. The + selected body is normalized to UTF-8. A record with no URL pattern, no XPath, or a pattern that is not a regular expression is dropped when the file is loaded rather than being allowed to fail @@ -1203,9 +1194,10 @@ next. title: Daily Digest ``` -**The four AI filters.** The plugins that follow each send an item's +**The AI filters.** The plugins that follow each send an item's description to one AI service and put the answer back in its place. They are -four plugins rather than one with a `provider` setting, and that is the design +separate service-specific plugins rather than one plugin with a +`provider` setting, and that is the design rather than an accident: the services differ in endpoint, authentication, request body, answer shape, error format and available models; each of those moves without asking the others; and a Recipe naming `FilterClaude` says on its @@ -1216,11 +1208,10 @@ Changing service is changing that one line. the item's description is the text it applies to, so summarizing, translating, extracting, reformatting and classifying are the same plugin with a different prompt. There is no default prompt: a Recipe without one is refused with an -`ArgumentError` rather than being given a purpose it did not ask for. The two -are sent as separate fields — a system instruction and a user turn — so that +`ArgumentError` rather than being given a purpose it did not ask for. The instruction and the item text are sent as separate fields — a system instruction and a user turn — so that what an article says is text to be worked on, never an instruction to obey. -What the four have in common: +What the AI filters have in common: | Point | What it is | | --- | --- | @@ -1258,7 +1249,7 @@ for new integrations, and authenticates with the token as a bearer token. | `retry` | integer | Attempts after a failure. Default `0`. | | `interval` | integer | Seconds between attempts. Default `0`. | -The endpoint is not a setting: there is one, an operator has no version of this +The endpoint is fixed by this plugin rather than exposed as a setting, an operator has no version of this plugin that talks to a different host, and a setting for it would be a way to send the token somewhere else. The answer is read out of the typed `output` array, from the `output_text` of the assistant's message. @@ -1267,7 +1258,7 @@ array, from the `output_text` of the assistant's message. - module: FilterOpenAI config: token: sk-... - model: gpt-5.6 + model: prompt: | Summarize the following articles as one digest, in Japanese. retry: 2 @@ -1280,7 +1271,8 @@ array, from the `output_text` of the assistant's message. API, `https://api.anthropic.com/v1/messages`, and replaces it with the answer. Anthropic authenticates with an `x-api-key` header rather than a bearer token, requires an API version header, and requires a `max_tokens` — so this plugin -sends all three, and has one setting the others do not. +sends those required values and exposes the required `max_tokens` setting in +addition to the shared settings. | Key | Type | Meaning | | --- | --- | --- | @@ -1300,7 +1292,7 @@ blocks of other kinds are passed over. - module: FilterClaude config: token: sk-ant-... - model: claude-opus-5 + model: prompt: | Summarize the following articles as one digest, in Japanese. max_tokens: 2048 @@ -1336,7 +1328,7 @@ description. - module: FilterGemini config: token: AIza... - model: gemini-3.5-flash + model: prompt: | Summarize the following articles as one digest, in Japanese. retry: 2 @@ -1368,7 +1360,7 @@ Recipe that says where the text goes for a Recipe that does not. - module: FilterSakuraAI config: token: ... - model: gpt-oss-120b + model: prompt: | 以下の記事群について、個別記事の要約を羅列するのではなく、 全体を一つのダイジェストとして日本語で要約してください。 @@ -1403,7 +1395,7 @@ plugins: - module: FilterSakuraAI config: token: ... - model: gpt-oss-120b + model: prompt: | 以下の記事群について、個別記事の要約を羅列するのではなく、 全体を一つのダイジェストとして日本語で要約してください。 diff --git a/doc/POLICY.md b/doc/POLICY.md index 181d47d..7bef9f0 100644 --- a/doc/POLICY.md +++ b/doc/POLICY.md @@ -410,7 +410,7 @@ appear in the order shown below. An executable uses this canonical form: - A `Description::` line may be added below `Name` when the file's purpose is not obvious from its name. New plugins should have one. - The magic encoding comment is redundant on the supported Rubies. It is left in - place in existing files, because removing it from forty files is a diff with + place in existing files, because removing it across existing files is a diff with no benefit, and it is not required in a new file. Files do not carry a per-file version history. This repository versions at the diff --git a/doc/QUICKSTART.md b/doc/QUICKSTART.md index 03dec9a..20b0981 100644 --- a/doc/QUICKSTART.md +++ b/doc/QUICKSTART.md @@ -1,12 +1,15 @@ # Quick Start -This guide takes four public pages through one short Automatic Ruby pipeline and -leaves what they publish as one Markdown document. It needs no account, no -credential, no paid service and no database server. - -It does need two of the optional gems — because of the plugins the Recipe -names, not because of the framework — and installing exactly those is a step of -this guide rather than a footnote to it. Automatic Ruby installs what the +This guide takes the public pages listed in its Recipe through one short +Automatic Ruby pipeline and leaves what they publish as one Markdown document. +It needs no account, no credential, no paid service and no database server. + +It does need optional gems because of the plugins the Recipe names, not because +of the framework. The exact requirements are derived in +[Install what the Recipe needs](#4-install-what-the-recipe-needs) from the +canonical optional-dependency table in +[`DEPLOYMENT.md`](DEPLOYMENT.md#optional-plugin-dependencies), rather than +being maintained as a separate count here. Automatic Ruby installs what the framework needs and leaves a plugin's gems to the operator who uses that plugin, so "which plugins does this Recipe name, and what do they need" is a question every Recipe asks. Skipping it is the usual way a first run stops half @@ -38,7 +41,7 @@ overwritten. ## 3. Write the Recipe A Recipe is one job: the plugins it runs, in order, with their settings. This -one reads four public index pages as HTML and makes a feed of the articles each +one reads the public index pages listed under `sites` as HTML and makes a feed of the articles each lists — which is what to do for a page whose feed you do not have — keeps a record of what it has already seen, and appends the rest to a Markdown document. @@ -101,7 +104,7 @@ plugins: mode: append ``` -Three plugins, and each hands its result to the next: +Each plugin hands its result to the next: - **`CustomFeedWeb`** fetches each page and makes a feed of the article links it lists. `include` is what tells an article from a navigation link, `interval` @@ -117,7 +120,8 @@ Three plugins, and each hands its result to the next: Read the Recipe you have just written, plugin by plugin, and look each one up in the table of optional plugin dependencies in [`DEPLOYMENT.md`](DEPLOYMENT.md). -That table gives, for these three: +For the plugins in this Recipe, that table resolves the requirements as +follows: - **`CustomFeedWeb`** — `nokogiri`, which it reads the pages with. In a checkout, the group `html`. @@ -221,10 +225,10 @@ bundle exec bin/automatic scaffold bundle exec bin/automatic -c ~/.automatic/config/web2markdown.yml ``` -Step 4 is the step that differs, because a checkout resolves its gems through -Bundler rather than through RubyGems. Each optional gem is in a Bundler group, -and the Recipe's groups — `html` and `store`, from step 4 — are selected -together and installed once: +The "Install what the Recipe needs" step is the step that differs, because a +checkout resolves its gems through Bundler rather than through RubyGems. Each +optional gem is in a Bundler group, and the Recipe's groups — `html` and +`store`, from that step — are selected together and installed once: ```sh bundle config set --local with "html store" @@ -233,7 +237,7 @@ bundle install `gem install nokogiri` does **not** work here: the gem installs, and the checkout still reports it as missing, because it is not in the bundle. That -difference, the commands that show what the bundle holds, and the same three +difference, the commands that show what the bundle holds, and the same Recipe plugins taken step by step through choosing their groups are in [`DEPLOYMENT.md`](DEPLOYMENT.md) under "Working out what a Recipe needs, in a checkout". diff --git a/doc/RELEASING.md b/doc/RELEASING.md index 87b7fc1..da628f2 100644 --- a/doc/RELEASING.md +++ b/doc/RELEASING.md @@ -33,8 +33,9 @@ gem --version bundle --version ``` -The release Ruby need not reproduce the complete CI matrix locally. Required -GitHub Actions checks must be green on Ruby 3.3, 3.4 and 4.0 before publication. +The release Ruby need not reproduce the complete CI matrix locally. Required GitHub Actions checks must be green for every Ruby version in the +current matrix of [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) +before publication. ## 2. Authentication and credentials @@ -85,8 +86,8 @@ X.YY release in year X, month YY X.YY.PATCH correction to an earlier release in the same month ``` -Section 10 of `POLICY.md` is authoritative. A release updates these three -places together in one release-metadata commit: +Section 10 of `POLICY.md` is authoritative. A release updates these release-metadata sources together in one +release-metadata commit: - `VERSION` contains the package version; - `lib/automatic/version.rb` defines `Automatic::VERSION` for the CLI; @@ -207,7 +208,7 @@ Confirm that: - the name is `automatic` and the version equals `VERSION`; - the summary and homepage describe this project; - `source_code_uri` is the Automatic Ruby repository; -- the required Ruby version is `>= 3.3.0`; +- the required Ruby version matches `automatic.gemspec`; - licenses contain both `GPL-3.0-only` and `LGPL-3.0-only`; - the authors are correct; - runtime and development dependencies match `automatic.gemspec`, and the diff --git a/doc/REQUIREMENTS.md b/doc/REQUIREMENTS.md index c765c0e..778ed98 100644 --- a/doc/REQUIREMENTS.md +++ b/doc/REQUIREMENTS.md @@ -491,8 +491,9 @@ Two statements are made here, and they are deliberately different. dependency the project needs moves it, or when the version drops out of the distributions the project is used on. -**The continuously validated versions** are **3.3, 3.4 and 4.0** — the ends of -the range and the release in the middle. +**The continuously validated versions** are the versions in the matrix of +[`.github/workflows/ci.yml`](../.github/workflows/ci.yml). That workflow is +authoritative for the set checked on every commit. - CI runs representative versions rather than every intermediate release. The cost of a matrix entry is paid on every commit, and a third entry between two @@ -503,9 +504,10 @@ the range and the release in the middle. - Adding a released Ruby to the matrix is how support for it becomes continuous, and is a small change. -One statement of the supported range lives in the gemspec, one statement of the -validated set lives in the CI matrix, and the README and the documents agree -with both. +`automatic.gemspec` is authoritative for the supported Ruby requirement, and +`.github/workflows/ci.yml` is authoritative for the continuously validated +set. User-facing documents state the supported range where readers need it +and refer to the matrix rather than duplicating its current membership. ## 21. Portability @@ -534,9 +536,10 @@ with both. ## 23. Simplicity -The framework is under seven hundred lines of Ruby and is meant to stay that -size. It is the small fixed part that plugins are written against, and it earns -its keep by not changing. +The framework is meant to remain the small fixed part that plugins are written +against. Its responsibility boundary, rather than a duplicated current line +count, is the maintained constraint; see +[`BASIC_DESIGN.md`](BASIC_DESIGN.md). - A capability that can live in a plugin lives in a plugin. - A framework feature that only one plugin would use does not belong to the diff --git a/doc/VERSIONS b/doc/VERSIONS index aeba796..6f49b9a 100644 --- a/doc/VERSIONS +++ b/doc/VERSIONS @@ -4,9 +4,9 @@ automaticruby Repository Version History v26.09 (Release Date: TBD) -------------------------- - Refresh the bundled LDRFullFeed siteinfo database from the newer upstream snapshot. -- Add FilterLimit, FilterBatch, and FilterPresent for pipeline-wide limiting, fixed-size batching, and required-field filtering. +- Add FilterLimit, FilterBatch and FilterPresent for limiting, batching and required-field filtering. - Add per-fetch interval handling to FilterFullFeed, FilterImageSource, and FilterDescriptionLink. -- Make the Section 6 plugin catalogue the single source of truth instead of duplicating current plugin counts and lists. +- Make doc/PLUGINS.md section 6 the single source of truth for the current plugin catalogue. v26.08 (2026-08-22) ------------------- diff --git a/plugins/filter/full_feed.rb b/plugins/filter/full_feed.rb index aeec80c..0d0d4b1 100644 --- a/plugins/filter/full_feed.rb +++ b/plugins/filter/full_feed.rb @@ -17,10 +17,8 @@ class FilterFullFeed SITEINFO_TYPES = %w[SBM INDIVIDUAL IND SUBGENERAL SUB GENERAL GEN].freeze # One siteinfo record, reduced to the four things a match needs and with - # its URL pattern compiled once. The database ships with 3,504 usable - # records, so compiling them per item -- which is what matching against - # the raw JSON did -- was several thousand `Regexp.new` calls for every - # link in every feed. + # its URL pattern compiled once. Compiling patterns when the database is + # loaded avoids rebuilding regular expressions for every item. Entry = Struct.new(:url, :pattern, :xpath, :encoding) def initialize(config, pipeline = []) @@ -107,13 +105,10 @@ def match(link) @siteinfo.find { |record| links.any? { |candidate| record.pattern.match?(candidate) } } end - # The database was last updated in 2013 and 3,448 of its 3,504 records - # anchor on a scheme, nearly all of them `^http://`. The sites they name - # have since moved to HTTPS, so an https link out of a feed matches none of - # them and the filter silently does nothing. Matching the link under either - # scheme is what keeps those records reachable; a record is about a site's - # layout, not about how it is transported. Only the match is rewritten -- - # the page is fetched from the link the feed gave. + # A stored siteinfo pattern may name HTTP while a current feed supplies HTTPS, + # or the reverse. A record describes a site's layout rather than its current + # transport scheme, so matching tries both schemes. Only the candidate used + # for matching changes; the page is fetched from the original item link. def schemes(link) case link when %r{\Ahttps://} then [link, link.sub(%r{\Ahttps://}, 'http://')] @@ -150,9 +145,9 @@ def body(link, entry) # database is full of sites that declare their charset only in a meta tag, # and for those the difference is the whole article in mojibake. # - # A record's own `enc` is the last resort, for a page that declares nothing - # anywhere: it was recorded in 2013, and trusting it ahead of what the page - # says would break every site that has changed encoding since. + # A record's own `enc` is the last resort for a page that declares nothing. + # Page declarations take precedence because a site's encoding can change after + # a siteinfo record is written. def document(link, entry) page, declared = fetch_page(link) parsed = Nokogiri::HTML.parse(StringIO.new(page))