Manifest V3 Subscriptions

MV3 subscriptions

EWE keeps track of filter lists through subscriptions.. With the introduction of Manifest V3 (MV3), all extensions must define filtering rules in a specific format that's incompatible with eyeo's filter list format.

Filter lists have to be converted to rulesets, then, which are referenced in the extension manifest file. With MV3, both rulesets and subscriptions must be bundled and distributed together within the extension.

On this page, you'll learn about how subscription lists work, how to convert them to rulesets, and how to add custom subscriptions.

Subscription list basics

A subscription list is a JSON file with information that defines the subscription's details and where it should be fetched from.

The following example shows the EasyList subscription list:

[
  {
    "id": "8C13E995-8F06-4927-BEA7-6C845FB7EEBF",
    "type": "ads",
    "languages": [
      "en"
    ],
    "title": "EasyList",
    "homepage": "https://easylist.to/",
    "url": "https://easylist-downloads.adblockplus.org/v3/full/easylist.txt",
    "mv2_url": "https://easylist-downloads.adblockplus.org/easylist.txt"
  },
  ...
]

Subscriptions

A filter list is a text file with filtering rules, referenced by its URL or ID in subscription lists. In Manifest V2, a subscription consisted mostly of filter list in a specific format:

[Adblock Plus 2.0]
! id: 8C13E995-8F06-4927-BEA5-6C885FB7EEBF
...
testpages.adblockplus.org###eh-id
...

DNR rules and rulesets

DNR rules are the filtering rule format used in Manifest V3. They are provided in ruleset files, which are formatted in JSON, as in the following example:

[
  {
    "priority": 1000,
    "condition": {
      "urlFilter": "&werbemittel=",
      "isUrlFilterCaseSensitive": false
    },
    "action": {
      "type": "block"
    },
    "id": 8163
  },
  ...
]

Manifest file changes

To filter network requests, rulesets have to be listed in the extension manifest file, as in the following example:

{
  "name": "eyeo's Web Extension Ad Blocking Toolkit Test Extension",
  "version": "0.0.1",
  "declarative_net_request": {
    "rule_resources": [
      {
        "id": "0798B6A2-94A4-4ADF-89ED-BEC112FC4C7F",
        "enabled": false,
        "path": "rulesets/0798B6A2-94A4-4ADF-89ED-BEC112FC4C7F"
      }
    ]
  },
  ...
}

You may also need to make other changes to the manifest file. For more information, refer to the official Manifest V3 documentation.

Converting filter lists to DNR rulesets

Follow these steps to convert a filter list to a DNR ruleset:

Get at least one filter list.
(Optional:) Merge several filter lists into a single list.
Download filters.
Convert filters to DNR rules.
Add DNR ruleset information to the manifest file.

Command line scripts

Depending on your project's structure, your extension may need a separate, custom workflow. Several command-line scripts are available to help with this process.

These scripts are designed to run one after another, using the previous script's output as the following script's input, while allowing custom steps (like data transformation) between scripts to accommodate various workflows.

In the most common use case, where no specific requirements or changes are needed, the script's calls chain might look as follows:

# download default subscription list to "scriptsOutput/subscriptions_mv3.json"
npm exec subs-init

# merge all given subscription lists into 
# "scriptsOutput/custom-subscriptions.json"
# in this case, there is only one subscription list, so effectively it is just 
# renamed.
npm exec subs-merge

# download subscriptions defined in "scriptsOutput/custom-subscriptions.json" to 
# "scriptsOutput/subscriptions"
npm exec subs-fetch

# convert all subscriptions in "scriptsOutput/subscriptions" to rulesets in 
# "scriptsOutput/rulesets"
npm exec subs-convert

# generate manifest file fragment into "scriptsOutput/rulesets/rulesets.json"
npm exec subs-generate

subs-init

This script downloads default subscription list to given output file location. (The default subscription list is located at "https://easylist-downloads.adblockplus.org/v3/index.json".)

Usage:

npm exec subs-init -- [--output/-o output_file]

Options:

output/o: Output file. Default: "scriptsOutput/subscriptions_mv3.json"

Example:

npm exec subs-init -- -o output/subs.json

subs-merge

This script merges subscription lists. This is useful for workflows in which a main subscription list needs to be merged with a custom list.

If there is only one subscription list, this step can be skipped. If skipped, however, the subs-fetch script won't work with default options, as it by default expects the output file of this script as input file.

Usage:

npm exec subs-merge -- [--input/-i input_file...]
                       [--output/-o output_file]
                       [--space/-s space]

Options:

input/i: Input file(s). Default: "scriptsOutput/subscriptions_mv3.json"
output/o: Output file. Default: "scriptsOutput/custom-subscriptions.json"
space/s: Number of spaces to be used for indentation in the output file. Default: 0

Example:

npm exec subs-merge -- -i /data/main_subs.json /tmp/product_subs.json -o /subs/merged.json -s 2

subs-fetch

This snippet downloads filter lists from the subscriptions defined in input files.

Usage:

npm exec subs-fetch -- [--input/-i input_file...]
                       [--output/-o output_dir]
                       [--ignoreFetchErrors/-ife]

Options:

input/i: Input file(s). Default: "scriptsOutput/custom-subscriptions.json"
output/o: Output directory. Default: "scriptsOutput/subscriptions"
ignoreFetchErrors/ife: Whether to ignore fetching errors. This is useful when download URL might be unreliable. Default: false

Example:

npm exec subs-fetch -- -i subs/merged.json -o data/fetched -ife

subs-convert

This snippet converts filter lists to rulesets. Conversion only applies to network request rules, that is, URL filters. Other filters will continue to be provided with the filter list files after this step.

Usage:

npm exec subs-convert -- [--input/-i input_dir]
                         [--output/-o output_dir]
                         [--report/-r]
                         [--report-output/-O report_dir]

Options:

input/i: Input directory. Default: "scriptsOutput/subscriptions"
output/o: Output directory. Default: "scriptsOutput/rulesets"
report/r: Boolean flag. If this flag is passed the report is generated. You can also use --no-report to force the report to not be generated. Default: false
report-output/O: Report directory. Default: "scriptsOutput/report"

Example:

npm exec subs-convert -- -i data/fetched -o data/rulesets -r -O data/report

subs-generate

This snippet generates web extension manifest fragment. Contents of this file must be added to the manifest file. See Manifest file changes for details.

Usage:

npm exec subs-generate -- [--input/-i input_dir]
                          [--output/-o output_file]
                          [--prefix/-p prefix_text]

Options:

input/i: Input directory. Default: "scriptsOutput/rulesets"
output/o: Output file. Default: "scriptsOutput/rulesets/rulesets.json"
prefix/p: Prefix text to be used in the fragment. (This is mainly used for specifying the path.) Default: ""

Example:

npm exec subs-generate -- -i data/rulesets -o data/rulesets.json -p path/to/rulesets/

For convenience and testing purposes, running npm run build will automatically run the previous scripts to create a meaningful subset of data used in our tests.

Custom subscriptions list

The default provided subscriptions can be overridden by adding a file called custom-subscriptions.json in the scriptsOutput directory and then using the following command: npm run subs-merge -- [-i ...] -o $(pwd)/scriptsOutput/custom-subscriptions.json to merge the files.

Note the following:

use -i argument to add a file to the list of input files if necessary
one can use $(pwd) on nix-based environments or ../../custom-subscriptions.json for convenience
By default these scripts output ruleset files into a folder called scriptsOutput/rulesets and the full subscription files into scriptsOutput/subscriptions.

Examples:

merge product-specific subscriptions file with default subscriptions:

npm run subs-merge -- \
  -i /tmp/product-subscriptions.json \
  -o $(pwd)/scriptsOutput/custom-subscriptions.json

merge two subscriptions files:

npm run subs-merge -- \
  -i /tmp/product-subscriptions.json \
  -i /tmp/language_en-subscriptions.json \
  -o $(pwd)/scriptsOutput/custom-subscriptions.json

The scripts are also available via npm symlinks:

npm exec subs-init -- -t mv3
npm exec subs-merge [-- -i ... -o ...]
npm exec subs-fetch [-- -i ... -o ...]
npm exec subs-convert [-- -i ... -o ...]
npm exec subs-generate [-- -i ... -o ...]

Adding subscriptions

Before subscriptions are added, availability of subscriptions (that are added in manifest file) through the Solution has to be checked:

EWE.subscriptions.getRecommendations();
/* Returns an array of subscriptions
[
  {
    "id": "00000000-0000-0000-0000-000000000000",
    "title": "Test MV3 Custom Subscription",
    "url": "http://myhost.com/subscription.txt",
    ...
  }
  ...
]
*/

Then, subscriptions can be added using their URLs:

EWE.subscriptions.add('http://myhost.com/subscription.txt');

To check which subscriptions are added and available:

await EWE.subscriptions.getDownloadable();
/*
[
  {
    "id": "00000000-0000-0000-0000-000000000000",
    "enabled": true,
    "title": "Test MV3 Custom Subscription",
    "url": "http://myhost.com/subscription.txt",
    ...
  }
]
*/

Last updated 1 year ago