Email Benchmark Comparability Worksheet

Capture the metric definition

Write the exact event being measured and the unit of analysis: message, delivered message, recipient, account, session, or attributed outcome. Note whether the value is unique, total, gross, or net and whether automated activity is filtered.

Capture the numerator

Record the counted event, deduplication rule, qualification rule, and observation deadline. For replies or conversions, state whether every event counts or only a reviewed class such as positive replies or qualified outcomes.

Capture the denominator

Record whether the base is attempted, accepted, delivered, opened, clicked, or eligible recipients. For bounce measures, state how temporary 4.X.X and permanent 5.X.X SMTP outcomes are classified rather than treating every delivery status as equivalent.

Source: SMTP Enhanced Status Codes Registry

Record audience and list context

Capture relationship, acquisition or consent path, segment, lifecycle stage, organization type, list age, and relevant sender maturity. A subscriber campaign, customer message, and unrequested prospecting campaign should not share a benchmark merely because each uses email.

Separate message type

Classify the message as prospecting, subscribed marketing, lifecycle, transactional, or relationship mail and preserve the source's own categories. The FTC says US CAN-SPAM covers commercial messages, including B2B, while it treats transactional or relationship categories narrowly; legal scope is not interchangeable with metric comparability.

Source: CAN-SPAM Act: A Compliance Guide for Business

Record mailbox-provider scope

Capture the destination-provider mix and whether the source covers all mailboxes or a provider-specific dataset. Gmail Postmaster data concerns mail to personal Gmail accounts, can omit low-volume data for privacy, and uses provider-defined dashboard calculations; Yahoo publishes its own sender practices. Do not relabel either source as a universal email population.

Sources: Email sender guidelines, Postmaster Tools dashboards, Sender best practices

Record geography and recipient type

Capture sender and recipient geographies, jurisdiction assumptions, and whether recipients are consumers, corporate subscribers, sole traders, or another class relevant to the comparison. The ICO distinguishes UK corporate subscribers from sole traders and says its B2B marketing guidance is under review after legal changes, so record the access date and obtain current legal review where needed.

Source: Business-to-business marketing

Record the window and date

Write the send-date range, event observation period, reporting time zone, lag, season, and publication or extraction date. Keep a point-in-time result separate from a rolling average and from a benchmark aggregated across undisclosed periods.

Source: Postmaster Tools dashboards

Test population compatibility

List each population difference and explain why it would or would not change the intended decision. If the source does not disclose a decision-critical population field, mark it unknown; a broad category label is not evidence of a match.

Treat opens as instrument-dependent

Record the tracking method, client mix, unique-event rule, bot or proxy treatment, and privacy features. Apple says Mail Privacy Protection prevents senders from seeing whether a protected recipient opened a message and hides the recipient's IP address, so open observations collected under different client and privacy mixes may not be comparable.

Source: Use Mail Privacy Protection on iPhone

Define click and conversion attribution

Record redirect or analytics tooling, bot filtering, attribution model, lookback window, and UTM taxonomy. Google Analytics documents campaign parameters for identifying referral traffic and warns that missing relevant parameters can create unset reporting values.

Source: Collect campaign data with custom URLs

Keep personal data out of campaign labels

Document privacy filtering and redaction before comparing attributed results. Google says Analytics data must not contain information it could recognize as personally identifiable and specifically warns against putting PII in custom campaign parameters.

Source: Best practices to avoid sending personally identifiable information

Record provider-dashboard limits

Note coverage, authentication prerequisites, latency, time zone, privacy suppression, and calculation scope. Gmail Postmaster says its data is not real time, some dashboards cover DKIM-authenticated mail, low volume can suppress data, and some dashboard datasets differ slightly.

Source: Postmaster Tools dashboards

Separate observed from inferred events

Label direct events, modeled or attributed events, provider feedback, and analyst classifications separately. Do not combine them into one rate unless the external source uses the same rules and the combination supports the named decision.

Test measurement compatibility

Explain whether instrumentation differences are immaterial, can be normalized, or invalidate the comparison. Keep any adjustment formula and the unadjusted values so another reviewer can reproduce the decision.

Record provenance

Save the original URL, publisher, named author or owner, publication and update dates, access date, and whether the source is a primary standard, provider or regulator document, first-party dataset, vendor analysis, or unsourced roundup.

Inspect the sample and exclusions

Look for sample construction, unit of analysis, inclusion and exclusion rules, deduplication, weighting, missing-data handling, and aggregation. Record each undisclosed field as unknown rather than estimating it.

Check reproducibility and conflicts

Ask whether another reviewer could calculate the result from the disclosed method. When sources conflict, preserve both definitions and investigate the scope difference; do not average unlike numbers to create false agreement.

Use

Use the benchmark only for the named decision when every decision-critical field is present and matched, or a difference is explicitly judged immaterial. Attach the completed record and describe the value as source- and scope-specific.

Reject

Reject the comparison when a formula, population, window, instrumentation, or source-quality mismatch could change the action, or when a critical field is unknown. Keep the source in the record so the same unsuitable comparison is not repeated.

Build your own baseline

When external evidence does not pass, freeze your metric definition, segment by the fields that matter, preserve campaign and measurement changes, and collect a dated first-party series. Report uncertainty and revisit segmentation when the operating context changes.