How to Run an SEO Experiment You Can Actually Learn From

Your team rewrote twenty category-page titles. A month later, organic clicks rose. Should you apply the same pattern to every category?

That depends on what else happened. Search demand may have increased, the site may have recovered from an outage, or the selected pages may simply have rebounded after an unusually weak period. A before-and-after chart cannot distinguish those explanations from the effect of your titles.

A useful SEO experiment starts with a decision, records the conditions under which the comparison is credible, and ends with an honest statement of uncertainty. Claude or ChatGPT can help organize that work. RankParse can capture public-page evidence before and after deployment. You supply Search Console, analytics, crawl exports, and the deployment history needed to connect a page change to an outcome.

This guide uses a title-change example throughout. All page counts and performance figures are illustrative, not measured RankParse or customer results.

Write the decision before choosing pages

Start with a specific proposal: “Should we add a meaningful product attribute to the titles of eligible category pages?” That is easier to evaluate than “Does better SEO increase traffic?”

Record the proposed mechanism. A more descriptive title might help searchers recognize a relevant category and click. It might also affect query relevance or Google’s title-link selection. Google can generate a different title link, so editing the HTML title does not establish that searchers saw your wording. See Google’s title-link documentation when defining this uncertainty.

Choose one primary outcome that answers the business question. Total Google organic clicks to the eligible page set might suit a discovery experiment. Qualified organic conversions might suit a commercial decision, provided volume and measurement are sufficient. Use impressions, CTR, query composition, and conversions as supporting diagnostics or guardrails.

Do not make CTR the sole verdict when query mix or position changes. CTR can rise because low-CTR impressions disappear. Define the country, device coverage, search type, URL scope, reporting timezone, and conversion definition before collecting results. Keep those settings identical across the comparison.

Then state what result would justify action. An estimated gain smaller than implementation and maintenance costs may not warrant expansion. Write down the smallest benefit worth acting on and the largest acceptable downside. These are business thresholds, not guarantees that the data will distinguish effects that small.

Decide which kind of evidence you can produce

Two designs can be useful, but their conclusions differ.

Matching does not turn a chosen rollout into a randomized experiment. A team may choose pages with promising merchandising plans, better stock availability, or unusually low recent traffic. Those differences can influence outcomes even when historical click totals look similar.

For randomization, first identify the unit that can reasonably receive treatment independently. It may be a category page, or it may need to be a whole group of overlapping categories. Randomly assigning individual URLs is misleading if their traffic routinely shifts between them.

Use supplied query-to-page and crawl exports to identify overlap. If two pages compete for the same queries, a title change could move clicks from one to the other without growing the site’s traffic. Group strongly connected competing pages when feasible, and measure the combined query or page family as a guardrail. Larger groups reduce the number of independent units available for analysis.

If the available site structure cannot support a credible randomized comparison, run a cautious observational pilot. That still helps decide what to investigate next. Describe it accurately instead of attaching a causal label to a weak design.

Assemble a bounded evidence packet

Create a folder for one experiment, with a stable identifier such as category-title-attribute-01. Include:

• An eligible URL list, category families, treatment eligibility rules, and exclusions decided before assignment.
• Daily Search Console exports spanning the intended baseline, with clicks and impressions under consistent filters. Keep query-level diagnostics separate from page totals because omitted or anonymized queries can prevent reconciliation.
• Analytics exports for the agreed guardrails, with tracking changes and attribution definitions documented.
• A crawl export showing relevant internal-link destinations and competing category relationships.
• A release calendar covering promotions, stock changes, migrations, template releases, and other SEO work.

Preserve the original files. Record extraction time, included dates, filters, and known limitations. Search Console’s latest data can be incomplete; use the same reporting maturity rule for both groups. For help preparing inputs, see finding content opportunities in Search Console.

Connect RankParse using the MCP setup guide. The secure OAuth endpoint at https://connect.rankparse.com/mcp provides read-only tools. In this workflow it does not retrieve your Search Console account data or edit your website. Those inputs and deployments remain your responsibility.

Give the assistant the packet and a bounded first task:

Review category-title-attribute-01 using only the supplied files.
Identify eligible page families, overlapping query groups, missing
measurement fields, and concurrent changes that threaten comparison.
Propose a randomized design if independent units are available;
otherwise label the design observational. Do not assign treatment yet.
Return the hypothesis, primary metric, guardrails, exclusion rules,
and unresolved assumptions. Do not invent missing performance data.

Review the proposal yourself. An assistant can identify suspicious overlap; it cannot infer every business event from a traffic series.

Use the baseline to test feasibility

A baseline serves several purposes: measuring variability, comparing historical behavior, identifying measurement breaks, and deciding whether the experiment could answer the question at a useful cost.

There is no universal minimum of pages or weeks that makes an SEO experiment valid. Detectability depends on traffic, variability, effect size, correlation between units, and the assignment structure. Thousands of daily observations from a few related pages do not become thousands of independent experimental units.

For a material rollout decision, ask an analyst to estimate detectable effects using the proposed design and historical data. Simulations should preserve relevant time dependence and page-family structure. If the plausible effect is far smaller than the noise, change the design, expand genuinely eligible units, or accept that the pilot will mainly validate implementation.

For an observational comparison, inspect several baseline windows for similar trends, weekday patterns, and sensitivity to promotions. Similar historical trends make a comparison more plausible, but do not prove that future trends would have remained parallel without treatment.

Avoid selecting only pages that suffered the largest recent drop. Extreme measurements often move back toward more typical values: regression to the mean. A rebound can therefore occur without an effective intervention. Define eligibility using stable business and page characteristics, and retain the selection rule in the record.

Seasonality requires more than choosing equal-length windows. A winter clothing category and a garden furniture category may share a template but respond differently to weather. Match or stratify by plausible demand drivers before looking at the outcome. Prior-year data can provide context, with its own caveats about inventory, competition, and site changes.

Capture the public-page baseline with RankParse

Before deployment, call get_page_seo with the url parameter for the agreed URLs. Save each raw response alongside the requested URL and a capture timestamp that your workflow records. See the Page SEO response documentation for the exact fields.

For a title experiment, extract data.title.text, data.final_url, data.status_code, data.canonical.url, and data.robots.raw. Include data.headings.h1 if an unexpected template change would matter. A stored baseline lets you distinguish the intended title edit from an accidental canonical or heading change.

Do not treat data.title.optimal as a predicted performance score. It reflects a documented length check. Likewise, data.robots.index describes robots meta directives, not proof that Google indexed the page. The canonical field reports the declared canonical, not Google’s selected canonical.

RankParse parses server-returned HTML. If the title is changed only by client-side JavaScript, inspect the rendered page separately. Its internal and external link fields are counts, not destination lists; use your crawl export when investigating linking interference. The final URL does not provide a complete redirect chain.

Use a second prompt to produce an auditable baseline:

For the URLs in approved-urls.csv, use get_page_seo with url.
Preserve each raw response and record capture time in the output packet.
Build a baseline table of title.text, final_url, status_code,
canonical.url, robots.raw, and headings.h1 from data.
Flag failed fetches, unexpected destinations, and non-200 target status.
Do not replace missing values with assumptions or expand the URL list.
Return evidence references for each row. Do not edit the website.

A prompt limits the requested scope, but it is not a hard execution cap. Use client controls when strict limits matter. Check that response files actually exist in your environment; an assistant’s statement that it “saved the baseline” is not sufficient evidence.

Freeze assignment, deploy, and verify exposure

For a randomized experiment, generate assignment once using a recorded method and seed, with any predefined strata or matched pairs preserved. Save the result before deployment. Do not reshuffle groups because one allocation looks likely to produce a better story.

For an observational design, freeze the chosen comparison and matching method. Document why each group was selected and what differences remain. Keep a dated version of the hypothesis and analysis plan in both cases.

Deploy the approved titles through your normal CMS or engineering process. Then repeat the scoped RankParse checks for treatment and control URLs. Compare exact title strings and inspect changes to status, canonical, robots meta, and headings. A control page changed by a shared template is contamination, even if nobody edited it directly.

A successful fetch confirms what the server returned at that moment. It does not establish when Google recrawled the page, whether Google used the new title, or when users encountered it. Record deployment time separately from any available crawl or exposure evidence. Server logs and Search Console inspection can help investigate those questions, but must be supplied or checked separately.

Predefine how deployment failures and delays will be handled. In a randomized design, analyzing the originally assigned groups preserves the benefits of randomization; silently removing treatment pages where deployment failed can bias the result. Report implementation compliance separately and get analytical help if you need to estimate effects among successfully treated pages.

If RankParse encounters a timeout or bot-blocking page, mark that verification as missing and inspect the page manually. Do not count a failed fetch as evidence that the planned title shipped.

Analyze the comparison without manufacturing certainty

The following totals are illustrative and use equal-duration, comparable reporting windows:

The changed pages gained 1,800 clicks. Calling all 1,800 incremental would ignore growth in the comparison group.

Under a stated proportional-trend assumption, the comparison suggests an expected untreated total of 10,000 × 1.10 = 11,000. The changed pages exceeded that benchmark by 11,800 − 11,000 = 800 clicks, or approximately 800 / 11,000 = 7.3%. The difference between the raw growth rates is eight percentage points; that is a different quantity from the 7.3% relative excess.

This calculation is a descriptive adjustment, not a confidence interval or proof of causation. Aggregate totals cannot show the uncertainty, unit-level concentration, or validity of the underlying comparison. An additive model would make different assumptions; select the analysis before viewing favorable results, not afterward.

Analyze outcomes at the assignment level, accounting for repeated observations and clustering where applicable. Report uncertainty around the estimate using a method appropriate to the design. Do not calculate narrow intervals by pretending every impression or daily row is independently randomized.

Look for concentration: did one category account for most of the apparent gain? Examine the predeclared cannibalization guardrail. If treatment pages gained 800 clicks while related pages outside the test lost a similar amount, expansion requires a different justification.

Use the final staged prompt after supplying the completed follow-up exports:

Use the frozen analysis plan, assignment file, baseline evidence,
deployment ledger, and follow-up exports. Reproduce the agreed
comparison with explicit formulas and denominators. Separate observed
changes, adjusted estimates, and causal claims the design can support.
Check predefined guardrails, contamination, query shifts, and missing
verification. Do not invent confidence intervals from aggregate totals.
Draft a decision memo with expand, revise, revert, or inconclusive
as options, and cite the evidence supporting the recommendation.

Review the calculations outside the prose summary. The assistant’s most confident sentence should never carry more certainty than the analysis supports.

Keep an immutable ledger and a usable decision

Use an append-only experiment ledger in storage that supports immutable versions or write-once retention. Hashing evidence files can reveal changes, but hashes alone do not prevent someone from replacing files and their hashes. Choose storage controls appropriate to the importance of the decision; RankParse does not automatically provide this experiment ledger.

Each event should identify the experiment, event time, actor, affected URLs, evidence file or version, and reason. Record the initial plan, assignment, raw RankParse responses, deployment, verification failures, analysis inputs, and decision. Correct errors by appending a correction that references the earlier event.

Continue checking safety and commercial guardrails during the run. Avoid repeatedly testing for significance and stopping at the first favorable result unless the analysis plan explicitly supports sequential decisions. Set the analysis date or information target in advance, with rules for incomplete data and serious implementation problems. More time will not repair a contaminated comparison or a broken outcome definition.

Copy this decision record into the experiment folder:

Experiment ID and owner:
Decision and smallest worthwhile benefit:
Hypothesis and proposed mechanism:
Eligible population and assignment unit:
Design: randomized / matched observational
Primary metric, filters, and guardrails:
Assignment or matching method and frozen file:
Baseline dates and feasibility assessment:
Deployment dates and exposure uncertainties:
Raw public-page evidence and immutable ledger location:
Analysis timing, method, and stopping rules:
Observed result, adjusted estimate, and uncertainty:
Contamination, missing data, and sensitivity findings:
Decision: expand / revise / revert / inconclusive
Next action, accountable owner, and review date:

An inconclusive result is useful when it identifies what prevented learning: insufficient information, inconsistent deployment, overlapping pages, or a comparison that stopped being credible. Keep those distinctions in the memo rather than translating all uncertainty into “the change did not work.”

Start with one decision your team can act on, freeze its measurement plan, and capture the existing pages before anyone edits them. That evidence makes the eventual result reviewable, even when the answer is to run a better experiment.