Tracing a metric that moved

Tracing a metric that moved

rca

use when a metric has moved and you need to say why - a drop in conversion, revenue, add-to-cart, CTR, or any figure that broke from its baseline

You can measure a break precisely and you cannot see the store. `query` and `run_report` read
orders and behaviour; `get_changelog` is the only thing on this platform that knows a price moved, a
collection emptied or a theme was published. **RCA is the join between them, on time.**

Work the ladder in order. Skipping a step does not make you faster, it makes you confident and
wrong: without step 2 every change in the window looks equally like a candidate, which is exactly
how a coincidence gets reported as a cause.

## 0. Establish the break

`query` the metric daily over a window long enough to hold a baseline. Name the break date.

If `covered` is false the data does not reach that window. **Stop and say there is no data.** A
`covered: false` window is not a drop to zero, and reporting it as one is the worst failure
available here: it is a confident finding about nothing.

## 1. Localise

One query, every stage, by day:

```jsonc
{ "select": ["sessions","pageViews","productViews","addToCarts","checkoutsStarted","orders","conversionRate"],
  "over": "day", "since": "-30d", "until": "today" }
```

When the store has no Glood app, those measures are unavailable: use the store's own funnel, at hour
grain for a break younger than a day (Glood-owned measures take `-48h` and `over: hour`).

```jsonc
{ "select": ["storefrontSessions","storefrontProductViews","storefrontAddToCartSessions","storefrontCheckoutsStarted","storefrontOrders","storefrontConversionRate"],
  "over": "hour", "since": "-48h", "until": "today" }
```

Take each stage's ratio to the stage above. **The stage whose ratio broke is the stage that broke;
everything downstream inherits it.** Sessions flat, product views flat, add-to-cart down 40%,
checkouts down 40%, orders down 40% is ONE finding at add-to-cart, not three.

## 2. Split

Re-run the broken stage grouped by `device`, then `country`, then `market`, then `channel`, then
`browser`, then `pageType`. One dimension at a time: the query layer refuses some pairs, and two
dimensions in one table are harder to read than two tables.

- **Concentrated in one segment** means delivery: a theme, an app embed, a script, a browser
  regression, a market setting.
- **Spread evenly** means merchandising or demand: pricing, a discount, stock, traffic mix.

This step is what makes step 3 falsifiable. Do not skip it.

## 3. Correlate

`get_changelog` over the break window plus a day either side. A change is a **suspect** only when
both hold:

- its `occurredAt` brackets the break, before it and not after, **and**
- its subject matches the segment from step 2.

A theme publish does not explain a drop confined to one product. A price edit on one product does
not explain a store-wide drop. A market price adjustment explains a drop in that market and nowhere
else. **When the timing lines up and the subject does not, say the coincidence out loud and do not
promote it.**

**And who.** Shopify's admin feed names the staff member or app behind a change to a product or
discount:

```jsonc
{ "select": ["adminChanges"], "by": ["adminAuthor","adminApp","adminAction"],
  "where": [{ "field": "adminSubjectId", "op": "eq", "value": "<product or discount id>" }],
  "since": "-48h", "until": "today" }
```

A dense burst comes back as one `episode` row with a count and a distribution. That row IS the
finding: "3,412 products repriced between 02:00 and 02:20, median -17%". Do not ask for the
individual records to pick a culprit from them.

## 4. Drill in

The funnel names the subject; the change-log says what happened to it. `get_changelog` with
`subjectId` returns that subject's records at variant or source grain.

**Never scan the change-log for outliers.** Funnel first, change-log second, always. Reading the log
first means finding something that changed, which always succeeds, and then looking for a metric to
blame it for.

## 5. Confirm the mechanism

A suspect that survives its own test is a cause. One that does not is a coincidence worth naming and
discarding.

| Suspect | Test |
| --- | --- |
| Price change | That variant's own add-to-carts before and after |
| Theme publish | Drop confined to the segments that theme serves |
| Discount change | `discountCode` and `discountedOrders` over both windows |
| Market change | Funnel `by market`, drop confined to it |
| Stock | `cutSizePercent` and `inStock` against that product's add-to-carts |
| Theme file edit | Probe errors or speed on the pages that template serves, after the edit against before (`probe*` by `pageType`) |
| App embed | Third-party requests from its domain after the change, absent before (`probeVisits` by `requestDomain`) |

## 6. Rule out the phantoms, then disclose the gaps

**Phantoms first, because they invalidate the measurement rather than explain it.** If the window
brackets a `currencyCode` or `ianaTimezone` change, the comparison itself is invalid and THAT is the
finding. Revenue moving by an exchange rate, or day buckets shifting under a timezone change, is not
a drop.

Then disclose, every time, what this data cannot see:

- **Another app being installed** is not observable. No API, no webhook.
- **Theme edits are polled.** Theme file edits and app embeds switched on or off are recorded, but by
  a poll: an edit and its revert inside one poll interval are invisible.
- **Probe timings come from one region every few hours**: read them as trends, not absolute numbers.
- **Collection membership is resync-observed, not webhook-observed.** Shopify fires no webhook when
  a smart collection's membership changes by rule. Its timestamp is the sweep, not the edit, so
  treat membership timing as approximate.
- **A membership change above 100 members** is recorded as a `productsCount` move with no
  per-product records. The count is the finding; the names were not kept.
- **`covered: false` is not zero.**
- **A capped list has a tail.** If the result carries a "more than N records match" note, you have
  not seen everything in that window.
- **Pixel figures are online-store-only**, and the `market` dimension is pixel-observed.
- **On a polled platform every change time is bounded by the poll interval.**

## 7. File one episode

`record_finding` with a fingerprint naming the CONDITION and never the window: `conversion-drop`,
not `conversion-drop-2026-09-21`, so a week-long problem bumps one episode instead of raising seven
alerts.

`because` carries the whole chain: stage, segment, suspect, confirmation. `impactMinor` comes from a
figure you actually read, not from one you estimated.

## What this skill does not do

It does not decide severity policy, which is the soul's bar. It does not alert; `record_finding`
does. It does not write to Shopify, and nothing in the agent lane can.

## Worked example

Conversion 2.4% to 1.5% from Sep 18. A repricing app repriced 3,412 products at 02:00; on one, the
XL variant took a decimal slip.

```
query            funnel by day             -> add-to-cart broke Sep 18; checkout/orders inherited
query            addToCarts by productId   -> broad dip, and Oxford Shirt collapsed 90%
get_changelog    window                    -> EPISODE: 3,412 products repriced, 02:00-02:20
                                           -> theme "Dawn 12.1" published 09:12
get_changelog    subjectId=7781            -> price, variant 91: 2999.00 -> 24990.00 (+733%)
                                           -> listPrice cleared on the same variant
query            addToCarts by variantId   -> variant 91 to zero; siblings rose
record_finding   "price-error:7781"
```

The theme publish bracketed the break and did not match the segment, so it was named as a
coincidence and discarded. The reprice bracketed it AND matched, and the variant-grain test
confirmed the mechanism.