RankWiser
RankWiserGet $100 free

Setup guide

Domain DR snapshot: a monthly setup for your BI tool

A step-by-step setup that turns one API call into a trusted time series, from the first key to the chart.

Claymation workbench with hourglass, key, and succulent showing monthly setup

Setup, in order

  1. 1

    Create the key

    Sign in with Google at /auth/google.php. Your account opens with $100, which is 1,000,000 credits. Copy the key that starts with sk_.

  2. 2

    Make the table

    Columns: domain, dr, ref_domains, pulled_on, source. Primary key on domain plus pulled_on, so a rerun cannot double a month.

  3. 3

    Schedule one call

    Run it on the 5th of each month. Our index refreshes about monthly, so more often adds rows, not information.

  4. 4

    Parse and insert

    Read the JSON, check ok is true, and insert each row with the pull date.

  5. 5

    Flag changes

    Compute month-over-month deltas and mark anything above 3 points. Smaller moves are mostly noise.

  6. 6

    Chart it

    One line per domain, your own domain in a heavy colour, a rival band in grey.

The one request

Cron entry, run on the 5th at 06:00. The cost per run is a single credit.

bash
0 6 5 * * curl -s -X POST "https://rankwiser.app/api.php?action=bulk_dr&key=$RW_KEY" \
  -d "domains=$(paste -sd, /etc/rw/domains.txt)" > /var/rw/dr_$(date +%F).json

Choosing what a domain dr number is allowed to answer

Domain DR is a count in disguise. Plain. It rises when more distinct sites link to a domain, and the log scale makes each new point harder than the last. Keep that sentence in the chart notes.

A good tile answers one trend question per domain set: are we closing on the three rivals, are partners growing, did a campaign move anything. A bad tile mixes in traffic or revenue and invites a causal story the data cannot carry.

Store the raw referring-domain count as well. If the score changes by a point, you can see whether that was 40 new domains or 4,000, and that tells you whether to worry.

Clay shelf with minimal pottery and stone illustrating essential data columns

What the table should and should not hold

A snapshot table is a memory. It should hold what you need to explain a chart in a year, and nothing that invites a wrong explanation. Start with the smallest set of columns that answers the trend question. Add one only when a meeting asks for it.

Some columns earn their place on day one. Fewer than you think. The domain, the score, the referring-domain count, the pull date and the source name cover most questions. A traffic column looks helpful, but it comes from another source and another date, so it belongs in another table.

Leave out anything you can recompute from the rest. A month-over-month change is a subtraction, so keep it in a view rather than as a fourth copy of the same fact.

Write the column list into the chart notes too. A reader who opens the chart in a year should see where each number came from, and which pull filled it.

Keys that stop double rows

The primary key on domain and pull date does more than block duplicates. It also tells you which pull a row came from, which matters when a rerun happens after a timeout. Two rows for the same domain in one month should be impossible, and the schema should say so.

Source names you can filter on

Write the source as a fixed label, such as rankwiser_bulk_dr, not as a free-text note. A filter on that label keeps this score apart from any other score you load later. Keep the label list short and write it down next to the table.

Run costs, worked at three list sizes

Calculated from the credit prices above. Examples, not measurements.
Tracked domainsCalls per monthCreditsDollars
5011$0.0001
50011$0.0001
5,00033$0.0003
When the run fails

A 401 means a missing or wrong key. A 402 means the credit has run out, so top up with USDT from the account page. Alert on any response where ok is not true, rather than writing zeros into the chart.

Test the pipeline on three domains first

  1. 1

    Pick three known domains

    Use one site you know well, one rival and one small site. Three is enough to show the shape of the response.

  2. 2

    Send the test request

    Call bulk_dr with the three domains and save the raw response, not only the parsed rows.

  3. 3

    Check ok and the balance

    Confirm that ok is true and read _credits. The call should cost one credit, and the remaining balance should drop by that amount.

  4. 4

    Compare one row by hand

    Look up one of the domains with the free lookup on this site. If the score matches what your parser stored, the parser is sound.

Guardrails worth adding on day one

Keep a second table of pulls that failed. A row with the HTTP status, the time and the response body beats a silent gap in the chart. Alert on anything but ok true.

Pin the domain list in version control. Short diffs help. When someone adds a rival, the diff shows who added it and when, and the chart gets a note. Otherwise the line for that rival appears from nowhere and people ask why it started in March.

Plan for the change of source, too. If you later add Ahrefs or Moz numbers, give them their own source value in the table, as discussed when we compared Ahrefs and Majestic.

Clay balance scale with spheres illustrating health check for monthly runs

A quick health check for every monthly run

Before you trust a chart, check the job that filled it. Then look at the chart. A healthy run returns ok set to true, one row for each domain you asked for, and a balance that dropped by the expected amount. Anything else is a reason to read the response, not to draw the line anyway.

Count the rows that moved. If a large share shifted by three points or more, check the domain list before you blame the index. A sudden flood of moves usually means the list changed, not the web.

A row count that matches the list is the minimum. Fewer rows means a domain was dropped somewhere between the file and the table. Find it before the next run.

What a flat month tells you

A month with almost no moves is not proof of stability. The index refreshes roughly once a month, so a flat row may simply not have been updated yet. Read a quiet month next to its pull date, and remember that the chart still follows the calendar.

Who gets the alert

An alert that goes to a shared inbox gets ignored within a month. Name one person for the job and a second for the weeks that person is away. The alert should name the domain list, the HTTP status and the time, so the reader does not have to open the script to act.

Testing a change before it goes live

If you change the list or the schedule, run the old and new versions side by side for one cycle. Matching rows show the change did nothing else. A difference you cannot explain should stop the rollout, however small it looks.

A note on rate and rhythm

Nothing here needs speed. A monthly run is a handful of requests, even for large lists, and each response arrives as one JSON document. If your scheduler retries on failure, make the retry idempotent, which the primary key on domain and pull date already gives you.

Resist the urge to run the job daily just because it is cheap. Don't. Our index refreshes roughly once a month. Daily pulls return the same rows for weeks, and your chart fills with flat lines that look like stability but only reflect the crawl calendar. Read the Majestic alternative recipes for other pulls worth scheduling, such as anchors and intersections.

Owning the job after launch

A monthly job outlives the person who built it. Six months on, someone new opens the cron entry and wonders why the list lives in a file. Write a short note. Put it at the top of the script with the purpose, the schedule and the credit cost. That note is cheaper than a meeting about it.

Review the domain list each quarter. Remove what no longer matters, and add the rivals a decision actually turns on. Each change goes into version control with a date, so the chart can show the change instead of hiding it.

Budget is part of the review too. One credit a month. The cost stays tiny. Check the balance on the same day as the chart, and write the next top-up date into the same note.

Questions people ask about the monthly job

Four questions come up once the job has been running for a few months. Each answer is a rule you can write into the setup note.

Why not pull daily, since the calls are cheap?

The data changes on a monthly calendar, so a daily pull repeats the same rows for weeks. The cost is small, but the chart fills with flat lines that look like stability. A chart going to a board also needs the board runbook footnote.

What if a rival drops off the list?

Keep its history, mark the domain as removed and stop pulling it. A missing line looks like a data gap, so write the reason beside the chart.

Can I run the job on a different day?

Yes, if you pick one day and keep it. The point is a steady rhythm. Moving the date by a few days each month makes the trend harder to read.

What do I do about a 402 in the middle of the month?

Stop the job, top up from the account page with USDT, and rerun only the failed request. Keep the failed response in your log, so the gap in the chart has a written explanation.