SWPanelBot-SEO

How it requests

What it requests

The home page, /robots.txt, the sitemap declared by robots.txt (or /sitemap.xml) and HTML pages on the same domain found by following links. For the report, additionally, a screenshot of the home page with a windowless browser, which today arrives under the name SWPanelBot-Code.

Methods

GET only.

Redirects

It follows them, to know where each link leads.

Certificate

It doesn't cut the visit short because of a certificate problem.

Cookies

It neither stores nor sends cookies.

JavaScript

It doesn't run it: it reads the HTML as the server serves it. The report's screenshot does use a browser.

Waits

Up to 12 seconds to connect and 20 for the response on each page; on robots.txt, 10 and 15.

Retries

None. If a page fails, it notes it and moves on to the next.

Origin

In preparation From SWPanel's servers. We're reserving some outgoing IPs just for the bots; once they exist, we'll publish them here in a JSON file.

Reverse DNS

In preparation Each IP will have a name under bots.swpanel.com that resolves back to the same IP.

Standards it applies

Which websites it visits

Compliant It only visits websites of active SWPanel services and stops as soon as the service is cancelled: it checks before every request. The only exception is the report a user requests manually, which may be for a website that isn't in SWPanel. Even then it respects robots.txt.

robots.txt

Compliant It requests it under its name and stores it for 24 hours at most. It first looks for the SWPanelBot-SEO group and, if there isn't one, the SWPanelBot group, and if not, the * group, as RFC 9309 dictates. If robots.txt returns 404, it understands it can crawl everything; if it returns a 5xx error or doesn't answer, it doesn't crawl.

Crawl-delay

Compliant It respects the Crawl-delay of the SWPanelBot-SEO group or, if there isn't one, of the SWPanelBot group: it waits as long as requested between two pages of the same website. If it asks for more than 60 seconds, it looks at one page and leaves the website until the next pass; if the waits in one pass add up to more than 2 minutes, it does too.

If the website asks to slow down

Compliant On 429 or 503 it waits as long as Retry-After says if it's 60 seconds or less (10 if it doesn't say). If it asks for more, it leaves the website until the next pass. After 3 consecutive 5xx errors, it also leaves it.

HTTP cache

Compliant It requests with If-None-Match and If-Modified-Since. If the page hasn't changed (304), it doesn't download it again and uses the copy it already has.

Signature

Compliant It signs every request with Web Bot Auth (RFC 9421, Ed25519): Signature, Signature-Input and Signature-Agent headers. The public key is at https://www.swpanel.com/.well-known/http-message-signatures-directory.

Contact in the request

Compliant It carries From: [email protected], as RFC 9110 asks of automated clients.

What it does with the data

What it reads

Titles, descriptions, headings, links, response codes, canonical, hreflang, structured data, word count and weight of each page.

What it stores

The metrics of each page and the list of problems, in the owner's SWPanel.

How long

Problems, about 30 days. Metrics have no deletion date: they serve to see the evolution.

What for

Only for the SEO report and tracking of the service owner. Some reports use AI to write the conclusions from this data. It is not used to train AI models, nor sold or shared.

Privacy

SWPanel's general privacy policy.

The report also requests data from third-party services, which visit the website under their own name: Google's PageSpeed Insights measures the home page on the Premium plan and the W3C validator checks the HTML. Those visits are not SWPanelBot.