SEO reports and tracking
SWPanelBot-SEO
Crawls the website as a search engine would for the SEO report and tracking: titles, descriptions, headings, links, response codes, canonical, hreflang and structured data.
- When it visits
- Scheduled, on websites with the SEO module active, and every time the owner requests a report. A user can also manually request the report of a website that isn't in SWPanel.
- Frequency
- Every 30 days on the Basic plan and every 7 days on Professional and Premium, plus reports requested manually.
- What it requests
-
GET of the home page, of
/robots.txt, of the sitemap (the one robots.txt declares or/sitemap.xml) and of the HTML pages on the same domain that it finds by following links. - Volume and pace
- Up to 10, 100 or 200 pages per pass depending on the plan, at most 5 clicks from the home page. One request at a time, never in parallel, and a maximum of 15 minutes per website. It slows down if the website answers 429 or 503. It doesn't re-download pages that haven't changed.
- robots.txt
-
It requests it under its name and stores it for 24 hours at most. It first looks for the
SWPanelBot-SEOgroup and, if there isn't one, theSWPanelBotgroup, and if not, the*group, as RFC 9309 dictates. If robots.txt returns 404, it understands it can crawl everything; if it returns a 5xx error or doesn't answer, it doesn't crawl. - What it stores
- Titles, descriptions, headings and metrics of each page, and the list of problems, in the owner's SWPanel.
The Premium report also asks PageSpeed Insights to measure the home page: that visit arrives under Google's name, not ours.