Website Crawler
Website Crawler scores 49/100 on xpay's Agent-Ready Index v2: partial pricing visibility; self-serve checkout is available; a free entry path exists.
Website Crawler pricing
Basic
Free
Silver
$19.99/mo
or $17.99/mo billed annuallyGold
$59.99/mo
or $53.99/mo billed annuallyPlatinum
$89.99/mo
or $80.99/mo billed annuallyHow Website Crawler scores on the agent-ready dimensions
Public pricing visibility
15 / 15
Self-serve checkout
15 / 15
Public API + docs
0 / 20
Usage-based / metered pricing
0 / 20
Free entry path
7 / 7
No sales call required
7 / 7
Card / wallet / x402 payment
5 / 6
Machine-readable bonus (MCP / OpenAPI)
0 / 10
Total
49 / 100
What it actually costs at typical scale
Basic free plan, 10 pages/day
$0/mo
Silver plan, 30k URLs, 2 users
$19.99/mo
Gold plan, 200k URLs, 4 users
$59.99/mo
Can an autonomous agent transact here?
Can create account
Can enter payment method
Can upgrade plan
Can buy more usage
Can cancel online
Payment methods
Human approval recommended
Pricing transparency
full
Purchase motion
self_serve
Demo required
Quote required
Annual commit required
Auto-renewal mentioned
Cancellation policy visible
Refund policy visible
Invoice payment available
Purchase orders supported
MSA required
DPA available
Security review required
Trial
Available
Annual discount
10% off
Positioning & ecosystem
“Cloud crawler for professionals.”
Buyer persona
Six-step check: can an agent actually buy from Website Crawler?
Discover price
Select a plan
Pay per task
Avoid a sales call
API docs without auth
Estimate cost upfront
Pros and cons for AI agents
Observational summary written by xpay from the signals captured on 2026-05-06. Not a review of the product — only of its current pricing posture for agent buyers.- Pricing is publicly visible on an indexable page — agents can read tiers without scraping past auth.
- Every advertised tier is self-serve; no tier requires scheduling a call.
- A free tier exists, lowering the bar for an agent builder to prototype before committing budget.
- No per-unit price was advertised, so an agent has no way to estimate the cost of a single task.
- API documentation is gated or absent; an agent cannot inspect the integration surface without authentication.
- No /.well-known/ai-pricing.json or equivalent machine-readable pricing manifest is published — agents must rely on HTML scraping.
How Website Crawler could lift its score
Add a per-unit price (e.g. $X / 1K calls) to the pricing page so an agent can compute its own cost before committing.
| pricing_visible | true |
| headline_phrasing | Website Crawler Plans and Pricing — Cloud crawler for professionals. Extract data or audit/monitor your sites 24/7. Trusted by 4,000+ pros. |
| tier_count | 1 |
| lowest_paid_entry_usd | 0 |
| free_tier | true |
| free_tier_terms | null |
| per_unit_price | null |
| annual_required | false |
| self_serve_paid_tiers | 0 |
| sales_only_tiers | 1 |
| public_api_docs_url | null |
| api_docs_auth_walled | null |
| ai_pricing_json_present | false |
| agents_txt_present | false |
| anonymous_purchase_path | false |
| per_unit_classification | null |
| usage_headline_present | false |
| custom_tier_present | true |
| agent_friendly | {"ai_pricing_json":false,"agents_txt":false,"llms_txt":false,"sitemap_xml":true,"mcp_server_card":false,"agent_skills_index":false,"x402_supported":false} |
View raw extracted page text →
Claim this scorecard & lift your score.
Get the breakdown of every signal we measured, the one-line fix, and how your peers in Seo stack up.
Compare the agent-ready picks in Seo.
If you're building or running an AI agent that needs to buy from this category, see who's scoring highest right now.
See agent-ready picksWebsite Crawler
49
/ 100 (rubric v2.0)Seo
Not advertised
1
Yes
Free
0 / 1
Auth-walled
No
Not published
2026-05-09
Tier-based subscription
Website Crawler pricing starts at $0/mo for the Basic plan; paid plans range from $19.99/mo to $89.99/mo. Billed monthly or annually with 10% off annual billing.
Basic free plan (Free)
Discovery files and protocols
Side-channel signals — informational, not part of the score. Each protocol is independent; adoption signals the publisher is thinking about agent buyers.ai-pricing.json
agents.txt
llms.txt
sitemap.xml
MCP Server Card
Agent Skills
x402 / MPP
Compare Website Crawler with peers
Highest-scoring companies in the same category that an AI agent might evaluate as an alternative.Ranklytics
85
IndexCheckr
85
LinkBox
80
Boostramp
78
AvesAPI
75
iSpionage
72
Rankz
71
DAXRM
71
Explore the Agent-Ready SaaS Index
25,480+ SaaS scored on agent-buyability — browse by readiness band, category, or jump straight to a pay-per-run tool on xpay.tools.