Link Collector

How It Works

• Crawls agency websites discovered via SERP to find property listing URLs.
• Parses sitemaps (sitemap.xml, sitemap_index.xml) and follows sitemap index files recursively.
• URLs are tagged with source type and stored in property_urls table.
• Deduplication prevents storing the same URL twice for a given domain.