HultigCrawler

HultigCrawler is a web crawler that recursively collects all the text from a given website. Each page is stored as an item with its URL, title, tags and text, and saved to a database through Scrapy pipelines.

Supported by project C4 - Cloud Computing Competences Centre, financed by Portugal 2020.