Web crawler and price algorithm for tourism company

When it comes to data crawling Python Selenium can be a good choice to crawl data from Javascript websites, which are non-static and require the crawler to navigate through the website. In other words, Selenium allows you to transform your webbrowser into a robot - a tool that can be used in many use cases such as site maintenance, performance analysis, or data crawling. One recent use case for such a browser bot was a data crawler for an Irish tourism company, which allowed the company to extract data from Booking.com and calculate price changes.

By using Selenium I can program the webbrowser to navigate through a website, define search parameters and execute search requests. As a response the search request generates an output consisting of a list of accomodations and their associated data, which can be crawled by the bot. As part of the project the crawling bot performed a variety of search requests for each Irish district with different search parameters such as the type of room, the number of persons or the check-in date.

Since the extracted data was unstructered due to the inconsistent availability of different information, one of the main challenges of the project was to figure out how the unstructured data can be transformed into a structured dataset. Moreover, I developed a price algorithm which was fed by the extracted data and which generated average prices for each districts.