Transport Data Curation Scrapping
Transport data scrapping software to provide user with integrated travel route plan
The client ( based in Europe running a successful aggregator business ) wanted to build an advanced scraping software which provides users with a complete integrated travel route from point A to point B based on their predefined preferences. This advanced scraping project had multiple transport agencies hence multiple data source involved and the challenge was to still find the best route stitching travel routes from multiple agencies. There was a lot of security measures like unstructured dom, cross checking user mouse movements, cookie validation etc and we overcame all the challenges to provide the exact software that the client had in mind.
When user inputs data in the metasearch engines to find the desired travel itinerary, we run a script in the background that scrapes all the possible routes and save them in a centralised database. We then aggregate the results into a list of available routes according to either cheapest price or least distance time. Users can then choose from those route options to find the best suitable path according to their preference.
Python, Selenium, mysql, Javascript
