A Comparison of Open Source Web Crawlers for E-Commerce Websites
- Resource Type
- Conference
- Authors
- Yang, Desheng; Thiengburanathum, Pree
- Source
- 2020 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON) Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON), 2020 Joint International Conference on. :200-205 Mar, 2020
- Subject
- Communication, Networking and Broadcast Technologies
Components, Circuits, Devices and Systems
Computing and Processing
Fields, Waves and Electromagnetics
General Topics for Engineers
Power, Energy and Industry Applications
Robotics and Control Systems
Signal Processing and Analysis
Transportation
Crawlers
Web pages
Testing
Uniform resource locators
Licenses
Media
Market research
Open source crawler comparison
Information Retrieval
E-commerce websites
- Language
Web crawlers are important tools for retrieving data such as text or image from the internet. It is an automated program that can retrieve information from the websites. E-commerce websites are essential areas for crawler applications. Moreover, large-scale crawling involves many problems on the internet. Poor performing web crawlers can waste many resources for development and maintenance. Thus, choosing a suitable open source crawler becomes a huge challenge. This paper attempts to review the published previous studies on open source crawlers. The paper focuses on summarizing the performance evaluation methods of open source web crawlers, possible research trends and related research gaps. In addition, a proposed framework of the open source crawler evaluation was presented.