Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms
Vandana Shrivastava1, Harvir Singh2, Arvind K. Sharma3
1Vandana Shrivastava*,Research Scholar, School of Computer and System Sciences, Jaipur National Univetsiry, Jaipur, India.
2Harvir Singh, Research Scholar, School of Computer and System Sciences, Jaipur National Univetsiry, Jaipur, India.
3Arvind K. Sharma, Research Scholar, School of Computer and System Sciences, Jaipur National Univetsiry, Jaipur, India.
Manuscript received on November 11, 2019. | Revised Manuscript received on November 20 2019. | Manuscript published on 30 November, 2019. | PP: 11642-11651 | Volume-8 Issue-4, November 2019. | Retrieval Number: D9430118419/2019©BEIESP | DOI: 10.35940/ijrte.D9430.118419
Open Access | Ethics and Policies | Cite | Mendeley | Indexing and Abstracting
© The Authors. Blue Eyes Intelligence Engineering and Sciences Publication (BEIESP). This is an open access article under the CC-BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Abstract: World Wide Web (WWW) is a massive resource of information that contains data in zeta bytes. Large number of users surfing Internet for various purposes need a quick response from web. Search engines are preferable, easily available and reliable source to retrieve data very rapidly with minimum resources. Search engine uses Web Crawler to search abundant data from web. It is a program that automatically passes through hypertext structure of the web and access webpage and hyperlink within it. Web crawling is a technique that focuses on obtaining relevant and fruitful links. To make the traversing accurate and productive, a properly designed web crawling algorithm is required. Web crawler and crawling techniques are kept secret as it is a hidden formula for the success of any search engine. In this paper, we have presented various web crawling algorithms along with their pros and cons to be utilized towards web search. Also we have proposed a novel web crawling algorithm to enhance the security and improving the performance of web crawler using single threaded and multi-threaded crawling algorithms.
Keywords: Crawling Algorithms, Search Engine, Uniform Resource Locator, Web Crawler, World Wide Web.
Scope of the Article: Data Analytics Modelling and Algorithms.