Distributed Web 2.0 crawling for ontology evolution

The World Wide Web as a social network reflects changes of interest in certain domains. It has been shown that free online content available through blogs, wikis, news media and online forums is a valuable source of information to identify trends in certain domains. Utilizing this data, one can construct ontologies that describe this information and provide a semantically correct overview of a domain. Tracked over time this also enables a user to identify trends and hypes. The decentralised structure of the Internet, the huge amount of data and upcoming Web2.0 technologies pose several challenges to a crawling system for ontology learning, evolution and trend analysis. This paper presents a distributed crawling system with browser integration for Web2.0. The proposed crawler is a high performance Web data retrieval system aimed to gather browser-equivalent textual Web content and prepare it for ontology learning.