Knowlesys web data mining system

The web is an ocean of information containing more than 10 billion web pages, wherein 90% of them are in non-structured or semi-structured formats. At present, it is expanding with an increasing rate of 1 million pages per day. The information is increasing at an explosive speed while people¡¯s time and energy are limited. The information absolutely valuable for enterprises or individuals is just lying in this worldwide ocean of the Internet, and how to extract them has become one of the most imperative tasks confronting the research institutions that are engaging the important topics of Information Retrieval, Data Mining, Knowledge Management and Competitive Intelligence etc. The Knowlesys Web Data Mining System(KWDMS) is like a huge blue whale who cruises in this information ocean everyday and is capable of automatically and accurately extracting valuable data for you from the webpage ocean wherein a multitudes of useless text (such as page headers and footers, column listings and advertisement messages) shall be excluded. In more than five year¡¯s time, the Knowlesys Software, Inc. had developed the KWDMSíºC a powerful web information extraction system. It has a stratified structure and a loosely coupled module design comprising many sub-systems. The KWDMS can extract designated information in big volume from the web, and integrate them into specified relational databases, thus to help customers to excavate precious stones from the Internet minefield. Since the process converses the information from the semi-structural form into the structural form, from their dispersed state to the concentrated state, and changes them from the remotely existed information to your locally hoarded treasure, as well as from the visual file into the digital record, you can surely extensively use them in the future. The KWDMS is capable of doing data extraction from various types of websites. In addition to extracting field data of semi-structured construction, it can also extract some free text information like e-mail addresses and many types of multimedia files. The KWDMS is characterized as a stable running, intelligent crawling and accurate extracting software. The KWDMS is an information extraction platform. When new extraction task is required, it is necessary to use this platform to configure the new web crawling and extraction script and parameters. A general database access layer is developed in the KWDMS that enables its back end connect to any relational database, such as MS SQL Server, Oracle, DB2, Sybase, MySQL and InterBase etc, even those file database like the Access database. Regardless which type the database is, the extracted data can be checked with a general database browser, as well as export them into various formats such as XML, CVS, HTML, Excel and so on.