Reply to comment

WorldWideScience provides a one-stop search engine to mine global scientific databases in the deep web

The internet has revolutionized society by changing the way people communicate, find information, and enjoy entertainment. But a standard internet search misses at least 90 percent of the information available. 

The internet is separated into two unequal pools of information. The surface web contains pages of information that are utilized by popular search engines. The second pool of information is locked away in the deep web, which consists of countless databases world wide.

According to Walt Warnick, Director of the DOE Office of Scientific and Technical Information (OSTI), "The deep web is huge."

Common browsers like Google and Yahoo crawl across the thousands of internet pages on the surface web, but are unable to dig into the databases to retrieve information from the deep web.

"Asking a scientist, engineer, or educator to find information in their field using common web browsers is like asking a doctor to diagnose disease without X-rays, MRI, or any other piece of diagnostic equipment" said Warnick.

Information in the deep web can only be mined for data using search engines designed for that particular database. Many of the search engines that are available to mine databases often do not use relevance ranking, making filtering through the information a crap shoot.

"Under the current system, finding information in the deep web is a series of practical impossibilities, placing internet users, especially scientists and science educators, at a severe disadvantage" said Warnick.

To address the global science need, OSTI has launched WorldWideScience.org, a science gateway that accelerates the search for data in national and international scientific databases and portals on the internet.  The data spans physical and life sciences as well as medical studies.

Warnick called the development of this technology "a series of sequential miracles" that makes the deep web accessible to a wider audience where billions of dollars worth of government-sponsored scientific research results reside.

WorldWideScience provides a one-stop search engine for global scientific databases. When a query is entered into the search engine, it is transmitted to the gateway server at OSTI in Oak Ridge, Tennessee. Going beyond the capabilities of the common web crawler, which searches the internet horizontally; this new search engine adds the enhanced capability of simultaneously searching a select group of databases vertically.

Since the debut of the search engine prototype in 2007, WorldWideScience has expanded from 10 to 56 participating countries. China is the most recent country to contribute to the search engine, opening a wealth of scientific information to the world. The search engine now scours more than 375 million pages of scientific information contained in deep web databases.

The multilateral WorldWideScience Alliance consists of the participating countries and was established in June 2008 to govern this rapidly growing online gateway to international scientific research information#595959">.

Despite these advances, there is more work to do. Currently, WorldWideScience is limited to searching databases with English titles and abstracts. This constraint confines the number of databases accessible by the search engine. The Alliance is now exploring translation technologies to expand the network of databases accessible to the worldwide community and is making progress toward deploying this capability.

This work is supported by the Department of Energy (DOE) Office of Scientific and Technical Information (OSTI), within the Office of Science. DOE invests in science and solving critical issues impacting people's daily lives and the nation's future. For more information, visit http://www.science.doe.gov/index.htm.

OSTI advances science and sustains technological creativity by making the research and development findings available and useful to DOE researchers and the public.

 

By Stacy W. Kish

OSTI

202-586-5278

July 14, 2009

 

read more...

Reply

Comments policy

We welcome your comments and you submission of web links to the OSTIblog and look forward to civil discourse on a variety of science and technology information topics. We will review comments before posting and we reserve the right to not post comments.

We prefer comments and links that are specific to the subject of the OSTIblog post.

You are fully responsible for everything that you submit, and all posted comments are in the public domain. This means that your comments could be distributed widely.

You may comment anonymously. Your name, website, and email are not required.

By selecting the preview button, submit button, and/or by submitting anti-spam answers, you accept these terms and conditions.

(If you're a human, don't change the following field)
Your first name.
The content of this field is kept private and will not be shown publicly.
By submitting this form, you accept the Mollom privacy policy.
http://www.osti.gov/ostiblog/comment/reply/1538?node=
www.osti.gov
crawl=CC-MAIN-2013-20
29458
application/http; msgtype=response
sha1:Y7RRE64E77ESEVKIMU4SG5YFELYQ44DE
<urn:uuid:74295bdd-b902-4e80-bfb4-b9a64fc790df>
2013-05-26T05:38:37Z
192.107.175.222
sha1:TFOON5OSGGFNF4BQNISUJLDQLCLV2SHJ
<urn:uuid:09830a63-8893-47aa-8062-46ba39ba4166>
http://www.osti.gov/ostiblog/comment/reply/1538?node=
response
<urn:uuid:5722e491-bd8b-4659-9fd4-e4662d3236c0>
robots: classic
hostname: ip-10-60-113-184.ec2.internal
software: Nutch 1.6 (CC)/CC WarcExport 1.0
isPartOf: CC-MAIN-2013-20
operator: CommonCrawl Admin
description: Wide crawl of the web with URLs provided by Blekko for Spring 2013
publisher: CommonCrawl
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf