A recent article in the NY Times discusses the ongoing work to help search engines "see" the Deep Web. The deep web refers to information contained in databases and other repositories in which the search engines "spiders" can't search directly. Now computer programs are sending search queries into these databases to try to get a sense of what the content is.
"“Most search engines try to help you find a needle in a haystack,” Mr. Rajaraman said, “but what we’re trying to do is help you explore the haystack.” That haystack is infinitely large. With millions of databases connected to the Web, and endless possible permutations of search terms, there is simply no way for any search engine — no matter how powerful — to sift through every possible combination of data on the fly. "