J Merrill urges we look at http://www.dtsearch.com/, saying they might be expensive but they might also have software we could use as subroutines.
Andy Fluck at www.thataway.org says he uses the Fluid Dynamics Search Engine (written in PERL) in a minimal way. In the near future he expects to put its full capability to work.
Looking at the results of Freefind.com on SwimGold.org today on behalf of Michelle Davidson and remembering my experience on http://www.nytimes.com/ it is clear that we must have a search engine that does a better job than "one or more words appear one or more times". Google is way better than that. We should be also.
The following are Jim Matysek's comments.
USMS.org uses as its search engine ht://Dig whose website can be found at www.htdig.org. This is a framed site, so you can use the links on the left to navigate, but the specific page showing the tool's features is at www.htdig.org/require.html, and a partial list of sites using this tool is at www.htdig.org/uses.html.
I can't speak for these yet in any detail, but one that I have seen recommended is http://www.mnogosearch.org/. If I were to be looking for a new search engine now, this is where I would start.
I've also seen references to http://www.agl.uh.edu/~saljxk/site_Search/, but with a quick glance there I believe this is not in the same class as ht://Dig or mnoGoSearch.
I used to use a perl script that I believe came from Matt's Script Archives. Can't recall the URL, but doing a web search for Matt's Script Archives should find it easily. This type of search did the search in real-time, covering all pages in a directory at the point of the search. This type of search without an index is not appropriate for larger sites like ours, so I changed as our site grew.