From Fedora Project Wiki
(→Software Investigation and Evaluation: Added table for Comparison by Requirements) |
(→Comparison by Requirements: Change heading from Search to Front-End, since that's what's meant) |
||
| Line 185: | Line 185: | ||
!{{rh}}| Source Language | !{{rh}}| Source Language | ||
!{{rh}}| Integrated Crawler | !{{rh}}| Integrated Crawler | ||
!{{rh}}| Integrated | !{{rh}}| Integrated Front-End | ||
!{{rh}}| Programmable Categories | !{{rh}}| Programmable Categories | ||
!{{rh}}| Application Interface | !{{rh}}| Application Interface | ||
| Line 290: | Line 290: | ||
! Source Language | ! Source Language | ||
! Integrated Crawler | ! Integrated Crawler | ||
! Integrated | ! Integrated Front-End | ||
! Programmable Categories | ! Programmable Categories | ||
! Application Interface | ! Application Interface | ||
Revision as of 05:15, 4 November 2009
Points of Contact
Project Sponsor
Name: Mike McGrath
Fedora Account Name: mmcgrath
Group: Infrastructure
Infrastructure Sponsor: mmcgrath
Secondary Contact info
Name: Huzaifa Sidhpurwala
Fedora Account Name: huzaifas
Group: Infrastructure
Name: Allen Kistler
Fedora Account Name: akistler
Group: Infrastructure
Project Info
Project Name: Search Engine Enhancement
Target Audience: All users of Fedora web sites
Expiration/Delivery Date (required): F13
Description/Summary
Fedora needs a search engine[1]
Requirements
- Crawl the web sites (wiki and non-wiki)
- Search the web sites (wiki and non-wiki)
Preferences
- Python-based
- Note: Other languages are permitted, but Java must be the GCJ/OpenJDK versions in RHEL5. Sun/IBM/BEA Java is not acceptable.
- Programmable keywords to have control over what pages get displayed for certain keywords
- XML or library interface so other applications can use it
Project Plan
- Investigate and evaluate existing open source search engines
- Select candidate software
- Create public test instance of candidate software
- Test for functionality, performance, and impact (re-evaluate, if necessary)
- Create capacity and deployment plans
- Deploy
Resources Needed
- Public Test for testing candidate software
- Permanent home(s) for deployment
- Web server(s)
- Database server(s)
Software Investigation and Evaluation
In Progress
- DataparkSearch [2]
- written in C
- Egothor [3]
- written in Java
- Ferret [4]
- Ruby port of Lucene
- KinoSearch and Ferret intend to merge as Lucy [5]
- Indri [6]
- written in C/C++
- Isearch [7]
- written in C++
- KinoSearch [8] - akistler examined
- Description
- Perl/C port of Lucene
- Maintainer Rectangular Research appears to be just one guy, who considers KinoSearch to be alpha software
- KinoSearch and Ferret intend to merge as Lucy [5]
- Evaluation
- Search engine library with sample indexer and search page rather than fully-functional application. Stores indices in Berkeley DB files with JSON interfaces. Allows custom-designed indices, including categories (exact match) to fulfill "programmable keywords" requirement. Each document index on each document source is a single write-once file collection (BDB and JSON) in a unique directory. Rerunning the indexer creates a new directory, obsoleting the old directory if all the old documents are included. The old directory then needs to be cleaned up. Postings can, however, be deleted from an index. Additionally, only the new documents can be indexed, but that's not efficient.
- Requirements
- buildrequires
- gcc
- (EPEL) perl-Module-Build
- requires
- (EPEL) perl-JSON-XS
- Problem: Desires 1.53, but EPEL has 1.43
- Note: http://web.archive.org/web/20071122035408/search.cpan.org/src/MLEHMANN/JSON-XS-1.53/
- Note: works with 1.43, anyway
- (EPEL) perl-Lingua-Stem-Snowball
- (EPEL) perl-Lingua-StopWords
- (EPEL) perl-Parse-RecDescent
- sample indexer reads files from the file system and requires
- (EPEL) perl-HTML-Tree
- sample cgi search script requires
- (CPAN) Data::Pageset (which requires Data::Page)
- (EPEL) perl-Test-Exception
- (EPEL) perl-Class-Accessor-Chained
- Lucene [9]
- written in Java, but ported to others [10]
- mnoGoSearch [11]
- written in C
- Namazu [12] - Huzaifa investigating
- written in Perl
- Nutch [13]
- written in Java
- based on Lucene
- Swish-e [14]
- written in C
- Swish++ is a rewrite in C++
- Terrier (TERabyte RetrIEveR) [15]
- written in Java
- Xapian [16]
- written in C++
- Bindings to Python, Perl, PHP, Ruby, Java, and more
- Omega (builtin) provides a more complete search engine experience on top of core Xapian
- Built-in web crawler is a script that interfaces with ht://Dig
- Flax [17] is another search engine built on top of Xapian and CherryPy
- Zettair [18]
- written in C
Not Suitable
- Grub [19]
- written in C#
- Heritrix [20]
- written in Java
- archives content rather than indexing it
- ht://Dig [21]
- written in C++
- not actively maintained
- HtdigSearch [22]
- It's just a MediaWiki plugin, not suitable for searching non-wiki sites
- MWSearch [23]
- Requires EzMwLucene (Java, not desirable) to be running on the servers to be searched
- EzMwLucene is wiki-only, therefore MWSearch is wiki-only
- OpenFTS [24]
- written in Perl or TCL on top of PostgreSQL
- Python interface available
- not actively maintained
- Plucene [25]
- Perl port of Lucene
- not actively maintained
- RigorousSearch [26]
- Crawls the MediaWiki database, not the web site
- Doesn't work for non-MediaWiki web sites, including any non-wiki web site
- Sphinx [27]
- written in C++
- designed to index SQL tables, not web pages
- SphinxSearch [28]
- Written in C++
- MediaWiki plug-in, so it's wiki-only
- YaCy [29] - Examined by Huzaifa
- written in Java, but requires Sun Java
- Well maintained
- Support for peer search engine database exchanges
- Customized search parameters
- Fast indexing and web interface for querying the back end db.
Comparison by Requirements
| Engine Name | Source Language | Integrated Crawler | Integrated Front-End | Programmable Categories | Application Interface |
|---|---|---|---|---|---|
| DataparkSearch | C | ||||
| Egothor | Java | ||||
| Ferret | Ruby | ||||
| Indri | C/C++ | ||||
| Isearch | C++ | ||||
| KinoSearch | Perl/C | No (sample included) |
No (sample included) |
Yes | Yes (BDB/JSON) |
| Lucene | Java | ||||
| mnoGoSearch | C | ||||
| Namazu | Perl | ||||
| Nutch | Java | ||||
| Swish-e | C | ||||
| Terrier | Java | ||||
| Xapian | C++ | ||||
| Zettair | C | ||||
| Engine Name | Source Language | Integrated Crawler | Integrated Front-End | Programmable Categories | Application Interface |
Public Testing
<tbd>
Deployment Plan
<tbd>
References
- ↑ "Fedora Search Engine". Infrastructure/Tickets. https://fedorahosted.org/fedora-infrastructure/ticket/1055.
- ↑ "DataparkSearch". DataparkSearch. http://www.dataparksearch.org/.
- ↑ "Egothor". Egothor. http://www.egothor.org/.
- ↑ "Ferret". David Balmain. http://ferret.davebalmain.com/.
- ↑ 5.0 5.1 "Lucy". Apache Software Foundation. http://lucene.apache.org/lucy/.
- ↑ "Indri". The Lemur Project. http://www.lemurproject.org/indri/.
- ↑ "Isearch". Isite. http://isite.awcubed.com/.
- ↑ "KinoSearch". Rectangular Research. http://www.rectangular.com/kinosearch/.
- ↑ "Lucene". Apache Software Foundation. http://lucene.apache.org/.
- ↑ "Lucene Implementations". Apache Software Foundation. http://wiki.apache.org/lucene-java/LuceneImplementations.
- ↑ "mnoGoSearch". LavTech. http://www.mnogosearch.org/.
- ↑ "Namazu". Namazu Project. http://www.namazu.org/.
- ↑ "Nutch". Apache Software Foundation. http://lucene.apache.org/nutch/.
- ↑ "Swish-e". Swish-e. http://swish-e.org/.
- ↑ "Terrier". Terrier Project. http://ir.dcs.gla.ac.uk/terrier/.
- ↑ "Xapian". Xapian Project. http://xapian.org/.
- ↑ "Flax". Flax. http://www.flax.co.uk/products.shtml.
- ↑ "Zettair". Search Engine Group, Royal Melbourne Institute of Technology. http://www.seg.rmit.edu.au/zettair/.
- ↑ "Grub". Wikia, Inc.. http://grub.org/.
- ↑ "Heritrix". Internet Archive. http://crawler.archive.org/.
- ↑ "ht://Dig". The ht://Dig Group. http://www.htdig.org/.
- ↑ "HtdigSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:HtdigSearch.
- ↑ "MWSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:MWSearch.
- ↑ "OpenFTS". SourceForge. http://openfts.sourceforge.net/.
- ↑ "Plucene". CPAN. http://search.cpan.org/~tmtm/Plucene-1.25.
- ↑ "RigorousSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:RigorousSearch.
- ↑ "Sphinx". Sphinx Technologies. http://sphinxsearch.com/.
- ↑ "SphinxSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:SphinxSearch.
- ↑ "YaCy". Karlsruhe Institute of Technology. http://yacy.net/.