From Fedora Project Wiki
(Add Xapianinfo pointing to Flax)
(→‎In Progress: KinoSearch - Add basic evaluation; Add info on sample requirements)
Line 72: Line 72:
* KinoSearch <ref name="KinoSearch">{{cite web|url=http://www.rectangular.com/kinosearch/|title=KinoSearch|publisher=Rectangular Research}}</ref> '''- Allen investigating'''
* KinoSearch <ref name="KinoSearch">{{cite web|url=http://www.rectangular.com/kinosearch/|title=KinoSearch|publisher=Rectangular Research}}</ref> '''- Allen investigating'''
:* Perl/C port of Lucene
:* Perl/C port of Lucene
: buildrequires
:* Evaluation
:* gcc
::Search engine library with sample indexer and search page rather than fully-functional application.  Stores indices in Berkeley DB files with JSON format specifications.  Allows custom-designed indices.  Each document source is indexed as a single write-once index, which is a directory.  Rerunning the indexer creates a new directory, obsoleting the old directory if all the old documents are included.  The old directory then needs to be cleaned up.  Postings can, however, be deleted from an index.  Additionally, only the new documents can be indexed, but that's not efficient.
:* (EPEL) perl-Module-Build
:* Requirements
: requires
:: buildrequires
:* (EPEL) perl-JSON-XS = 1.53
::* gcc
::  Problem: EPEL has 1.43
::* (EPEL) perl-Module-Build
::: major interface change at 2.0
:: requires
::: CPAN no longer has source earlier than 2.25
::* (EPEL) perl-JSON-XS
::: Note: DAG has 1.53
:::  Problem: Desires 1.53, but EPEL has 1.43
::: Note: <nowiki>http://web.archive.org/web/20071122035408/search.cpan.org/src/MLEHMANN/JSON-XS-1.53/</nowiki>
:::: Note: <nowiki>http://web.archive.org/web/20071122035408/search.cpan.org/src/MLEHMANN/JSON-XS-1.53/</nowiki>
:* (EPEL) perl-Lingua-Stem-Snowball
:::: Note: works with 1.43, anyway
:* (EPEL) perl-Lingua-StopWords
::* (EPEL) perl-Lingua-Stem-Snowball
:* (EPEL) perl-Parse-RecDescent
::* (EPEL) perl-Lingua-StopWords
::* (EPEL) perl-Parse-RecDescent
:: sample indexer reads files from the file system and requires
::* (EPEL) perl-HTML-Tree
:: sample cgi search script requires
::* (CPAN) Data::Pageset (which requires Data::Page)
::* (EPEL) perl-Test-Exception
::* (EPEL) perl-Class-Accessor-Chained


* mnoGoSearch <ref name="mnoGoSearch">{{cite web|url=http://www.mnogosearch.org/|title=mnoGoSearch|publisher=LavTech}}</ref>
* mnoGoSearch <ref name="mnoGoSearch">{{cite web|url=http://www.mnogosearch.org/|title=mnoGoSearch|publisher=LavTech}}</ref>

Revision as of 22:23, 22 October 2009

Points of Contact

Project Sponsor

Name: Mike McGrath
Fedora Account Name: mmcgrath
Group: Infrastructure
Infrastructure Sponsor: mmcgrath

Secondary Contact info

Name: Huzaifa Sidhpurwala
Fedora Account Name: huzaifas
Group: Infrastructure

Name: Allen Kistler
Fedora Account Name: akistler
Group: Infrastructure

Name: Keiran Smith
Fedora Account Name: affix
Group: Infrastructure

Project Info

Project Name: Search Engine Enhancement
Target Audience: All users of Fedora web sites
Expiration/Delivery Date (required): F13

Description/Summary

Fedora needs a search engine[1]

Requirements

  • Crawl the web sites (wiki and non-wiki)
  • Search the web sites (wiki and non-wiki)

Preferences

  • Python-based (no Java)
  • Programmable keywords to have control over what pages get displayed for certain keywords
  • XML or library interface so other applications can use it

Project Plan

  1. Investigate and evaluate existing open source search engines
  2. Select candidate software
  3. Create public test instance of candidate software
  4. Test for functionality, performance, and impact (re-evaluate, if necessary)
  5. Create capacity and deployment plans
  6. Deploy

Resources Needed

  • Public Test for testing candidate software
  • Permanent home(s) for deployment
    • Web server(s)
    • Database server(s)

Software Investigation and Evaluation

In Progress

  • DataparkSearch [2]
written in C
written in C/C++
written in C++
  • KinoSearch [5] - Allen investigating
  • Perl/C port of Lucene
  • Evaluation
Search engine library with sample indexer and search page rather than fully-functional application. Stores indices in Berkeley DB files with JSON format specifications. Allows custom-designed indices. Each document source is indexed as a single write-once index, which is a directory. Rerunning the indexer creates a new directory, obsoleting the old directory if all the old documents are included. The old directory then needs to be cleaned up. Postings can, however, be deleted from an index. Additionally, only the new documents can be indexed, but that's not efficient.
  • Requirements
buildrequires
  • gcc
  • (EPEL) perl-Module-Build
requires
  • (EPEL) perl-JSON-XS
Problem: Desires 1.53, but EPEL has 1.43
Note: http://web.archive.org/web/20071122035408/search.cpan.org/src/MLEHMANN/JSON-XS-1.53/
Note: works with 1.43, anyway
  • (EPEL) perl-Lingua-Stem-Snowball
  • (EPEL) perl-Lingua-StopWords
  • (EPEL) perl-Parse-RecDescent
sample indexer reads files from the file system and requires
  • (EPEL) perl-HTML-Tree
sample cgi search script requires
  • (CPAN) Data::Pageset (which requires Data::Page)
  • (EPEL) perl-Test-Exception
  • (EPEL) perl-Class-Accessor-Chained
  • mnoGoSearch [6]
written in C
  • Namazu [7] - Huzaifa investigating
written in Perl
written in C
Swish++ is a rewrite in C++
written in C++
Bindings to Python, Perl, PHP, Ruby, Java, and more
It looks to me like we need a web crawler built on top of this. Searching via google turned up Flax[10]
written in C
written in C

Not Suitable

written in Java
written in C#
  • written in Java
  • archives content rather than simply indexing it
  • written in C++
  • not actively maintained
It's just a MediaWiki plugin, not suitable for searching non-wiki sites
written in Java, but ported to others [19]
  • Requires EzMwLucene (Java, not desirable) to be running on the servers to be searched
  • EzMwLucene is wiki-only, therefore MWSearch is wiki-only
  • written in Java
  • based on Lucene
  • written in Perl or TCL on top of PostgreSQL
  • Python interface available
  • not actively maintained
  • Perl port of Lucene
  • not actively maintained
Crawls the MediaWiki database, not the web site
Doesn't work for non-MediaWiki web sites, including any non-wiki web site
written in C++
designed to index SQL tables, not web pages.
Written in C++
Wiki-only (?)
  • Terrier (TERabyte RetrIEveR) [27]
written in Java

Public Testing

<tbd>

Deployment Plan

<tbd>

References

  1. ↑ "Fedora Search Engine". Infrastructure/Tickets. https://fedorahosted.org/fedora-infrastructure/ticket/1055. 
  2. ↑ "DataparkSearch". DataparkSearch. http://www.dataparksearch.org/. 
  3. ↑ "Indri". The Lemur Project. http://www.lemurproject.org/indri/. 
  4. ↑ "Isearch". Isite. http://isite.awcubed.com/. 
  5. ↑ "KinoSearch". Rectangular Research. http://www.rectangular.com/kinosearch/. 
  6. ↑ "mnoGoSearch". LavTech. http://www.mnogosearch.org/. 
  7. ↑ "Namazu". Namazu Project. http://www.namazu.org/. 
  8. ↑ "Swish-e". Swish-e. http://swish-e.org/. 
  9. ↑ "Xapian". Xapian Project. http://xapian.org/. 
  10. ↑ "Template:Citation error". Flax. http://www.flax.co.uk/products.shtml. 
  11. ↑ "YaCy". Karlsruhe Institute of Technology. http://yacy.net/. 
  12. ↑ "Zettair". Search Engine Group, Royal Melbourne Institute of Technology. http://www.seg.rmit.edu.au/zettair/. 
  13. ↑ "Egothor". Egothor. http://www.egothor.org/. 
  14. ↑ "Grub". Wikia, Inc.. http://grub.org/. 
  15. ↑ "Heritrix". Internet Archive. http://crawler.archive.org/. 
  16. ↑ "ht://Dig". The ht://Dig Group. http://www.htdig.org/. 
  17. ↑ "HtdigSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:HtdigSearch. 
  18. ↑ "Lucene". Apache Software Foundation. http://lucene.apache.org/. 
  19. ↑ "Lucene Implementations". Apache Software Foundation. http://wiki.apache.org/lucene-java/LuceneImplementations. 
  20. ↑ "MWSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:MWSearch. 
  21. ↑ "Nutch". Apache Software Foundation. http://lucene.apache.org/nutch/. 
  22. ↑ "OpenFTS". SourceForge. http://openfts.sourceforge.net/. 
  23. ↑ "Plucene". CPAN. http://search.cpan.org/~tmtm/Plucene-1.25. 
  24. ↑ "RigorousSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:RigorousSearch. 
  25. ↑ "Sphinx". Sphinx Technologies. http://sphinxsearch.com/. 
  26. ↑ "SphinxSearch Extension". MediaWiki. https://secure.wikimedia.org/wikipedia/mediawiki/wiki/Extension:SphinxSearch. 
  27. ↑ "Terrier". Terrier Project. http://ir.dcs.gla.ac.uk/terrier/. 
Retrieved from "https://fedoraproject.org/w/index.php?title=Infrastructure/Search&oldid=131907"