FYI Riak’s search backend is also Solr. (I’ve recently had to hack a few Solr schemas in Riak)

I needed a crawler for something else anyway so I put a quick project together to try all this out. Used the elixir ecosystem to handle each aspect of search. It’s interesting the design concerns that come up as you can see from my growing wish-list on the readme. That search article you linked @achempion has a lot of great food for thoughts I may try to address.

Anyway running a search site has been pretty fun so I posted a demo site with only a couple sites partially indexed (elixirforum and infogalactic… the index will grow a little every day). And I actually need Riak for something too so I may try swap that in. The readme lists the tech choices I made thus far.

Source Code
Demo site

It’s an interesting topic to me as I’ve been thinking of doing the same thing… my impetus was poor results in medical based searches.
Had also chosen Sphinx for reasons of speed/scale/ease of uptime. Not much to add for storage… think I’d go with Couch or Riak over Cassandra just to keep the stack in line and I don’t think there’s enough potential additional upside in Cass to consider it (if someone knows otherwise I’m interested to hear the argument). Even Riak’s timeseries optimization could have interesting uses in this case. I’ve never run sphinx indexer off those solutions; are you planning to update the index via xmlpipe2 or have something else in mind?

No concrete answer yet but I want to keep up with this convo

I’ve done some research and apparently Cassandra is more appropriate of these three options.

Riak unfortunately is vastly outdated running on r16/r17 and founded company looks like a dissolved.

CouchDB also an interesting solution but too much hustle with adding/removing nodes and redistributing the data.

What I like though is C++ based rewritten version of Cassandra called ScyllaDB.

Couple of more useful links to read about solutions to power search (Solr, Sphinx and so on)

• https://stackoverflow.com/questions/737275/comparison-of-full-text-search-engine-lucene-sphinx-postgresql-mysql
• https://stackoverflow.com/questions/1284083/choosing-a-stand-alone-full-text-search-server-sphinx-or-solr