[dev thread] NoSQL users - inside please
 

So, I just read that manwin moved youporn off mysql and onto redis and are loving the performance gain. They say youporn gets 100million/day and their redis server is clustered.

I moved off mysql earlier this month and onto lucene when traffic started to seriously increase. This is for 5 million/day and lucene isn't clustered as it isn't even working hard.

I too was blasted by the massive improvement in performance - the visitors did too since page load time was a blink and search was a shitload more useful and accurate (number of daily searches went up 10x within a few days)

So, are you using either of these two, or another nosql flavour? If by any miracle, anyone has tried both, care to give a breakdown of which you settled for and why?

I haven't tried Redis, and after reading about it I think I'm better with lucene, simply because of its faceted search capabilities, that I use heavily...

And before anyone starts blowing the trumpet of mysql, yeah I love mysql too - I'm running other things, particularly one that has 30+million dailies that heavily uses mysql and mysql isn't anywhere near sweating.... but that is a different operation with different needs. What I'm on about are pages that heavily use key/value stores and require superior full text search capabilities.

Eric @manwin must surely have tested lucene, so I'd value his input :winkwink:

for the record, in my case, mysql is feeding the Solr/Lucene index nightly, so I'm not 100% sql-free, but simply using mysql as a backup of the solr index and a feeder to inject into that index.

Dont switch to soon from mysql. Some good tuning can do a lot.
Some of my stats
Traffic ø per hour
Received 0.7 TiB 0.2 GiB
Sent 18.7 TiB 5.0 GiB
Total 19.4 TiB 5.2 GiB

Query type ø per hour %
select 724 M 187.996 k 7.96%

:winkwink::winkwink:

For some things though, mysql is *not* the answer, which is what this thread is about...

i really doubt any of Manwin's programmers visit GFY, probably don't even know it exists.

what's your site running with lucerne?

he has an email address on the page linked from the thread on here - email him.

yeah, I have his email from his post announcing youporn/redis:

https://groups.google.com/forum/?fro...db/d4QcWV0p-YM

Ah well this is even more confusing considering there is somuch nosql databases. http://nosql-database.org/

Yeah there are many NoSQL variants. Last year we made a touchscreen environment running as a PaaS and using couch db combined with Ruby On Rails. Kind of slick stuff :thumbsup

Its a good job all my shit is happy on MySQL - This stuff is way out of my league.

Eddy is happy getting BJs from his wife.
Eddy would be happier if his wife was a supermodel giving said BJs.

I have only used MongoDB in development settings and it has been easy and fun to use. As for your full text searching Lucene is good but I don't think you can beat Sphinx.

You could not be more wrong .... :error

If all you have is a hammer, everything looks like a nail. MySQL is a great relational database but it's a poor full text index. I'm a Sphinx fan myself, it scales like a motherfucker and it's easy to configure, but Lucene should be just as good a bet in the right hands.

If you are not clustered, stay with lucene. Once you decide to go clustered, try MySQL NDBC cluster. The performance is completely different to non clustered MySQL because the data engine is completely different monster.
It still will be unable to beat nosql engines for full text indexing, but until you reach a really big volume the performance will be very good and you will still have all the relational db benefits.

caching is also good...

I was looking at Redis, and I think I read that they're dropping VM support. So if you have really large databases, you have to have the RAM to fit. They also made this comment about running OS X lol.

"Not all kinds of file systems are able to deal with large files in a good way, especially the Mac OS X file system which tends to be really lame about it."

So that means you stick with lucene ?

Bump for nosql users.