The new Lucene Based Search Architecture of Twitter

A new backend search architecture that was launched few weeks ago. The previous system was based on the original Summize search system that Twitter acquired in 2008. The old system was based on MySQL. This system became difficult to scale for such large data. They turned toward a new, modern search architecture that is based on a highly efficient inverted index instead of a relational database, which is the obvious choice for large data indexing for quick search.
                   They did lots of tuning for improving the current version of  Lucene. Some of the highlights of the  changes include:

• significantly improved garbage collection performance
• lock-free data structures and algorithms
• posting lists, that are traversable in reverse order
• efficient early query termination 

We can hope to find these contributions to the Lucene soon.

According to the blog, the new search system was designed to handle over 1,000 TPS (Tweets/sec) and 12,000 QPS (queries/sec) = over 1 billion queries per day . Besides the challenging query volume, the data needs to be available quickly, a tweet needed to be searchable in less than 10 seconds.

The main benefit to users is that the new system is much more scalable and can support an index that is twice as large as previous versions which means that you can search for tweets further back in time.