It started with some dates. An important aspect of the online shopping experience is that you cannot immediately receive your purchase. In Redbubble’s case it has to be both manufactured and shipped before reaching the buyer. Our customer service department frequently handles requests regarding arrival dates and possible late items. As such, we felt we could give customers a clearer understanding of the delivery time frame by providing this information (previously only on the shipping & delivery page) in more places. We decided we would build a new application that would take an item, shipping time and delivery location and return an estimated delivery date based on that data. Why build a new application? As part of the work page (where
Consul is awesome, and super powerful, but takes a bit of understanding and setting up. We are looking at it now, because it could let us keep the same mechanism in place in development as we might use in production.
Every day hundreds of thousands of users come to Redbubble searching for art works. Given the large number of works on Redbubble – we host millions of works for our ever-growing community of artists – it’s important that our search engine return good results, because our users sure ain’t going to paginate through millions of works! Standing between us and our quest to produce relevant search results was our 3-year old Solr 3.6. It’s not so much the age that was bothering us, but rather its lack of boolean and relevance functions. Add in the fact that Solr 3.6’s higher memory demand was causing occasional performance problems, we had a solid case to say goodbye to this fella. So we
We’ve previously posted about measuring what’s happening on Redbubble, but with NewRelic’s new plugin system our lives just got a bit easier. We’ve been using NewRelic for a while now and frequently make use of it’s application and server monitoring for diagnosing problems. Generally the timescales it keeps data for are sane, integration with other tools is good and the UI is easy to use. There are already a plethora of plugins waiting for you to install, although NewRelic’s definition of a plugin is not what we expected. The plugins are small applications (agents) that poll your apps and report back to NewRelic using your licence key and an agent_guid for identification. So installation isn’t all that straightforward, depending on what you want to monitor.
A little while ago, we realised that at having information about what was happening on Redbubble right now would be useful for many things; including tracking and alerting us to the sorts of issues that wouldn’t necessarily be caught by more traditional means (such as Airbrake alerting). Having come across this post by Etsy, we looked at statsd as a way to collect this information in a quick way without adding much processing overhead.
Recently we noticed that one of our Solr slave processes (a piece of software we use to power the search feature at RedBubble) was taking longer to respond to search queries than usual – enough to trigger a warning alert on our monitoring systems. On first inspection, we saw that the machine was using a lot of memory – enough to go into swap space, but this was sitting around the same level as our other server, which was not experiencing problems. A little more digging, and we noticed, thanks to the JVM monitoring provided by NewRelic, that the Solr process was spending a lot of time (around 15% on average, but up to 50%) doing garbage collection. This means