BIG DATA USE CASE
Revolutionizing the Online Map
Maptimize, the company behind the ‘One Million Tweet Map’, provides a clustering application to display massive amounts of information – such as Tweets, users or points of interest – on a map. A US real estate agency uses Maptimize to show sales across the country. Manchester United in the UK displays its fans on a map with Maptimize. French newspaper 20 Minutes uses Maptimize to present photographs from readers on a map.
Displaying one million points on a map did not come easy, however. The first version of Maptimize, a mix of Ruby and SQL, was very slow and not scalable. The next version, written in Java, was also very hard to scale. In addition, real-time updating was not possible, so data points could not be added or removed quickly, which was a serious issue for some users. Maptimize was also initially only available as a Software-as-a-Service (SaaS) product, and a lot of potential customers wanted to install the software on their own servers.
To solve all the challenges and make the system scalable, the company developed an Elasticsearch plugin.
Elasticsearch Puts Maptimize Back on the Map
Originally many companies liked the idea of Maptimize’s clustering service, and asked if the solution could handle one million points. Before Elasticsearch it was not possible. The company had to tell potential clients they could not do it. The first implementation of Maptimize was limited to about 10,000 points.
We saw immediately that Elasticsearch could solve many of our issues,” explains Sebastien Gruhier, Developer, Maptimize. “Elasticsearch is very easy to scale because you just have to add a node on your cluster. This allowed us to handle an unlimited number of points, because we can add as many servers as we need to the cluster and Elasticsearch replicates automatically. It’s like magic. Today, applications can go from a handful to thousands of users in a month. If you can’t scale, you’re dead. That’s why the scalability of Elasticsearch is very important. I can now say to our customers: if you’re data doubles, triples, increases 10 times, you will have no problem displaying the points on a map. With Elasticsearch, we could finally put a million points on a map. That was the rebirth of Maptimize.”
Real-time Updates
Real-time updating is another very important advantage of Elasticsearch. For example, a company may want to display online users, which log on every few seconds, on a map. With real-time updating, the new users can be displayed immediately. Before Elasticsearch, Maptimize had to update all the data points, not just the new data points, which took a long time.
Real-time update is a built-in Elasticsearch feature,” Gruhier explains. “Maptimize can now add or remove a point in milliseconds versus 5 or 10 minutes before Elasticsearch.”
Meeting Customer Expectations: Scalability and Reliability
Before Elasticsearch, building the clustering engine was a challenge for Maptimize. The company not only had to develop the algorithm itself, but also several capabilities including scalability, node/network management and monitoring, all of which were even more complicated than the algorithm.
Elasticsearch allowed us to focus on our core competency, the clustering algorithm,” says Gruhier. “Everything else is handled perfectly by Elasticsearch. Elasticsearch offers Maptimize scalability in terms of the number of requests. Elasticsearch provides a very fast response,” says Gruhier. “One client asked if our system could handle 1,000 requests per second, which we never achieved before. A quick test in the cloud proved it was possible.”
Elasticsearch replicas ensure reliability for Maptimize customers. Every node is redundant in the cluster.
If one node goes down, Elasticsearch continues work without any downtime,” confirms Gruhier.
Monitoring System Health with Elasticsearch
Marvel Maptimize uses Elasticsearch Marvel, a real-time management and monitoring tool, to understand and monitor their SaaS product. With Marvel, Gruhier can see which index is active and which index is not. He can see who is using the SaaS version and who is not. He can even see how many documents are in each index, how many points are on the map, how much traffic is on a map and real-time search requests.
Having a monitoring tool like Marvel is awesome,” Gruhier raves. “I no longer have to spend a lot of time checking logs to find out what is going on – if I have a red light from Marvel, I know instantly what server I need to add or what index I need to migrate. Marvel gives me the confidence that everything is working in Elasticsearch as needed.”
Maptimize’s Benefits Using Elasticsearch
Unlimited scalability – Elasticsearch’s scalability empowered Maptimize to add one million points on a map. This powerful capability made Maptimize’s commercial viability possible.
Real-time updating – With built-in real-time indexing, Elasticsearch enables Maptimize to add or remove a data point in milliseconds.
Solid reliability – Elasticsearch offers Maptimize reliability and redundancy to eliminate downtime.
Visibility into system health – By monitoring its deployment with Elasticsearch Marvel, Maptimize has a faster mean time to recovery.
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind