In this special guest feature, Slava Akhmechet of RethinkDB discusses where the industry is going in terms of real-time app development, and what the NoSQL database that powers the new wave of connected devices and real-time apps will need to look like. Slava is CEO of RethinkDB, the open-source, scalable database for the realtime web.
The rapidly evolving NoSQL landscape includes a broad spectrum of different technologies. It encompasses graph, column and document databases, as well as key/value stores. Some are general-purpose databases that support a wide range of uses. Others, however, offer specialized capabilities that are dedicated to particular usage scenarios. Consider ElasticSearch, for example, which is designed specifically for full-text indexing. Such databases, which address particular parts of the application stack, are often used alongside general-purpose databases in a complementary fashion.
There are also some databases that are designed to handle general-purpose workloads, while also offering specialized features that make them ideally suited for certain tasks. Couchbase, for example, has synchronization features that make it a good choice for building backend applications that sync with offline-enabled mobile clients.
The growing complexity of distributed applications and the need for more aggressive horizontal scalability contributed to the initial popularization of NoSQL databases. The ongoing proliferation of new NoSQL databases reflects the breadth of competing approaches and the demand for more specialized storage systems.
Large backend applications increasingly use multiple database systems to solve different kinds of problems at various layers of the stack. That trend will continue as more specialized databases continue to emerge. Database developers will also continue pursuing opportunities to simplify backend architecture by incorporating richer functionality directly into the persistence layer.
Realtime application development is one area where specialized database support could prove especially effective. Developers want to build realtime applications that span screens, delivering live updates in the web browser and on mobile devices. Modern technologies like WebSockets make it easier than ever before for applications to transport realtime updates to the frontend, but developers must still contend with considerable backend complexity when building and scaling realtime applications.
When scaling realtime applications horizontally, developers have to ensure realtime events propagate between individual instances of their application. Typically, developers use message queues and in-memory databases to implement publish/subscribe systems that relay realtime updates between server nodes. Moving that capability up to the persistence layer holds the potential to radically simplify the way that realtime applications are built and scaled.
Instead of building publish/subscribe systems on top of external message queues, the database itself could furnish an equivalent mechanism. Applications could then use simple queries to subscribe to database changes, flattening the traditional publish/subscribe system and avoiding the need for redundant external infrastructure.
At RethinkDB, we are building our own take on a database for realtime applications. Our database supports changefeeds, a feature that makes it possible for the application to receive callbacks when changes occur to a realtime query’s underlying result set.
Imagine a realtime chat application built on top of RethinkDB, using sticky load balancing to distribute the audience across multiple servers. Each server uses a changefeed to monitor a “messages” table for updates. When a server receives a new message, it writes it into the table. All of the servers that are subscribed to the table will receive the message, which they can broadcast to their connected clients through Socket.io or a comparable WebSocket abstraction layer. Without a database that supports realtime updates, you would need to use a message queue or something of that nature to propagate the event.
Some existing realtime web frameworks, such as Meteor, tail the MongoDB oplog to extract a stream of realtime events directly from the database. Although that approach works, it doesn’t provide the flexibility or granularity that you can achieve by using realtime database queries to filter for specific kinds of changes.
As live updates and mobile push notifications become more pervasive and indispensable, the need for native realtime query support in database systems will arguably increase.
Sign up for the free insideBIGDATA newsletter.
Speak Your Mind