Video:
Uber's Ringpop and the Fight for Flap Dampening
http://basho.com/posts/technical/ubers-ringpop-and-riak/
Ringpop is an open-source Node.js library developed at Uber that brings application-layer sharding to many of their dispatching platform services.
This additional abstraction layer, maintained through a consistent hashing ring familiar to any Riak enthusiast, provides a means by which Jeff can add additional dispatching services without service interruption.
http://ringpop.readthedocs.org/en/latest/getting_started.html
Ringpop is a library that maintains a consistent hash ring and can be used to arbitrarily shard the data in your application in a way that’s adaptable to capacity changes and resilient to failure.
Ringpop is best described by introducing its 3 core features: a membership protocol, a consistent hash ring and request forwarding.
its membership protocol provides a distributed application, whose instances were once completely unaware of one another, with the ability to discover one another, self-organize and cooperate. The instances communicate over a TCP backchannel and pass information between them in an infection-style manner. Enough information is shared to allow these instances to come to an agreement, or converge, on whom the participating instances, or members, are of the distributed application.
With a consistent membership view, Ringpop arranges the members along a consistent hash ring, divides up the integer keyspace into partitions and assigns ownership of the partitions to the individual instances of your application. It then projects a keyspace of your choosing, say the ID range of the objects in your application, onto that same ring and resolves an owner for each ID. In the face of failure, the underlying membership protocol is resilient and automatically reassigns ownership, also known as rebalancing, to the surviving instances.
./scripts/tick-cluster.js -n 7 -i node ./main.js
Ringpop by Uber
http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html
• Implements a variation of SWIM
• Flap damping
• Use checksums to verify correctness of ring state
• Proxying capabilities
var ringpop = new RingPop({
app: 'myapp',
hostPort: 'myhost:30000'
});
ringpop.bootstrap(['myhost:30001’, 'myhost2:30000']);
ringpop.on('ready', function() {
// do something
});
var node = ringpop.lookup(‘[unique-request-id]’);
if (node === ringpop.whoami()) {
// process request
} else {
// forward request
}
Ringpop Serial
• Simple ringpop wrapper
• Requests are queued by key
• Processed serially, one at a time
• Emulates transactions
Transactions
• Conflicts are possible during membership changes
• Need smart application level conflict resolution
Sevnup
• Open sourced node.js module
• Ringpop extension
• Key ownership hand-off
• Customizable recovery & release
• Pluggable persistence layer
Reliable Timers
• Node.js offers in-memory timers
• Use sevnup to make them reliable
• Riak as persistence layer
http://highscalability.com/blog/2015/10/12/making-the-case-for-building-scalable-stateful-services-in-t.html
Uber's Ringpop and the Fight for Flap Dampening
http://basho.com/posts/technical/ubers-ringpop-and-riak/
Ringpop is an open-source Node.js library developed at Uber that brings application-layer sharding to many of their dispatching platform services.
This additional abstraction layer, maintained through a consistent hashing ring familiar to any Riak enthusiast, provides a means by which Jeff can add additional dispatching services without service interruption.
http://ringpop.readthedocs.org/en/latest/getting_started.html
Ringpop is a library that maintains a consistent hash ring and can be used to arbitrarily shard the data in your application in a way that’s adaptable to capacity changes and resilient to failure.
Ringpop is best described by introducing its 3 core features: a membership protocol, a consistent hash ring and request forwarding.
its membership protocol provides a distributed application, whose instances were once completely unaware of one another, with the ability to discover one another, self-organize and cooperate. The instances communicate over a TCP backchannel and pass information between them in an infection-style manner. Enough information is shared to allow these instances to come to an agreement, or converge, on whom the participating instances, or members, are of the distributed application.
With a consistent membership view, Ringpop arranges the members along a consistent hash ring, divides up the integer keyspace into partitions and assigns ownership of the partitions to the individual instances of your application. It then projects a keyspace of your choosing, say the ID range of the objects in your application, onto that same ring and resolves an owner for each ID. In the face of failure, the underlying membership protocol is resilient and automatically reassigns ownership, also known as rebalancing, to the surviving instances.
./scripts/tick-cluster.js -n 7 -i node ./main.js
Ringpop by Uber
http://highscalability.com/blog/2015/9/14/how-uber-scales-their-real-time-market-platform.html
- The solution for scaling Node was ringpop, a consistent hash ring with a gossip protocol, implementing a scalable, fault-tolerant application-layer sharding.
- In CAP terminology ringpop is an AP system, trading consistency for availability. It’s better to explain away a few inconsistencies than it is to have a down service. It’s better to be up and occasionally make an error.
- ringpop is an embeddable module that’s included in each Node process.
- Node instances gossip around a membership set. Once all the nodes agree who each other are, they can make lookup and forwarding decisions independently and efficiently.
- This is really scalable. Add more processes and more work gets done. It can be used to shard data, or as a distributed locking system, or coordinating a rendezvous point for pub/sub or a long-poll socket.
- The gossip protocol is based on SWIM. A few improvements have been made to improve convergence time.
- A list of members that are up are gossiped around. As more nodes are added it is scalable. The ‘S’ in SWIM is for scalable and really does work. It has scaled to thousands of nodes so far.
- SWIM combines health checks with membership changes as part of the same protocol.
- In a ringpop system there are all these Node processes containing ringpop modules. They gossip around the current membership.
- Externally, if DISCO wants to consume geospatial, every node is equivalent. A random healthy node is selected. Wherever the request lands is responsible for forwarding the request to the right node by using the hash ring lookup. It looks like:
- It may sound crazy to have all these hops and peers talking to each other, but it yields some really nice properties, like services can be scaled by adding instances on any machine.
- ringpop is built on Uber’s own RPC mechanism called TChannel.
- It’s a bidirectional request/response protocol that was inspired by Twitter’sFinagle.
- An important goal was to control performance across a lot of different languages. Especially in Node and Python a lot of the existing RPC mechanisms don’t work very well. Wanted redis level performance. TChannel is already 20 times faster than HTTP.
- Wanted a high performance forwarding path so intermediaries could make forwarding decisions very easily, without having to understand the full payload.
- Wanted proper pipelining so there wasn’t head-of-line blocking, requests and responses could be sent in either direction at any time, and every client is also a server.
- Wanted to bake-in payload checksums and tracing and first class features. Every request should be traceable as it wends its way through the system.
- Wanted a clean migration path off of HTTP. HTTP can be encapsulated very naturally in TChannel.
- Uber is getting out of the HTTP and Json business. Everything is moving to Thrift over TChannel.
- ringpop is doing all its gossip over TChannel based persistent connections. These same persistent connections are used to fanout or forward application traffic. TChannel is also used to talk between services.
• Implements a variation of SWIM
• Flap damping
• Use checksums to verify correctness of ring state
• Proxying capabilities
var ringpop = new RingPop({
app: 'myapp',
hostPort: 'myhost:30000'
});
ringpop.bootstrap(['myhost:30001’, 'myhost2:30000']);
ringpop.on('ready', function() {
// do something
});
var node = ringpop.lookup(‘[unique-request-id]’);
if (node === ringpop.whoami()) {
// process request
} else {
// forward request
}
Ringpop Serial
• Simple ringpop wrapper
• Requests are queued by key
• Processed serially, one at a time
• Emulates transactions
Transactions
• Conflicts are possible during membership changes
• Need smart application level conflict resolution
Sevnup
• Open sourced node.js module
• Ringpop extension
• Key ownership hand-off
• Customizable recovery & release
• Pluggable persistence layer
Reliable Timers
• Node.js offers in-memory timers
• Use sevnup to make them reliable
• Riak as persistence layer
http://highscalability.com/blog/2015/10/12/making-the-case-for-building-scalable-stateful-services-in-t.html
Uber’s Ringpop - Gossip Protocol + Consistent Hashing
- Ringpop is a node.js library implementing application-layer sharding. (more info)
- Uber has the concept of a trip. To start a trip a user orders a car which requires the rider information and location information, data is updated during the trip throughout the ride, and the payment must be processed at the end of the trip.
- It would be inefficient for each of these updates to be load balanced to a different stateless server every time. The data would constantly be persisted to the database and then pulled back in again. This introduces a lot of latency and extra load on the database.
- The design implements routing logic so all the requests for a user can be directed to a single machine.
- The Swim Gossip Protocol is used to maintain cluster membership. It’s an AP cluster membership protocol so it’s not guaranteed to be always correct. Availability was chosen over correctness because it’s more important that a user can always order a car.
- Consistent hashing is used to route work throughout the cluster. This has the hot node problem and the only remedy is to add more capacity, even if other nodes are under utilized.
