Nexenta Blog

Global Leader in Software Defined Storage

CCOW-Negotiating Rendezvous Groups

By

CCOWTM Replicast is a storage protocol that uses multicast communications to determine which storage servers will store content and then retrieve it for a consumer. It also allows content to be accepted/delivered/replicated using multicast communications. Content can be placed once and received concurrently on multiple storage servers. Replicast can also scale to very large clusters and can support multiple sites, and each site can be as large as the networking elements will allow.

In order to understand how Replicast works, you must first understand how it uses Multicast addressing. Specifically, how the role of the Negotiating Group and Rendezvous Group differs from Consistent Hashing algorithms which are the normal solution for distributed storage systems.

How Conventional Object Storage Systems use Consistent Hashing

The Object name, sometimes referred to as the payload of a chunk, is used to calculate a Hash ID. This ID is then mapped to an aggregate container for multiple objects/chunks (for OpenStack Swift these are called “Swift Partitions” and for CEPH they are called “Placement Groups”). Although the quality of the hashing algorithm can vary, the content of a chunk has to map to a set of storage servers that is based on an Object Name in order to achieve a consistent hash algorithm. If you start with the same set of storage servers, the same content will always map to the same storage servers.

Promoters of Consistent Hashing make the point that Consistent Hashing limits the amount of content that must be moved when a set of storage servers change. If there is a 1% change in the cluster membership then 1% of the content must be relocated. In the long run, you actually want 1% of the content to move to the new servers. Should 1% of the content be lost, you will want to create new replicas of the lost 1% on other servers anyway.

Where CCOW Replicast differs is that it can be far more flexible about when that replication occurs and more selective as to which data is replicated. Replicast has a different method of assigning locations. These more efficiently deal with evolving cluster membership to achieve far higher utilization of cluster resources when the membership isn’t changing.

Negotiating Group

CCOW Replicast uses a “Negotiating Group” to effectively support the chunks “location”. An object name still yields a Name Hash ID (using the Name of the Named Manifest) but that Hash ID maps to a Negotiating Group. When a Manifest references a Chunk, it is found by mapping its Chunk ID (which is the Content Hash ID of the Chunk) to a Negotiating Group.

The Negotiating Group will be larger than the set of servers that would have been assigned by Consistent Hashing. Typically ten to twenty members of the Negotiating Group is preferred. The key is that the client, or more typically the Putget Broker on the client’s behalf, uses multicast messaging to communicate with the entire Negotiating Group at the same time. Effectively the Putget Broker asks “Hey you guys in Group X, I need three of you to store this Chunk”. A Negotiation then occurs amongst the members of the Negotiating Group to determine which three (or more) of members will accept the Chunk, when and at what bandwidth.

“Negotiating” sounds complex but the required number of message exchanges is actually the same as any TCP/IP connection setup. So the Negotiating Group can determine where the Chunk will be stored and with the same number of network interactions as Swift requires for the first TCP/IP connection. For the default replication count of three, Swift requires three connections to be setup.

More importantly, a consistent hashing algorithm (such as Swift uses) will always pick the same storage servers. This is independent of the workload of these servers. Many consider this as the price of eliminating the need for a central metadata server.

With Consistent Hashing, the 3 servers with the lightest workload are selected out of 3 storage servers (assuming the replication count is 3). Of course that also means you are also selecting the 3 busiest servers. With CCOW Replicast you select the 3 servers with the least workload from all the available servers.

Implications of Dynamically Selection

With dynamic load-sensitive selection, CCOW Replicast enables you to a) run your cluster at higher performance levels than Consistent Hashing would allow, and b) still have lower latency.

A well balanced storage cluster will at peak usage want individual storage servers to be loaded only 50% of the time. If they are heavily loaded less than 50%  of the time then the cluster could accommodate heavier peak traffic and you have overspent on your cluster. If they are loaded more than 50% of the time then some requests will be much delayed and your users could start complaining. Should the chance of a randomly selected storage server being busy is 50%, what are the chances that all 3 randomly selected storage servers will not already be working on at least one request.

When it is time to retrieve a Chunk, the client/putget broker does not need to know what servers were selected. It merely sends a request to the Negotiating Group. The negotiating group picks one of its members with the desired chunk and the rendezvous is scheduled to transfer the data.

Rendezvous Group

While the Negotiating Group plans a transfer, it is executed by the Rendezvous Group. The Rendezvous Group implements Replicast’s most obvious feature: Send Once, Receive Many times. Transfers sent via the Rendezvous Group are efficient not only because they only need to be sent once, but also because all Rendezvous Transfers are using reserved bandwidth which they can start at full speed. There is no need for a TCP/IP ramp up.

An important aspect of Rendezvous Groups is that they are easily understood and verified with a known relationship to the membership in the Negotiating Group:

  • Put transactions – every member of a Rendezvous Group is a member of the Negotiating Group with a planned Rendezvous Transfer or a slave-drive under active control of a member
  • Get transactions – the principle member of the Rendezvous Group is the client or Putget Broker that initiated the get transaction. Additional storage servers could have also been part of a put transaction. These additional targets are piggy-backing creation of extra replicas on the delivery that the client required anyway.

Cheers – It’s been a blast! – From Evan Powell

By

After six years I’m leaving Nexenta.

I could not be prouder of what we’ve built at Nexenta.  We took an idea that at the time was radical – let’s bring openness right to the foundation of IT, to the storage itself.  And we pulled it off.

Along the way I learned a lot including:

  • Team, team, team – the best team beats the brilliant individual every time
  • ZFS is great – and not perfect; thank you Sun for ZFS and Solaris (now Illumos).  It says a lot about the requirements of storage that even Solaris, arguably the 2nd most deployed OS for mission critical enterprise environments was not fully mature for storage. We’ve added a lot of fixes to Illumos over the last few years.
  • And much more that I’ll just call “experience.” Much of what you learn when you start and build companies ends up sounding like common sense; for example, what kind of executive is right for what stage of a company or how do you make money on open source?

So, what am I up to now?  A few answers:

  • Continuing to spend time as an EIR with xSeed.  Over the last several months I’ve beenblessed to meet many start-ups while an EIR at xSeed.  xSeed is an unusual seed fund.  They are deep in enterprise and have arguably the best ties of any early stage enterprise focused fund into Stanford and Berkeley.  They are old school in that they are definitely NOT spray and pray (which can be a great strategy actually).  Instead, they seek to go deep and to understand a domain and to find within that domain the very best entrepreneurs; they then surround that entrepreneur with coaches and advisors and much more than just capital.  In addition to meeting many companies I’ve been privy to many mind expanding discussions at xSeed about companies, and domains – about everything from the chess of strategy to the tough soft stuff of finding, retaining and sometimes transforming teams.  So I’ll be spending a bit more time with my colleagues at xSeed.
  • Finally, the rumors are true – I’m helping found another company.  We are in stealth mode.  I can tell you it is NOT in storage (although storage infects everything in IT).  And that it IS still very much on the side of openness and open source.  As you might imagine, I did a huge amount of diligence and founder dating before deciding to bet the next X years of my life on the opportunity.  So it is out of the frying pan and into the fire for me.

Here’s what I’d like you to do:

  • Stay in touch.  Maybe I’m getting to be sentimental in my middle age, but the people I’ve met as Nexenta has grown up are truly extraordinary.  You know who you are. You are the early channel partners who never shied away from offering “constructive feedback.”  You complain too much, but I love you.  And likewise you are the team members that parked your personal lives for weeks, months and years at a time as we fought the good fight.  Despite many missteps and the joy of starting a company during the worst financial collapse since the 1930s, you stayed true to a vision of more open storage and of what we now call software defined storage.  And maybe most importantly – you are the thousands of customers that bet your company’s data on Nexenta and sometimes bet your jobs and careers as well.  Follow me and, yes, complain to me at @epowell101.
  • Insist on excellence.  Like many entrepreneurs I think I start companies because half assed solutions to important problems make me sick (plus I probably have a complicated relationship with authority).  At a fundamental level – at the level of a life’s mission – I am dedicated to finding waste and calcification and cracking through it somehow.  How many of the world’s problems would be less serious if we all talked less and fixed stuff more?
  • Insist on openness and transparency.   Go open, it is the future, and it will make your company a smarter, more competitive business; don’t cop out and defer the decision onto the next guy in your job.  Eventually people will be fired for buying vendor X’s product; don’t be the last buyer to abandon ship when a legacy vendor goes aground.  Also, please be open within your organization as well.  Collectively we are all smarter than any one of us; by remaining open, we’ll get better technology, better team work, and happier lives.

I guess I did get sentimental.

Thanks for tolerating that sentimentality and for reading this blog and thanks everybody for helping a crazy vision come true.  Stay tuned here or via @epowell101 on twitter.

-Evan Powell

How Does Software Defined Data Storage Equate to Savings?

By

Ask any CIO what their greatest concern is, and they’ll invariably come up with some variation of concern over the budget. It’s a Catch-22 for many businesses when it comes to technology: There is always a faster, more reliable option, but it always comes at a cost. So how did the big guys get to the top? How have they learned to strike a balance between cost and effectiveness without compromising either entirely? When it comes to data, more and more have chosen to look at software defined data centers. Here’s why:

There’s no question that data is growing exponentially as we frequently use technology for every aspect of our lives – from shopping and paying bills to reconnecting with friends on social media. All of these things produce data – a lot of data in fact. Studies have estimated that we create 2.5 quintillion bytes of data each day. All of this data needs to be stored, and it can be quite costly. Some organizations are spending as must as 40% of their IT budget on storage solutions.

Therefore, government agencies, credit card companies, health care facilities, social media sites, retailors and many other entities are constantly looking for ways to store this data efficiently and affordably. Software Defined Data Centers (SDDC) address all of these concerns and more because they take the virtues of virtualization and apply them to data storage.  In fact, our customers have reported that they have saved as much as 75% in storage costs.

Here are just a few of the ways businesses can save using software defined data storage:

Scalability: Did you know that 90% of the data out there has been created in the last two years? Just think about what this could mean in terms of data storage ten years from now. As data grows, more and more hardware must be purchased to keep up with demand. Software defined data storage solutions are more easily scalable, with greater savings.

Operational Cost Savings: We talked about scalability, but what about the problems that traditional storage methods present in regard to operational costs? Energy costs to cool the storage system, labor costs to monitor and troubleshoot the system, and maintenance costs are just some things that are negated through a software defined storage system.

Open Source Opportunities: Hardware storage systems are closed systems. You are bound by specific vendors, which could limit your flexibility and opportunities to adapt when needed. Because software defined data storage solutions are often open-source, you can take advantage of the latest technologies and adapt your storage solution for the lowest cost.

For more information about the benefits of software defined data storage solutions, contact us today.

VMware View Acceleration on Display

By

As VMworld Europe 2013 approaches, Nexenta continues to drive home the advancements in application-centric storage with Nexenta VSA for VMware Horizon View.  As the demo for the show is completed and desktop pools are created and tested, it is always exciting to see some independent test done and presented.  Just ahead of VMworld, VMware’s EUC team posted a blog detailing the results of their testing of Nexenta’s VSA for VMware Horizon View .  With a 72% increase in desktop density and a 38X reduction in physical SAN traffic, VMware found VSA for VMware Horizon View to be key to a successful VDI rollout.  These performance statistics are not just for show.  The reduction in traffic and increased density does not just help the balance sheet, but it can help stalled deployments move forward.

“With VSA for Horizon View, Nexenta has introduced an amazing product that unlocks outstanding user experience at a low TCO and makes it possible to recover stalled deployments without requiring a disruptive and painful rip and replace scenario.”  -John Dodge, VMware

If you would like to see this technology in action for deployments, or performance metrics, or the acceleration it provides, make sure to come by the Nexenta booth (Hall 8, S-300) at VMworld Europe.

Nexenta Systems-Powered Storage Solution Achieves 1.6 Million IOPS

By

Nexenta has achieved 1.6 million IOPS (Input/Output Operations per Second) and high-availability with no single point of failure. Comparable solutions from proprietary vendors cost significantly more than the Nexenta and Area Data Systems solution and cannot guarantee high-availability. With the combination of Nexenta’s Software-defined Storage, NexentaStor™, and high-performance, all-flash hardware, there is now a clear enterprise-class alternative to meet the scalability demands of big data.

“Our customers can now reach well over one million IOPS and capitalize on big data opportunities without breaking the bank on proprietary storage technologies that cost hundreds of thousands of dollars,” said Bridget Warwick, Chief Marketing Officer, Nexenta Systems. “This is further proof that Nexenta’s Software-defined Storage is changing the economics of the enterprise storage market.”

Nexenta is demonstrating the 1.6 million IOPS storage configuration at Intel® Solutions Summit 2013 from March 19-21, 2013 in Los Angeles, Calif. Nexenta is a Silver Sponsor and will be at its booth in the storage zone to discuss the enormous opportunity for Intel channel partners to drive ideal storage solutions, powered by Nexenta, to their customers.Architecture recipes using Nexenta and Intel products are listed on Intel’s website at: http://www.esaa-members.com/recipes/advSearchList/182.