All things Storageous in Storage!


Twitter’s Blob store and Libcrunch – how it works

TwitterYou may have read one of my previous posts “Arming the cloud” where I talked about why and how large cloud providers are using commodity hardware with intelligent API’s to separate the dumb data and intelligent data to give us a better service. Well in a world of distributed computing and networking you will probably not find larger than Twitter.

To me and you when we upload a photo to the cloud its in the “cloud” we do not care much for what goes on in the background all we care about is how long is takes to upload or download. And this has been Twitter’s challenge, how do they  keep all this data synchronized around the world to meet our immediate demands? It is a common problem of how do large-scale web and cloud environment’s allow users from anywhere in the world to use the photo sharing service overcoming latency which ultimately boils down to me and you waiting for the service to work.

So Twitter announced a new photo sharing platform, but what I am going to look at is how the company manage software and infrastructure to enable this service. Here is what Twitter released yesterday;

“When a user tweets a photo, we send the photo off to one of a set of Blobstore front-end servers. The front-end understands where a given photo needs to be written, and forwards it on to the servers responsible for actually storing the data. These storage servers, which we call storage nodes, write the photo to a disk and then inform a Metadata store that the image has been written and instruct it to record the information required to retrieve the photo. This Metadata store, which is a non-relational key-value store cluster with automatic multi-DC synchronization capabilities, spans across all of Twitter’s data centers providing a consistent view of the data that is in Blob store.”

Sound familiar to what I was discussing in my previous posts? Of course it is, this is a classic example of commoditizing storage\compute\network hardware and having the software API intelligently manage this data.

So what you have to consider with a platform like Twitter is speed and cost, they want users to be able to see the tweet with the picture as soon as possible but they have to be conscious of cost to deliver this service. Twitter has many data centers with many resources but the trade off is always going to be cost.

The next element of this is reliability, how do Twitter ensure that your photos exist in multiple locations on file but not too many to cost too much to Twitter, it also has to think about how and where it stores information on servers which indicate where the actual file exists (meta data). If we took the servers for example, and then thought about how many photos are uploaded to Twitter each day, that’s a lot of meta data to store, what if one of those servers then fails? Then you would lose all meta data and the service would be unavailable. To remedy this the original way of thinking is to replicate this data, but that is costly and time-consuming to keep synchronized and lets not forget will be using some serious space.

So Twitter introduced a library called “libcrunch” and here is what they had to say about it;

“Libcrunch understands the various data placement rules such as rack-awareness, understands how to replicate the data in way that minimizes risk of data loss while also maximizing the throughput of data recovery, and attempts to minimize the amount of data that needs to be moved upon any change in the cluster topology (such as when nodes are added or removed).”

Does that sound familiar again? This is the Atmos play from EMC which is using intelligent API’s to manage all aspects of an element of data, I referred to this last time as an “Object Store”, and the point of this that the API itself understands what to do with a particular piece of data in terms of replication, security, encryption and protection. So we are no longer administering pools of storage but the API is self managing itself, and in the case of Twitter you have to admit that this would be the only way of doing this.

So what does the infrastructure look like, well they use cheap hard drives to store the actual file and the meta data is served from EFD drives for increased speeds. Think of meta data as a search engine it allows you to find articles related to a query very quickly rather than looking at the entire web.

So to sum this up as we place more and more information in to the cloud which is a blend of distributed compute and network, locating information across them is becoming more difficult and slow. Thinking like this with API’s controlling the data according to policies is the right direction to take when using large cloud services.

If you are interested in looking at a cloud solution platform delivering intelligence like this go to EMC Atmos




Leave a comment


The Cloud is Closer than you think

So the cloud is here, but are you moving with the times or are you behind in your thinking? It’s a question people will never admit to, but the reality is becoming very apparent that SAN and NAS do not scale to large clouds such as Amazon or AT&T. So how do the big guns do cloud?

So lets take a service such as Amazon who have one huge infrastructure which spans global data centers with one huge flexible namespace which can grow with no complexities and minimal management costs. Amazons new offering which is Amazon Glacier released back in September, which is now 1 penny per GB per month! How do you get costs down to that price point and still turn a profit, and to give you some perspective on how big the Amazon infrastructure is this year alone 260 Billion objects were added! Imagine trying to manage that with traditional thinking such as silos of storage in SAN and NAS storage? Amazon’s pricing has dropped 12 times this year alone and they just under cut the market every time.

So lets look at their thinking, one thing that all the big cloud names have in common is that they do not use file systems for their cloud, this includes Facebook, Twitter, E bay, Amazon, You Tube. So why do they not use this, the answer is simple cost and scale. These infrastructures are huge and when they set about creating their clouds they wanted massive scale 10’s of petabytes if not hundreds with minimal growth disruption and management over head.

So lets take a different angle for one moment, it was us the end-user which created this costing point, because technology is so readily available now, for example if you have a credit card you can get a server and some storage from Amazon in a matter of minutes, but they are by no means the only ones doing this. The consumer market is so diverse now that if the price point is not right we just move on it’s as simple as that. How many times have you checked the price of something on the internet while shopping in a store?

So back to the point lets use Amazon, they use an object based API which incorporates, security, encryption, security, billing and a policy engine talking to commodity x86 servers and commodity storage. Hardware fails we accept that as everyone does, but the key is the software at the top layer. Object based storage does not work like traditional file systems and it spans one single namespace meaning you can geographically disperse your data centers and have one giant object store which according to policies set it replicates and protects your data. The simplest way of explaining this is “Drop box”, we all love and use drop box and it may surprise you to know that it too uses Amazons philosophy as above. Policies are the key in this object based world, lets take your free 10GB subscription with Drop Box, as that is a “Free” service it is very unlikely that a copy of your data is made, it is replicated or encrypted and they do not guarantee it will always be there.  But what if I pay the fee per month? Well then you would have a paid policy which would replicate to another data center, encrypt your data and more importantly bill you on usage metrics such as bandwidth and space used.

Now this is the key component here Object storage is subject to policies, an object contains meta data and the content, the intelligent API’s look at the meta data and decide what to do with this data according to policies set. This is key to understanding the management of the cloud. Let’s take E Bay how many photos do you think get uploaded to E Bay every single minute? Well I imagine this number is huge, but how do you manage how long those photos stay there? Before policies E Bay were having to run jobs to delete these photos every night, but there came a point where they were doing this constantly, so with policies in the API they simply set one up to delete these after 3 months has passed. It is as simple as that, all that management has gone and is automated.

The technology that Amazon and E Bay use is EMC Atmos. It is the intelligent API with commodity hardware underneath defining it as a purpose-built cloud platform giving you up to 1.3PB per floor tile. Atmos allows you to easily scale your cloud over geographic distances as it acts as one great big storage pool with one namespace, the API abstraction layer takes care of all the storage calls so developers who are paving the way in browser-based applications which are WAN friendly do not have to care what goes on below the software layer. Atmos takes care of all this, so lets imagine you have 5 data centers globally all connected and your objects are behaving according to your policies and automatically billing your end users based on your policies set (security, replication etc), which you don’t have to back up, Isn’t that the way to do things? Just imagine trying to back up Amazons cloud……………….no thanks.

As the intelligence in an Object store and resilience is also built-in you can lose multiple drives or nodes and your service does not go down, People such as Amazon and E Bay accept that hardware eventually fails, so they just stock pile this and when drives fail they replace them eventually, as it is not critical. Has E Bay ever gone down? The answer is no and there is good reason for this if E Bay went down it would cost E Bay $3,900 per second!

So EMC Atmos is arming the cloud, and the service providers are monetizing this platform in to services that me and you consume every single day. SAN and NAS ways of thinking are fast becoming limiting in the way they can scale in comparison to Object stores and this is now in my personal view why service providers are switching on to this change, traditional service providers are offering things such as “back up to the cloud” etc, and what they need to be doing is appealing to the developers who have written so many of their programs for Amazon S3, as their applications could run on Atmos as it understands S3. This would enable them to keep with the curve in this changing marketplace. And the best part is that this Atmos API is yours, you can edit, modify it, do what ever you like with it to make it work for your company and give a portal to your end users, and bill them accordingly.

So to sign off is Amazon trail blazing the way ahead? No, they have just done this before anyone thought of it, and they are now so large that they can just dynamically grow, and everyone thinks that cloud is slow, but look at Amazon using commodity hardware and servers, the sheer scale means the amount of compute and storage available is all there for the taking!