All things Storageous in Storage!

Leave a comment

What is the Social Customer?

I found the below a really insightful presentation on the social customer. The social customer is not social media but rather understanding customers as social beings.

This presentation may help you to understand how younger generations network and interact.

Insightful stuff and a great presentation, the word social media is becoming immensely over used and the true nature of being connected online all the time means that offline interaction is sparse and this is the most important element.



Twitter Under Attack

What started as a social media tool can now impact stock markets

What started as a social media tool can now impact stock markets

Never underestimate the determination of someone who is time-rich and cash-poor. Hackers in this case are experimenting with technology and pushing it in ways people have never imagined, bringing the issue to closer to home is the example below.

Lets take Twitter this week in which the associated press Twitter account was hacked, and the group responsible tweeted that there had been explosions in the White House and the President was injured. This must send out alarm bells to every major company that uses Twitter as such this Tweet was read by thousands instantly spreading like a rumour  being retweeted and posted over the internet even causing the Dow Jones stock market to slip 1%. The AP account came out shortly after announcing that this was a false Tweet and the account suspended, but it just shows how impactful a single Tweet can be.

This just shows how important Twitter has become, Twitter is now used in Business for all different purposes, for example Bloomberg now have Twitter feeds in their terminals to monitor market activity and chatter relating to their field of expertise, and they base market decisions not solely on this information set but it has an influence that is for certain. Businesses all over the world see Twitter as not only a social tool but a business tool, and as such they must not under estimate the people who are hacking these accounts, the reputation able damage that can be done to a company now is huge.

Stocks fell 1% due to one Tweet

Stocks fell 1% due to one Tweet

I went to a presentation by Peter Hinsenn only 12 months ago and listened to him speak about the importance and value of Twitter as a tool for businesses. True it is a great tool with audiences that are huge but the risk is now also becoming a concern. When you consider what you need to login, a UN and PW it is not difficult to imagine this could be broken. I have had my social media accounts hacked but they are personal and hold no key information on there, the most damage they did was upset my friends with dodgy links on their walls.

Twitter needs to incorporate a two factor authentication so the UN and PW entered can be verified, such as a text coming to you with a code in so you can verify to the server that you are the owner of this account, Microsoft, Google and Apple all now offer this but it seems the one behind the curve is arguably the most influential.

Security is a buzz word in the industry at the moment and everyone has different views on what security entails, especially with how they apply security in their business. I have said this before and I will say it again, if you understand security in this industry and can articulate it in a way that people get you will be hugely successful, its almost a grey area that no one wants to get involved in and why is that? Because it is not easy and is very complex! Twitter are actively recruiting now for senior security people to re vamp their security, the  new generation of cyber criminal needs a new generation of cyber police and Twitter are starting to get this.

If I were involved or responsible for a large companies security I would be looking at the above and giving this some serious thought along the lines of where are the lines of my security? and the answer to that is there are no strict lines of defence anymore, the internet and social media are accessible by all, the slightest leak in your ship and a flood will be immanent. I have been to a lot of RSA sessions and also security centres and the ability they have to monitor, track and analyse security is quite incredible and in my view if people are not looking to these measures or similar it is a big risk.

On another note you have to also consider how quickly this Tweet was revoked, do you think that this was just chance, someone casually browsing Twitter that happened to stumble upon this? Think again, major organisations have social media monitoring and analytics used for varieties of purposes, from customer satisfaction to rebuffing negative comments about their respective business on the internet. For example it was made clear through various blogs and posts about the Obama Office using data analytics to monitor peoples thoughts, posts and counter argue on forums.

That’s another story for another day but my point being here, look at what has been done with a single Tweet and some invested time resulting in huge impact. Security may be a subject on most IT professionals minds but it extends much further and should be looked at very seriously.


Twitter’s Blob store and Libcrunch – how it works

TwitterYou may have read one of my previous posts “Arming the cloud” where I talked about why and how large cloud providers are using commodity hardware with intelligent API’s to separate the dumb data and intelligent data to give us a better service. Well in a world of distributed computing and networking you will probably not find larger than Twitter.

To me and you when we upload a photo to the cloud its in the “cloud” we do not care much for what goes on in the background all we care about is how long is takes to upload or download. And this has been Twitter’s challenge, how do they  keep all this data synchronized around the world to meet our immediate demands? It is a common problem of how do large-scale web and cloud environment’s allow users from anywhere in the world to use the photo sharing service overcoming latency which ultimately boils down to me and you waiting for the service to work.

So Twitter announced a new photo sharing platform, but what I am going to look at is how the company manage software and infrastructure to enable this service. Here is what Twitter released yesterday;

“When a user tweets a photo, we send the photo off to one of a set of Blobstore front-end servers. The front-end understands where a given photo needs to be written, and forwards it on to the servers responsible for actually storing the data. These storage servers, which we call storage nodes, write the photo to a disk and then inform a Metadata store that the image has been written and instruct it to record the information required to retrieve the photo. This Metadata store, which is a non-relational key-value store cluster with automatic multi-DC synchronization capabilities, spans across all of Twitter’s data centers providing a consistent view of the data that is in Blob store.”

Sound familiar to what I was discussing in my previous posts? Of course it is, this is a classic example of commoditizing storage\compute\network hardware and having the software API intelligently manage this data.

So what you have to consider with a platform like Twitter is speed and cost, they want users to be able to see the tweet with the picture as soon as possible but they have to be conscious of cost to deliver this service. Twitter has many data centers with many resources but the trade off is always going to be cost.

The next element of this is reliability, how do Twitter ensure that your photos exist in multiple locations on file but not too many to cost too much to Twitter, it also has to think about how and where it stores information on servers which indicate where the actual file exists (meta data). If we took the servers for example, and then thought about how many photos are uploaded to Twitter each day, that’s a lot of meta data to store, what if one of those servers then fails? Then you would lose all meta data and the service would be unavailable. To remedy this the original way of thinking is to replicate this data, but that is costly and time-consuming to keep synchronized and lets not forget will be using some serious space.

So Twitter introduced a library called “libcrunch” and here is what they had to say about it;

“Libcrunch understands the various data placement rules such as rack-awareness, understands how to replicate the data in way that minimizes risk of data loss while also maximizing the throughput of data recovery, and attempts to minimize the amount of data that needs to be moved upon any change in the cluster topology (such as when nodes are added or removed).”

Does that sound familiar again? This is the Atmos play from EMC which is using intelligent API’s to manage all aspects of an element of data, I referred to this last time as an “Object Store”, and the point of this that the API itself understands what to do with a particular piece of data in terms of replication, security, encryption and protection. So we are no longer administering pools of storage but the API is self managing itself, and in the case of Twitter you have to admit that this would be the only way of doing this.

So what does the infrastructure look like, well they use cheap hard drives to store the actual file and the meta data is served from EFD drives for increased speeds. Think of meta data as a search engine it allows you to find articles related to a query very quickly rather than looking at the entire web.

So to sum this up as we place more and more information in to the cloud which is a blend of distributed compute and network, locating information across them is becoming more difficult and slow. Thinking like this with API’s controlling the data according to policies is the right direction to take when using large cloud services.

If you are interested in looking at a cloud solution platform delivering intelligence like this go to EMC Atmos