Affichage des articles dont le libellé est nosql. Afficher tous les articles
Affichage des articles dont le libellé est nosql. Afficher tous les articles

lundi, 7 mars 2011

NoSQL Overview @Webmardi

Tuesday 1st of March 2011 I gave an overview of NoSQL at the Webmardi monthly event. The presentation was hosted by Webdoc. Thanks all for the warm welcome !

Some pictures :




The slides are available on Slideshare : NoSQL Overview, implementation free.

It is called "Implementation free" because I didn't want to focus on a particular implementation of a NoSQL datasore, but rather generalize concepts of "What is NoSQL". And it's a tricky answer to formulate :)

So what is NoSQL ? It's a family of datastores that does not have SQL as main data access pattern.

The presentation was hosted by Webdoc : http://www.webdoc.com.

jeudi, 6 janvier 2011

Eventually Consistency demystified

In my crusade into the NoSQL world, Eventually Consistency is everywhere. I want to demystify this property a little bit here.

But let's begin with an example to have the same base for the discussion :

  • Let "Node1", "Node2" and "Node3" be three nodes (servers) that are part of our distributed datastore.
  • Let "User A", "User B", "User c" be three users wanting to read and write data in our fictive distributed datastore.
At time (1), "User A" write the value "A" to "Node1". "Node1" will replicate asynchronously this value to both "Node2" and "Node3" (specific to my example).
At time (2) the write call of "Node A" returns. But the replication of value "A" hasn't been completely propagate to "Node2" and "Node3".
At time (3), "User B" and "User C" will read value "A" from "Node1" and "Node2" respectively. "User B" got the latest value (because it reads the node which initiate the update), "User C" will read either the old or the new version of "A", but without any guarantee regarding what it will read.



In a future time (5), "User B" and "User C" re-read value "A" and then got the same value. At this point of time, the datastore is consistent.

Immediate Consistency

In a Immediate Consistency, opposing Eventually Consistency, the write call from "User A" should wait till the replication is done on other nodes before returning, and replica nodes ("Node2" and "Node3") should be synchronized to expose the new value at the same time.

Moreover, if "Node1" is unable to talk to "Node2", the write replication will probably fail then the write call from "User A" will fail.

As we can notice, Immediate Consistency is hard to scale (see two-phase commit or paxos algorithm), because it increases the latency of the writes and makes the system not redundant to failure.

Trade-off for scaling writes

Eventually Consistency is then a trade-off for scaling writes that seems reasonable in certain use-cases.

dimanche, 25 avril 2010

Implementing unique constraints with Cassandra

I was talking about my "Doodle clone with Cassandra backend" in a previous post, I want to explain a little bit how I implement 2 kinds of constraints with Cassandra.

Unique name constrain

When subscribing to a poll, a unique constraint on the name of the subscriber is applied. There is a first check in Javascript in the GUI, but there is also a check of the uniqueness of the names on the server side.

Before going deeper, I just need to say a word about the structure used :

To store the subscribers, I use a SuperColumnFamily, with the ID of the poll as key. For a poll having key "1234", it gives me something like :

"1234" => {
"subscriberKey1" => { name = "subscriberName1", options = ... }
"subscriberKey2" => { name = "subscriberName2", options = ... }
...
}
The trick about the subscriberKey* is to generate a unique ID based on a timestamp in order to be sorted by Cassandra.
The key will be something like : currentTimestamp - ClusterNodeId - ThreadId. Adding the ThreadId into the key give me the guarantee that using the System.currentTimeMillis() will be unique, because the processing will unfortunately take more than a millisecond :)

Using this key, I will write the subscriber row with ConsistencyLevel.QUORUM, and then I will read all the rows till the one I have inserted (with ConsistencyLevel.QUORUM again).
At this point I use Cassandra magic (rows are ordered) to be sure that I have selected all the subscribers they subscribe before the current one.
I remains me to check manually the uniqueness of the name. If another row has the same name, I will simply remove the current subscriber.

Limiting the number of subscribers per option

Another constraint I have implemented is the limitation of the number of subscribers per option.

The principle is the same as the unique name constraint : I count the number of previously inserted subscriber.
One point to keep in mind is that we can read a row with a duplicated name, which will be deleted later so it should not be taken into account for the actual constraint. This mean that we need to check if we are not counting duplicated names in this constraint too...

But the good news about this constraint is that it show a way to implement counters in Cassandra. Maybe I will write another post about this topic. The bad news is that we need to check the duplicated names in several places.

Limitation of the model

Complexity : adding more constraints grows the complexity in O(n^2), as every new constraints (duplicated names) should be implemented into every other constraints (limitation of subscribers per option).

Scalability : this model will scale very well for billions of polls with some tens of users. The inverse, some tens of polls with billions of users, needs a lot of resources for checking constraint, which will make the performance drop.

samedi, 24 avril 2010

Doodle clone with Cassandra backend

I follow Cassandra's activity for some months now, and I decided to try it. But what better than developing a small application to test real needs ?

That's what I did with my "Doodle clone with Cassandra backend".

What is Cassandra ?

According to the main page,

Cassandra is a highly scalable, eventually consistent, distributed, structured key-value store
Why a Doodle clone ?

I'm working during my free time on an event manager software, so it was a good pretext to analyze Doodle business :)
It appears to be quite simple, so it was a good starting point.

Which features to implement ?

I added the following feature to the doodle clone :
  • Easy (easier) to create polls
  • Login free, secret key based administration
  • No options limitation
  • Email feedback when someone subscribe the poll (if email provided)
  • No duplicate name of the subscribers
  • An optional limitation of the number of subscribers per options
Have a look !

Go to http://doodle.noisette.ch and try it, the result is really surprising.

Disclaimer

There is absolutely no affiliation with Doodle, neither any idea to become the next Doodle. The goal of this application is purely academic, and it is deployed as is, without any guarantee.

lundi, 5 avril 2010

Don't be abused by the term "commodity hardware"

NoSQL paradigm claims to build clusters of "commodity hardware". But don't be abused by this term : "commodity hardware" is not your living room's computer. Commodity hardware is two slots' computer, 16+GB RAM, (ten)giga ethernet, raid5 hard disks.


Commodity hardware is not 500$ computers. It's 5000$ computers. But building a 10 nodes' cluster for 50'000$ is still really cheap.

And remeber : clusters' primary bottlneck is I/O. Both disk and network. So there is no place for virtualisation here.