RaleighFS to enter in the in-memory key-value store market

A couple of guys asked me about RaleighFS, and why is called File-System instead of Database, and the answer is that the project is started back in 2005 as a simple Linux Kernel File-System, to evolve in something different.

Abstract Storage Layer
I like to say that RaleighFS is an Abstract Storage Layer, because is main components are designed to be plugable. For example the namespace can be flat or hierarchical, and the other objects don't feel the difference.

• Store Multiple Objects with different Types (HashTable, SkipList, Tree, Extents, Bitmap, ...)
• Each Object as it's own on-disk format (Log, B*Tree, ...).
• Observable Objects - Get Notified when something change.
• Flexible Namespace & Semantic for Objects.
• Various Plain-Text & Binary Protocol Support (Memcache, ...)

A New Beginning...
Starting weeks ago, I've decided to rewrite and refactor a bit of code, stabilize the API and, this time, trying to bring the file-system and the network layer near to a stable release.
First Steps are:

• Release a functional network layer as soon as  I can.
• Providing a pluggable protocol interface.
• Implement a memcache capable and other  protocols.

So, these first steps are all about networking, and unfortunately, this means dropping the sync part and keep just the the in-memory code (the file-system flush on memory pressure).

Current Status:
Starting from today, some code is available on github under raleighfs project.

• src/zcl contains the abstraction classes and some tool that is used by every piece of code.
• src/raleighfs-core contains the file-system core module.
• src-raleighfs-plugins contains all the file-system's pluggable objects and semantics layers.
• src/raleigh-server currently contains the entry point to run a memcache compatible (memccapable text protocol), and a redis get/set interface server. The in-memory storage is relegated in engine.{h,c} and is currently based on a Chained HashTable or a Skip List or a Binary Tree.

How it Works
As I said before the entry point is the ioloop, that allows clients to interactot through a specified protocol with the file-system's objects. Each "protocol handler" parse it's own format, convert it to the file-system one, and enqueue the request to a RequestQ that dispatch the request to the file-system. When the file-system has the answer push the response into the RequestQ and the client is notified. The inverse process is applied the file-system protocol is parsed and converted into the client one.

Having the RequestQ has a couple advantages, the first one is that you can wrap easily a protocol to communicate with the filesystem, the other one is that the RequestQ can dispatch the request to different servers. Another advantage is that the RequestQ can operate as a Read-Write Lock for each object allowing the file-system to have less lock...

For more information ask me at theo.bertozzi (at) gmail.com.

Python: NetStack/CoreAsync

Today I've added to my GitHub Respositories NetStack/CoreAsync a python "package" (It's much more a bunch of utility classes) that allows you to code in async/parallel way, that I use to build my networking apps.

def concurrent_func(text):
    for i in range(5):
        print text, 'STEP', i
        yield

coreasync.dispatch_concurrent(lambda: concurrent_func("Context 1"))
coreasync.dispatch_concurrent(lambda: concurrent_func("Context 2"))
coreasync.runloop()

Package contains a small Async HTTP Server implementation that you can easily use:

def handle_response(socket, addr, request, headers, body):
   yield socket.send(...)

def handle_error(socket, addr, error):
   yield socket.send(...)

coreasync.httpserver.httpServerLoop(HOST, PORT, handle_response, handle_error)
print 'HTTP Server is Running on', HOST, PORT
coreasync.runloop()

You can find Source Code and Examples at GitHub:

git clone http://github.com/matteobertozzi/netstack-coreasync.git

[TIP] Generic Binary Format

I Love use Binary formats, instead of XML, and JSON. Here is my Generic Binary Format for data transmissions or serializations. Data is composed by three blocks. The first one 1byte that describe all information about the object, like "is a single Int object" or "is a list", then tells you the second block length. The second block contains the size of the third block (The Data-Block) or the Number of Items in List.

Unified Notification Service: Avoid The Wheel Reinvention

Every programmer loves to reinvent the wheel, and reinventing the wheel is still my primary hobby. Sometimes you need to reimplement a Network protocol to use with your favorite language/library, sometimes is only for fun, but if you're in the Business World maybe is "better" (faster) to use one of the thousand existing libraries.

In the most cases you need to reimplement a Protocol to embed it in your application, and sometimes you have to reimplement two/three protocols that does the same job like IM Protocols (XMPP, AIM, Yahoo...).

A better solution, that avoid you to reinvent the wheel is to use an existent library to handle the protocol(s) that you need, and build an Abstract Interface, with your data format, that allows you to use a generic way to communicate between various provider. Below you can find a graphical example of what I mean.

You can have many providers, written in different languages. These providers talk with the Notification Service providing an abstract interface for the Apps. In this way, the end App has just to say "Write a Mail To X", "Download Todays Mail", "Send an IM to X"... and you can intercept notification to displays as a popup on your desktop... or something similar.

This solution will be used in MokoTouch Project, to provide Core Services to the Apps. For more information send me a mail.

A Bit of Distributed Computation...

In the last months I've worked on various 2D rendering projects, that requires lots of row power to be executed in smallest time as possible.

The rendering result is an aggregation of components (or better, Group of Components) that can be rendered independently of each other in a  process because each components has its own input data and until the aggregation process starts there're no dependencies between components. In a few words, foreach input data I've to call a Render method that returns the "computed" data, that at the end will be aggregated with all the computed values.

So, how speed-up the rendering? The standard answer is using Threads, to take advantage of multi-core system. But I need lots row power and lots of core to be fast enought. Another solution that doesn't require to spent a lot of money for a faster computer it to distribute the computation across different machines. And this is my attempt...

The Master receive an array of elements as input and splits up it according to the number of the available Nodes (Machines), and foreach node assigns its sub-array and sends an "informative message" to the node.

The Node waits for an "informative message" and when it has received it, starts its computation. Foreach Data Item that fetch runs the Computation and sends back the result.

The Master decompose data into equal-size partitions, so each node has  an equal-size queue to process, but if someone finishes its job and there're more data to process (in someone else queue) The Master dequeue couple of items from the slower node queue and add to the queue of the one that has finished its job. In this way you've done your computation faster then ever, and if one machine crashes (slow one case) its job will be taken by someone else that has finished its own.

And this is just a bit of General Theory, but the implementation is really simple. Maybe I'll try to reimplement something more generic in the near future (October, November) when I'll have a bit more of free time. If you've any question send me a mail!

Reliable Protocol on Unreliable Channel

When you work with Unreliable communication channel (like UDP), you need to implement your own "Reliable Mechanism" to be sure that packet that you've send was received correctly. Here there's my attempt.

Reliable Transmitter:

The schema below shows the operations that reliable transmitter has to do. "Red" are "error or unexpected operations", Green are "expected operations", Blue are TX Operations.

Transmitter send a packet and waits for the ACK packet (sended by receiver). If ACK doesn't arrive before the Tx Timeout packet is resended. When ACK arrive Tx send the CACK (Confirm ACK) packet and waits for a timeout, if another ACK Arrive TX resend the CACK, this because receiver doesn't have the confirm of transmitter. When TX timeout all is done, and the packet is sended correctly.

Reliable Receiver:

The schema below shows the operations that reliable receiver has to do. "Red" are "error or unexpected operations", Green are "expected operations", Blue are Rx Operations.

When RX receive a Packet sends an ACK, and waits for the CACK, if CACK doesn't arrive before timeout RX resend the ACK. When CACK arrive all is done. We've received the packet and the transmitter knows that we've received it.

OpenSSL: SSL Client/Server Example

I'm experimenting a bit with the new iPhone 3.0 SDK, Core Data, Game Kit, Push Notification Service and so on...

I'm using C, so today a little Example of SSL Client/Server written in C using OpenSSL, I've written a small wrapper for SSL Socket, and here is How to use it.

/* SERVER CODE 
 * ==============================
 */
SFSocketGlobalInit();       /* Initialize SSL */

/* Alloc Socket, Initialize SSL and Listen */
SFSocket *socket = SFSocketAlloc();
SFSocketInit(socket, CA_FILE, DH_FILE, KEY_FILE, KEY_PASSWORD, NULL);
SFSocketListen(socket, INADDR_ANY, PORT);

do {
   SFSocket *clientSocket;
   char buffer[64];
   int rdSize;

   /* Accept Client Connection */
   if ((clientSocket = SFSocketAccept(socket)) == NULL)
       break;        

   /* Read Data from Client */
   if ((rdSize = SFSocketRead(clientSocket, buffer, 64 - 1)) > 0) {
       buffer[rdSize] = '\0';
       printf("Client: %s\n", buffer);
   }

   /* Write to Client */
   strcpy(buffer, "Hello Client!");
   SFSocketWrite(clientSocket, buffer, strlen(buffer));

   /* Disconnect Client */
   SFSocketRelease(clientSocket);        
} while (1);

/* Close and Release Socket Resources */
SFSocketRelease(socket);

Above you've the simplified server code (without error check!) and below you've the client code. The client try to connects to server, send an "Hello" message and the server reply with other greetings.

/* CLIENT CODE 
 * ==============================
 */
SFSocketGlobalInit();       /* Initialize SSL */

/* Alloc Socket, Initialize SSL */
SFSocket *socket = SFSocketAlloc();
SFSocketInit(socket, CA_FILE, NULL, KEY_FILE, KEY_PASSWORD, NULL);

/* Connect to Host */
SFSocketConnectToHost(socket, HOSTNAME, PORT);

/* Send Message to Server */
char buffer[64];
strcpy(buffer, "Hello from Client!");
SFSocketWrite(socket, buffer, strlen(buffer));

/* Read Message from Server */
if ((rdSize = SFSocketRead(socket, buffer, 64 - 1)) > 0) {
   buffer[rdSize] = '\0';
   printf("Server: %s\n", buffer);
}

/* Close and Release Socket Resources */
SFSocketRelease(socket);

Remember that you need to generate, at least, the Authority Certificate, Server Certificate and Clients Certificates. and here is How to do it.

- AUTHORITY Certificate:
 openssl genrsa -des3 -out ca.key 1024
 openssl req -new -x509 -key ca.key -out ca.crt

- SERVER Certificate
 openssl genrsa -des3 -out server.key 1024
 openssl req -new -key server.key -out server.csr
 openssl x509 -req -in server.csr -out server.crt -sha1 \
                   -CA ca.crt -CAkey ca.key -CAcreateserial 

- CLIENT Certificate
 openssl genrsa -des3 -out client.key 1024
 openssl req -new -key client.key -out client.csr
 openssl x509 -req -in client.csr -out client.crt -sha1 \
                   -CA ca.crt -CAkey ca.key -CAcreateserial

The Full Source Code is Available Here: SSL Client/Server Example Source Code.