Things I'll Forget In A Month Tech scratchpad and musings

7Dec/110

P2Proto: a Java RPC library for P2P applications built on protocol buffers

Yet another coding project I've been working on - this one is an RPC library for Java applications using protocol buffers. This started a few years back when I had a peer-to-peer app using Java RMI that needed to handle a large volume (hundreds) of simultaneous connections and calls, and the app was getting crushed under the load. Another problem was that RMI doesn't really have asynchronous method calls, which was creating an additional performance bottleneck due to the massive numbers of thread required.

To deal with these issues, I switched to using Google's Protocol Buffers, which are compact and efficient, but don't have any decent built-in RPC capabilities. Thus, I set about rolling my own RPC library for protocol buffers. I used it a number of times for distributed programs over the years, and each time improved it somewhat as the project dictated. For my most recent project, I felt that it finally reached a state that could be classified as a usable library, so I added some decent documentation and cleaned everything up to release.

The library itself handles a number of functions not provided by protocol buffers themselves:

  • Delimiting and extracting protocol buffer messages from a persistent network stream.
  • Setting up two-way message channels over which to exchange protocol buffer messages.
  • Associating query and response messages, independent of message arrival order (i.e., there's no need to manually determine which query message a response corresponds to).
  • Convenient handlers for the various types of RPC errors that can occur (local send errors, remote processing errors, and communication timeout errors).

This is all done without any synchronization (except for when multiple threads are using a single connection) or busy waiting, so the library overhead should be low. Protocol buffer messages themselves are extremely compact, so the amount of network data overhead should also be minimal.

As usual, here are the download links: the main project page, the complete library source, the library jarfile only, and finally, the online Javadoc, which includes some code samples using the library.

Filed under: Projects No Comments