P2Proto: a Java RPC library for P2P applications built on protocol buffers
Yet another coding project I've been working on - this one is an RPC library for Java applications using protocol buffers. This started a few years back when I had a peer-to-peer app using Java RMI that needed to handle a large volume (hundreds) of simultaneous connections and calls, and the app was getting crushed under the load. Another problem was that RMI doesn't really have asynchronous method calls, which was creating an additional performance bottleneck due to the massive numbers of thread required.
To deal with these issues, I switched to using Google's Protocol Buffers, which are compact and efficient, but don't have any decent built-in RPC capabilities. Thus, I set about rolling my own RPC library for protocol buffers. I used it a number of times for distributed programs over the years, and each time improved it somewhat as the project dictated. For my most recent project, I felt that it finally reached a state that could be classified as a usable library, so I added some decent documentation and cleaned everything up to release.
The library itself handles a number of functions not provided by protocol buffers themselves:
- Delimiting and extracting protocol buffer messages from a persistent network stream.
- Setting up two-way message channels over which to exchange protocol buffer messages.
- Associating query and response messages, independent of message arrival order (i.e., there's no need to manually determine which query message a response corresponds to).
- Convenient handlers for the various types of RPC errors that can occur (local send errors, remote processing errors, and communication timeout errors).
This is all done without any synchronization (except for when multiple threads are using a single connection) or busy waiting, so the library overhead should be low. Protocol buffer messages themselves are extremely compact, so the amount of network data overhead should also be minimal.
As usual, here are the download links: the main project page, the complete library source, the library jarfile only, and finally, the online Javadoc, which includes some code samples using the library.
JSava: a Java bytecode interpreter in JavaScript
Here's another cool project I've been working on - it's an interpreter for Java bytecodes (i.e., like the JVM). What's more interesting about it, however, is that it's written in JavaScript. Furthermore, it runs on top of Mozilla Rhino, which means it's actually C code (the JVM) which runs Java code (Rhino) which runs JavaScript code (the interpreter, JSava) which runs Java code (a user program). The double-interpreter thing does take a hit on speed, but the point isn't to beat the JVM for speed - it's to just accomplish something cool in a quirky language like JavaScript.
This may seem like deja vu to Slashdot readers, which is what prompted me to post this. I think JSava is particularly interesting, however, because it can handle many fairly advanced features of the JVM and is capable of executing much (most?) of the JDK standard library. Some feature highlights include:
- All 200 JVM opcodes are implemented.
- All native Java types are supported, including 64-bit Longs, which are not natively supported by JavaScript. This is because JavaScript represents everything as 64-bit floating point numbers, which cannot store the full range of 64-bit integer values.
- Full support for the Java threading model, including multithreading, synchronization between threads, and monitors (wait() and notify()). However, note that multithreading does not imply parallelism, which is at the moment effectively impossible in JavaScript - JSava provides threading support by timeslicing a single actual thread of execution.
- JSava is complete enough to execute most major JDK library classes, such as java.util.HashMap and java.util.Random. Even some more exotic classes are mostly functional, such as java.util.concurrent.ExecutorService and company.
- Substantial support for the Reflection API. This may seem like a weird thing to support, but reflection is heavily used in the JDK library classes, so supporting it is critical in enabling complete functionality.
- All the usual Java goodness: exceptions, inheritance, interfaces, runtime error checking, etc... basically all the language features we know and love.
- Native method handling capabilities for handing off to JavaScript implementations (e.g., System.arraycopy(), Thread.start() and lots of JDK support methods).
- Rudimentary I/O support, accomplished through a lot of low-level trickery involving intercepting file descriptors and mapping them to native Java streams. JavaScript itself has effectively zero I/O support, but the only time Java is invoked to handle I/O is in the lowest level native methods (e.g., 'write these bytes to this file handle'). Everything above that (such as I/O buffering) is handled inside the interpreter.
The JDK class files which JSava was developed with are taken from the Mac OS X 10.6 JDK (Java SE 6). The interpreter may work on Linux, but probably won't on Windows without a little bit of tweaking, particularly for I/O (it's on my todo list).
As usual, I've also created a new program page for the project. The main download link (or Dropbox mirror) provides the entire JSava runtime environment, including Rhino and JDK library classes (3.3M). If you have Java installed, running the interpreter is as easy as executing one command (no building required!). If you only want to look at the source code and not actually run the interpreter, you can also download the source only (54K).
There were a lot of engineering tricks that went into JSava and I may write about some of them in the future. However, JSava is still very much a work in progress and I plan to extend it to handle even more than it already does.
plmtools-imeter
I've packaged up a new release of the modified plmtools package for talking to the iMeter. I think the code has diverged from the base plmtools enough to warrant a rebranding, so I'm (uncreatively) naming the fork 'plmtools-imeter'. I've posted a page for the project over on the main site (but for the impatient among you, here's a direct link).
Improvements from the previous release include support for both Revision 1.1 and 1.15 iMeters (thanks to Erik Wile for his detective work regarding that issue) and a variety of other smaller improvements.
A bit over a week ago I presented my research group's recent work on building monitoring using home automation devices at the BuildSys 2011 workshop in Seattle. Our paper on this work ('Exploiting Home Automation Protocols for Load Monitoring in Smart Buildings') was facilitated in part by the plmtools-imeter package.
plmtools for the iMeter Solo and/or the GuruPlug
Update (11/10/2011): I've posted an updated version of the code that fixes the issue with the new 1.15 iMeters: see my more recent post.
Update (9/26/2011): Smarthome has come out with a new revision of the iMeter that breaks this utility -- the meter firmware seems to have changed, and the new one does not provide all the information of the old. To see if you have the old meter (that does work), look for a sticker on the back side of the iMeter. If you see a "V1.1", then you have the old one (good). If you see a "Rev. 1.15", then you have the new one (bad). I'm working on figuring out the 1.15 protocol to get this utility working again with the new ones.
One of my current research projects involves instrumenting a house with a lot of Insteon home-automation devices and collecting data from them. I've been using the very useful Linux plmtools utilities to talk to the Insteon devices programmatically. However, the plmtools suite has a few shortcomings for my purposes: one, the current release doesn't compile or run on ARM-based machines (such as the GuruPlugs I'm using) and two, it has no functionality for talking to the new Insteon power meter, the iMeter Solo. The iMeter Solo basically works exactly like a Kill-A-Watt meter, except that since it's an Insteon device it's possible to interface with it.
I've dealt with both of the above problems. I've fixed the ARM-specific problems in the code so it runs great on the GuruPlug, and I've added functionality for querying the current wattage level from an iMeter Solo. I imagine there are others who would like to use plmtools on ARM-based machines, and as far as I know there are no existing tools that allow you to query the iMeter from within Linux. Now, you can query the current watt draw reported by the iMeter using a simple 'insteon mydevice meter' command.
Also note that the iMeter Solo is frequently advertised alongside a special 'HouseLinc'-enabled version of the PLM that's required to use the HouseLinc software that can talk to the iMeter. This is unnecessary for my solution, which works just fine on a regular PLM.
You can download the modified plmtools here. See the included README for details both on how the base plmtools utilities work as well as how the added meter functionality works. I'm trying to get in touch with the original author of plmtools to see if I can get these changes integrated into a future release on SourceForge, but so far haven't had any luck.