This is an overview page for my past and current Erlang programming ventures, together with some comments about status, utility, etc.
I’ve listed both the successful and the unsuccessful ones, in no particular order.
This early User Contribution was modeled after a read-write locker component in AXD 301. It uses a master-slave concept and has proven quite scalable. Perhaps the best thing about this contrib was that it provided input to Thomas Arts, Clara Benac Earle et al for their research in formal verification of Erlang code. The resulting discussions also led to the development of gen_leader.
As far as features go, the locker does support read and write locks as well as lock promotion (from read to write lock). It also detects partitioned networks and merges locks, detecting conflicts, if there are any.
This was a pair-programming excercise together with Thomas Arts (I love pair programming, esp. together with Thomas and John Hughes – makes me feel like a better programmer.) It came out of discussions around features of the locker (above). Thomas and Hans Svensson later went on to apply various forms of clever model checking on it, and uncovered some fairly disturbing bugs. The bugs forced a rewrite with a new leader election algorithm, and another result from that effort was a formal semantics for Distributed Erlang.
The latest version of gen_leader worth having can be found here,.
A few years ago, I set out to try to understand exactly where complex state machine programming becomes so unwieldy. It took a while to catch up, only to realise that Erlang veterans like Mike Williams (starring in Erlang The Movie) figured this out decades ago. It had always bugged me that the OTP behaviors offered debugging and software upgrade support at the expense of textbook Erlang state machine design. Plain_fsm is an attempt to rectify this. It is an adaptation of code actually being used in signaling applications at Ericsson.
VCC stands for “Virtual Channel Controller” or “Version-Controlled Communication” or “Version-Controlled Channels” – I could never decide which. This is perhaps in the true spirit of Erlang (which stands for ERicsson LANGuage, or refers to Agner Krarup Erlang, whichever you want.) The general idea that I was exploring was that software upgrade using suspend-code_change-resume, the standard Erlang way, becomes a bit too messy for high-performance, massively scalable IP telephony systems with tens of thousand active sessions. I thought about doing redundancy upgrade by cutting off Distributed Erlang, having vital applications keep talking through version-controlled virtual channels, converting one-half of the system off-line to a new major version, failing over, and then bootstrapping up the other half. It’s worked well in small experiments.
VCC (source here) is also intended to work between Erlang and non-Erlang applications. It has a number of nifty features, e.g. ability to combine state transformation functions for different upgrade paths, temporary request-reply channels, custom per-channel encode/decode, etc.
It’s very much a prototype, and what little documentation there is, is inconsistent and confusing.
This work started back in 1997 when we needed a way to import text files with configuration data into AXD 301. I wrote a small data dictionary and data import tool. Later I thought It’d be fun to expand it into a relational DBMS. Years later, I actually found a beta tester, Rudolph van Graan. This did wonders for actually making rdbms near-product quality.
There are some goodies in there, such as parameterized indexes (indexes are first-class tables), JIT compilation of the dictionary for minimal runtime overhead, and type, bounds and referential integrity checking.
I haven’t worked on it for quite a while, though. It would need to be ported to OTP R12B, and the feature set should be examined, not least in order to reduce the number of mnesia patches required.
One of the features I’ll probably remove is the support for ‘external_copies’ (user-defined table types). However, the approach used in rdbms was used as a starting point by Joel Reymont for his adaptation of mnesia to Amazon S3, and by the Dukes of Erl for mnesiaex and its support for TokyoCabinet in mnesia.
An early User Contribution for loading a Mnesia backup into ets tables for easy viewing. It may still work, I haven’t tried it in ages.
A fairly new project. Joe Armstrong and I discussed our ideas on how web-based group collaboration tools really should work. My own most rewarding experience involved a combination of SoftArc’s FirstClass and FileMaker databases.
One nice idea in Erlhive is that, apart from being a multi-user Content Management System, it could also be an on-line multi-user application development environment. Users can upload code and make APIs available to other users on the same Erlhive site. The concept was illustrated in this EUC presentation from 2006. Since then, I’ve added the ability to execute most types of Erlang code within Erlhive transactions (!).
The main weakness of Erlhive, as web application development environments go, is that it has very little, if any, web content support. One explanation for this was that I didn’t really need to develop any web content, and was more interested in the underlying mechanisms. My current thinking is that sgte would be a good templating system for Erlhive, and it wouldn’t be too difficult to come up with a naming scheme to allow templates to be reused in much the same ways as code.
Builder was a spinoff from some parts of the installation support used in AXD 301. One of the ideas behind builder is that you could build your system one application at a time. Builder will recursively collect environment variables etc from underlying applications, and will create boot scripts in the priv/ directory of the current application. I don’t know how many use it, but I’m pretty sure that CEAN does.
Extended Erlang Shell
A flexible distributed process registry, modeled after a central component in Ericsson’s Session Border Gateway. I presented this at the SIGPLAN Erlang Workshop 2007. You can find the source and the presentation here..
This code relies on a patched gen_leader, and uses a number of patches to OTP for demonstration purposes (they are not needed in order to use gproc). I am awaiting a new version of gen_leader that addresses some known problems, and then perhaps I’ll write an EEP to try to get this into OTP.
Parse Transform utilities
Bucket Grid and Gridfile
These early User Contributions (bucket_grid and gridfile) were part of an experiment that didn’t pan out. I had read an article about multi-dimensional indexing in Stonebraker’s Readings on Database Technology, and tried implementing it in Erlang. The big mistake was to try to make it type-agnostic. Since the algorithm relies on interval splitting, this was somewhat difficult to pull off, esp in a manner that was efficient and scalable. It works on simple examples, but doesn’t scale well, and most likely has some serious bugs in it. Not recommended.
An early User Contribution modeled after one of the approaches we tried for ATM call control in the AXD 301. The main reason for doing something like this was to reduce the number of active processes. Since the number of simultaneous processes is no longer a practical bottleneck in Erlang, we have since moved on to a different model (see gproc).
These contribs are no longer needed, as the
array module in OTP, written by Richard Carlsson and Dan Gudmundsson, is supported and much more efficient at that. The lines module does have some features that array doesn’t, mainly the ability to insert or delete a line in the middle of the array. The intended use of this was in an editor buffer.
A fun thing about the lines module was that we used that when John Hughes was demonstrating Erlang QuickCheck – since lines was a fairly simple module, well-tested and “obviously correct”. It was correct… except for the optimized function new(Size). The magic number was 46. When creating arrays larger than 46 elements using the optimized method, the structure was corrupted. Thank you QuickCheck.
As the main point of contact for the largest Erlang project ever, I’ve had reason to discuss improvements in many areas of OTP. In most cases, comments and suggestions have come from many different sources, but in a few cases, the ideas were (for better or for worse) fairly distinctly mine:
- Included applications
- Application start phases
- User-defined behaviours
- Encrypted debug info
XMerl came about during a week back in 2001, as I was reading up on XML from a book (not a very good book, btw). I wanted to play around with the mapping of XML to simple Erlang tuple trees. It was fun at first, but the deeper one gets into XML, the more frustrating it gets. I stopped short of trying to implement support for XSLT. Johan Blom, Mickaël Remond and Richard Carlsson were early adopters and for several years, they supported and improved on XMerl. Richard developed edoc with the help of XMerl, and this eventually became a strong argument for moving XMerl into OTP.
XMerl is fairly feature-complete (except for XSLT and XQuery, but it does have a module that provides an Erlang-style equivalent to XSLT). It does have a few warts: it isn’t very fast, it produces very large record structures by default, and the documentation lacks good support for getting started. The first two can be corrected by breaking out the token scanning and allowing for alternative front-ends (where one is backwards compatible.) I am fairly confident that these issues will be addressed soon.