Wacky Ideas 1: Inheritance is dead, long live mix-ins!

(Warning: I’ve just looked up “mix-in” on Wikipedia and their definition isn’t quite what I’m used to. Apologies if I’m using the wrong terminology. What I think of as a mix-in is a proxy object which is used to do a lot of the work the class doing the mixing says it does, but preferably with language/platform support.)

I’ve blogged before about my mixed feelings about inheritance. It’s very useful at times, but the penalty is usually very high, and if you’re going to write a class to be derived from, you need to think (and document) about an awful lot of things. So, how about this: we kill of inheritance, but make mix-ins really easy to write. Oh, and I’ll assume good support for closures as well, as a lot can be done with the Strategy Pattern via closures which would otherwise often be done with inheritance.

So, let’s make up some syntax, and start off with an example from the newsgroups. The poster wanted to derive from Dictionary<K,V> and override the Add method to do something else as well as the normal behaviour. Unfortunately, the Add method isn’t virtual. One poster suggested hiding the Add method with a new one – a solution I don’t like, because it’s so easy for someone to break encapsulation by using an instance as a plain Dictionary<K,V>. I suggested re-implementing IDictionary<K,V>, having a private instance of Dictionary<K,V> and making each method just call the corresponding one on that, doing extra work where necessary.

Unfortunately, that’s a bit ugly, and for interfaces with lots of methods it can get terribly tedious. Instead, suppose we could do this:

using System.Collections.Generic;

class FunkyDictionary<K,V> : IDictionary<K,V>
{
IDictionary<K,V> proxyDictionary proxies IDictionary<K,V>;

void IDictionary<K,V>.Add(K key, V value)
{
// Do some other work here

proxyDictionary.Add(key, value);

// And possibly some other work here too
}
}

Now, that’s a bit simpler. To be honest, that kind of thing would cover most of what I use inheritance for. (Memo to self: write a tool which actually finds out how often I do use inheritance, and where, rather than relying on memory and gut feelings.) The equivalent of having an abstract base class and overriding a single method would be fine, with a bit of care. The abstract class could still exist and claim to implement the interface – you just implement the “missing” method in the class which proxies all the rest of the calls.

The reason it’s important to have closures (or at least delegates with strong language support) is that sometimes you want a base class to be able to very deliberately call into the derived class, just for a few things. For those situations, delegates can be provided. It achieves the same kind of specialization as inheritance, but it makes it much clearer (in both the base class and the “derived” one) where the interactions are.

One point of interest is that without any inheritance, we lose the benefits of a single inheritance tree – unless object becomes a general “any reference”, which is mostly what it’s used for. Of course, there are a few methods on System.Object itself which we’d lose. Let’s look at them. (Java equivalents aren’t specified, but Java-only ones are):

  • ToString: Not often terribly useful unless it’s been overridden anyway
  • GetHashCode/Equals: Over time I’ve been considering that it may have been a mistake to make these generally available anyway; when they’re not overridden they tend to behave very differently to when they are. Wrapping the existing behaviour wouldn’t be too hard when wanted, but otherwise make people use IEquatable<T> or the like
  • GetType: This is trickier. It’s clearly a pretty fundamental kind of call which the CLR will have to deal with itself – would making it a static (natively implemented) method which took an object argument be much worse?
  • MemberwiseClone: This feels “systemy” in the same way as GetType. Could something be done such that you could only pass in “this“? Not a terribly easy one, unless I’m missing something.
  • finalize (Java): This could easily be handled in a different manner, similar to how .NET does.
  • wait/notify/notifyAll (Java): These should never have been methods on java.lang.Object in the first place. .NET is a bit better with the static methods on the Monitor class, but we should have specific classes to use for synchronization. Anyway, that’s a matter I’ve ranted about elsewhere.

 

What are the performance penalties of all of this? No idea. Because we’d be using interfaces instead of concrete classes a lot of the time, there’d still be member lookup even if there aren’t any virtual methods within the classes themselves. Somehow I don’t think that performance will be the reason this idea is viewed as a non-starter!

Of course, all of this mix-in business relies on having an interface for everything you want to use polymorphically. That can be a bit of a pain, and it’s the subject of the next article in this series.

Wacky Ideas – Introduction

I’ve been having a few wacky ideas recently, and I think it’s time to put them to virtual paper. They’re mostly around how we think about OO, and how future languages and platforms could do things. I very much doubt that any of them are new. I suspect they’ve been mulled over by people who really know how to think about these things, and then write papers about them. Probably using TeX. I’m not going to that much effort, so there will be several things I haven’t thought through at all. I won’t go so far as to say that’s your job, but knowing my readership you’re likely to come up with loads of things I’d never considered anyway.

Most are likely to be phrased in C#/.NET terms, but they’re likely to apply to Java anyway. Some may have a few better fits in one language than another – I’ll point them out when I think of them.

I don’t necessarily think these are good ideas. Some are probably stinkers. Some may well be useful. Some may even occur one day. Some are bound to exist already in languages I don’t know, of which there are many. Almost all of them are likely to introduce new syntax (or take some away) which makes them non-starters for many scenarios. Don’t take it all too seriously, but I hope you have fun.

What would make a good Java book?

So, Groovy in Action has been out for a little while, and I’m missing it – or rather, book writing. I’d like my next project to be a solo effort, almost certainly on Java. However, I’m interested in hearing what you good folks think would make a good Java book. I’ve got some ideas myself, but I’d rather hear unprejudiced opinions first. (I may be soliciting more feedback at a later date, of course.) So, shoot – what would you like me to write about?

Build and config friendliness counts

Yesterday, I bought a Toppy PVR when my Tivo died. The details of what it does are irrelevant (although quite fun). The important thing is that it’s very hackable – so there are lots of extensions and access programs available. While the Windows ones are typically in binary form, the Linux ones aren’t. The Toppy gives access via a USB port, and programs either access that directly or use FTP to transfer files to it via an intermediate server which basically converts FTP requests into whatever protocol the USB connection uses.

Now, I have a Linkstation with the OpenLink firmware installed on it – a hard disk running a very cut down Linux on a fairly pitiful processor. I had a few bits and bobs on it already, notably TwonkyMedia and Subversion. While TwonkyMedia was a binary installation, Subversion was built from scratch, which took a little bit of doing, mostly because the configure script required sort, which wasn’t provided in the tools for the Linkstation. Doesn’t sound too bad – you just need to download the right package to build and install sort, right? Guess what the configure script for sort requires? Yup – sort. Fortunately a friend helped me out, battling with makefiles until we had a verison of sort which worked well enough to rebuild it properly.

All of this is partly background, but partly the whole point of this blog – build annoyances.

Anyway, back to the Toppy. Over the course of the last 24 hours or so, I’ve fetched/built the following packages on the Linkstation:

  • puppy – a “direct connection” client to fetch/store files
  • ftpd-topfield – an FTP/USB proxying server
  • toppy-web – a web application allowing (limited) remote control of the Toppy
  • lighttpd – a light-weight web server to host toppy-web
  • php – PHP, required as a fastcgi module for lighttpd to run toppy-web
  • libxml2 – an XML library required for some PHP features
  • byacc – Berkeley’s lex/yacc implementation; libxml2 needs yacc
  • ncftp – a Linux CLI FTP client

(Apologies if any of the dependencies in here are wrong. It was getting pretty late by the time I’d got a php-enabled webserver…)

The build/install procedure of all of these varied immensely, and the impression I gained of the quality of the software reflects this. For those of you who don’t do Linux builds regularly, the normal procedure is to run ./configure, then make, then make install. Sounds simple, right? Well…

puppy was straightforward, although it didn’t have a make install – it just built a binary. Still, it was simple enough, and worked first time.
Running it without specifying any arguments gave a useful help message, and it was easy to get to grips with.

ftpd-topfield wasn’t bad either. This time there was a make install, and even a make test. On the other hand, running it without any arguments just returned back to the console with no hint of what was going on. Using --help produced a reasonable help message, but it’s still not clear to me what it does when you don’t specify -D for standalone mode. It could be for running from inetd, but I don’t know. Anyway, it worked pretty well, and I don’t remember any build problems, so it can’t have been that bad. I had to write my own rc.init script, but a simple one is sufficing for the moment.

toppy-web was where the problems began. Now, it doesn’t need to be built itself, but it requires a PHP-capable web server. It’s not quite “onto the next item” though, because this is the one thing I still haven’t got running – due to the configuration aspect. It comes with a bunch of sample configuration files, but you have to hunt around the web for documentation, which still seems to be flaky at best. Now, I can entirely sympathise with the developers, but the point of this blog post is the comparison of build/configure procedures. I’ll get it going at some point, I’m sure (given the effort I’ve put into the rest) but I’m not sure I’ve got the energy just yet. This is a web application – why can’t it be configured with a web page?

lighttpd – the web server itself. This wasn’t too bad, if I remember rightly. The docs have fairly good descriptions of how to configure it appropriately, including what’s required from a php build. Which brings us to…

php – oh dear. It’s never a good sign when the configure script complains about syntax errors to start with. After googling these errors and finding that other people who had received them were told that basically they could be ignored, I let the script continue. After quite a few minutes, it decided that I needed libxml2. At first, I tried to disable this – at which point (well, after starting the script again and letting it run for a few minutes) it complained that without libxml2 it wouldn’t be able to have any DOM support. I don’t know whether toppy-web requires DOM support, but it seemed like an important thing to be without. So, I decided to download libxml2 and build it. In retrospect, I should probably have looked through the toppy-web source to see whether I really needed it…

libxml2 fairly quickly announced that it needed yacc. That’s not particularly unreasonable, and I was slightly surprised not to have it. However, it was yet another step. Fortunately, byacc built and installed easily enough to not deserve its own paragraph here. Hurrah – finally I could configure and build libxml2. You wouldn’t expect that to take too long, would you? XML isn’t that hard. Hmm. It wasn’t actually difficult but I felt very sorry for the Linkstation afterwards. Most of the C files had to be compiled twice for some reason (I didn’t look at the depths of why – something to do with how the libraries are deployed, I believe) and some of them are large. Very large. xmlschemas.c is over 28,000 lines long, and the (fortunately auto-generated) testapi.c is over 50,000 lines long. C compilers are slow beasts at the best of times, and this is on a box which was really only meant to run samba. It took ages. Not only that, but the first time I had to link php with it, it failed. No idea whose fault that was (mine, php or libxml2 but it was frustrating.) Anyway, back to php…

With libxml2 built, I set php configuring and building – with the three features enabled which I knew I had to have (thanks to the lighttpd docs) and which all sound like they should be enabled by default. It got there in the end, so by the time I went to bed (very late) I had a working php-enabled web server.

Tonight, I decided to find a command-line FTP client for Linux. After finding a very useful Linux FTP client comparison page I decided to plump for ncftp. After the previous night’s frustrations, I was somewhat nervous. Fortunately, my fears were unwarranted. More than that – the authors of ncftp deserve awards.

The readme file is suitably undaunting. The configure script runs reasonably quickly. So far, so unremarkable… but the make file. Oooh… instead of showing you the exact command it’s running at any time (which in some cases for the previous packages was over a screenful for a single link, and frequently 5 or 6 lines per compile), it just tells you what it’s doing (e.g. Compiling c_utime.c... and has a colour-coded [OK] (in the same way that most Linux distributions do for starting and stopping services) for each build step. Green means a clean build, yellow means there were warnings (which are printed). At the end of the build, it shows you the files it’s built. At the end of make install it shows you the files you’ve installed. The difference in professionalism and the impression you’re left with is marked. I’ve no idea whether the ncftp guys wrote the makefile themselves or whether it’s a framework which will is generally available – but I’ve never seen anything as clean when it comes to make. Indeed, it leaves me wish that Ant builds were logged as cleanly (when they’re successful).

So, the moral of this post (which is rather longer than I’d anticipated)? Builds matter. Configuration matters. Documentation matters. There’s more to a build being good than just it working first time: giving feedback and a warm fuzzy feeling are important too. Making it look simple even if there are complex things going on behind the scenes makes life feel smoother. (I’m sure the ncftp makefile has options for seeing the commands themselves if you really need to.) I’ve understood the importance of a good build system before, but usually in terms of a developer having all the options they need. In the open source world, particularly on Linux where in many cases the end user will have to build the package in order to run it on their custom devices, the build is just as much a part of “ease of use” as the product itself – and if a user falls at the first hurdle, they’ll never see your pretty UI.

New blog, new project

I’ve started a new blog, which I’ll be sharing with a couple of colleagues. In brief, the idea is to try to do a “hobby” project as well as possible, the whole purpose being the learning experience.

Rather than waffle on about it here, I’ll just refer you to the blog of the
Quest for the Perfect Project. Hope it’s of some interest to some of you.

Oh, and this blog will carry on as normal.
 

Wii and MythTV: The future of my living room?

My Tivo has started playing up. A few months ago, it stopped padding programmes automatically like it used to (I haven’t investigated why – getting a console session on it is pretty tricky; it may well be a software update that removed the patch I’d put on) and now it glitches sometimes in a way that suggests the disk may be packing up.

Likewise, my cheap DVD player is pretty rubbish, and needs replacing, and my Freeview receiver is far from perfect.

I had been considering buying a spiffy DVD/HD recorder, and was probably going to start looking sometime in the next couple of months. However, using my Wii over Christmas has given me a different idea: using the Wii remote to control a MythTV box.

The Wiimote, for those of you who haven’t used one, is a small remote control that has relatively few buttons for a gaming controller, but which is able to detect when you point it at the TV, and which has accelerometers to determine tilting and motion. It’s wireless, using Bluetooth to talk to the Wii.

Now, imagine this:

  1. Point at the TV and press “home” to get to the main menu; select “Live TV” or “Recorded Programmes” by pointing at the relevant menu items and pressing the main button (A).
  2. Programme not at the right point? Pause it (with A), hold down the trigger on the back and tilt to the left or right (like a volume knob) to get to the right place quickly and accurately.
  3. Want to fast forward or reverse just a bit? Don’t bother pausing – just hold down the trigger and tilt to change the speed (and direction) of playback.
  4. Watching Live TV and want to change channel? Press “1” for favourite channels (chances are the channel will be on the first page) or “2” for all channels (in channel order, probably on more than one screen: either click on page up/down, or hold the trigger and “drag” up or down.
  5. Volume control? Why, that’s what the -/+ are for, naturally.

I’ve already got a Roku Soundbridge for playing my MP3s (from a network disk) but I suspect that choosing an album/song with a whole screen would be somewhat simpler than with a single line. (Don’t get me round – I love my Soundbridge. It’s stylish and does the job really well.)

Unfortunately, I have no experience whatsoever with MythTV, and very little time to hack around with it. The good news is that I’m not the only one to have come up with the idea (I’d be shocked if no-one else had thought of it) although it sounds like they’ve only done the buttons so far. I expect more full functionality will come along in time, assuming the underlying MythTV platform supports it.

There are two downsides I can think of at the moment:

  1. I suspect the Wii’s “sensor bar” (a misnomer as it doesn’t sense anything – it’s just an array of IR transmitters) is turned off when the Wii is on standby. I’d need to be able to give it power without the Wii being on, ideally.
  2. How can I tell the Wii that I’m controlling MythTV and vice versa? I’d rather not need an extra controller just for the sake of MythTV – although if it comes to that, they’re not that expensive.

I only had the idea of the whole package (tilting etc) earlier today, and haven’t stopped being excited about it. If I were Hauppage or a similar company, I think I’d already be contacting Nintendo to try to licence the technology – I suspect that the first company with a mainstream product which supports a “point/click/tilt” (rather than “find the button”) PVR UI could make a lot of money.

The irony is that the Wii is the only one of the three next-gen consoles not to have designs on becoming a media centre – and yet it’s the one that I suspect I’m most likely to use for that very purpose (albeit just the controller).

Groovy 1.0 released

Groovy 1.0 has finally been released, and is available for download from the Groovy home page.

For those of you who didn’t read my last blog post on it, Groovy is a dynamic language which runs on the Java platform and integrates very nicely with Java itself.

What I haven’t advertised on this blog until now is that a month or so after starting with Groovy, I became involved in the Groovy in Action book. I won’t claim that I wrote a lot of it – more polishing and adding some introductory material, really. I’m still extremely proud of it, however, and obviously I hope it’ll sell bucket-loads.

So, if you haven’t played with Groovy yet, now’s a very good time to start: download it, have a bit of a play, and then buy the book!

Everything old is new again

I feel I’m too young to be making this kind of statement, but the sense of deja vu I get when reading about the layouts in WPF makes me nearly laugh out loud. Of all the things I can remember about Java 1.0 (this was before any number of things we take for granted now) I know that LayoutManagers exist – including the insanely hard to use GridBagLayout. Fortunately I learned over time that it wasn’t always the most appropriate manager (far from it) but it was powerful.

Ever since I started using .NET I’ve been annoyed at WinForms’ lack of layout management facilities. I’m pretty sure that when I complained about this to Chris Anderson once he said that it’s all there really, but just not exposed very well. Hmm. I’ve tried and lost patience a few times – I suspect there are ways of sorting it all out, but it’s not exactly straightforward.

Fortunately it all changes with WPF, which not only has various layout managers (not sure whether there’s anything like Java 1.4’s SpringLayout, which seemed useful last time I looked at it) but also makes them relatively easy to use with XAML. No more GridBagConstraints messes! Hurrah!

Anyway, it’s good to see that .NET 3.0 has caught up with Java 1.0 :) (And no, I’m not under the impression that Java invented the concept of layout managers. It just happens to be the first environment I used them in.)

The CLI memory model, and specific specifications

A while ago, I was directed to a disturbing (in my view) post on GrantRi’s blog, to do with the .NET memory model. I’m slightly less disturbed having read his clarification, but there’s a fairly deep problem here. Here’s part of a sample class:

string name;
    
public void WriteNameLength()
{
    string localName = name;
    if (localName!=null)
    {
        Console.WriteLine (localName.Length);
    }
}

Now, other threads may be changing the value of name all over the place, and there’s an issue in WriteNameLength in terms of whether or not it shows the “latest” value, but my question is: can the above throw a NullReferenceException?

It looks like it can’t, because even if name becomes null during the method, surely the value of localName can’t change – it’s either null or it’s not, and we don’t try to dereference it if it’s null.

Unfortunately, it looks from Grant’s blog post as if a JIT should be free to treat the above as:

public void WriteNameLength()
{
    if (name!=null)
    {
        Console.WriteLine (name.Length);
    }
}

Now the above clearly can throw an exception, if name becomes null in another thread after the “if” and before the dereference (and if that change is noticed by the thread running WriteNameLength).

This surprises me – just as it surprised lots of people reading Grant’s blog. It surprised me so much that I checked the CLI specification, and couldn’t work out whether it was correct or not. This is even more worrying – so I mailed Grant, and his (very speedy) reply was along the lines of “I’m not an expert, but it looks to me like the spec is too vague to say for sure whether this is legitimate or not.” (I apologise if I’ve misrepresented the reply – in some ways it doesn’t matter though.)

When trying to write performant, reliable systems, it is surely crucial to have a memory model specification which can be reasoned with. The Java memory model was reasonably well defined before 1.5, and then (after years of detailed discussion) it was updated in way which I believe was designed to give backward compatibility but lay out very clear rules. Surely the CLI deserves a specification with a similar level of detail – one which both JIT developers and application developers can use to make sure that there are no surprises amongst informed individuals. (There will always be people who write multi-threaded programs while remaining blissfully unaware of the importance of a memory model. It’s very hard to cater for them without crippling JIT optimisation, effectively synchronising all the time. I’m not too worried about that.)

Usually, when I’m writing multi-threaded code, I err on the side of caution – I tend to use locks when I could get away with volatile variables, for instance, just because I need to think slightly less hard to make sure everything’s correct. There are people for whom that’s not just good enough – their performance requirements make every context switch, every locking operation, every optimisation restriction valuable enough to really need to know the details of the memory model. There should be an effort on the part of MS and/or the ECMA committee to clearly and specifically define what the CLI memory model does and doesn’t guarantee. I doubt that anyone reading this blog is in a position to instigate such an effort – but
if you are, please give it careful consideration.

Unit tests rock, I suck, news at 11

I’ve just started looking at my Miscellaneous Utility Library again, after quite a while. I’m currently running Vista on my laptop, which means I can’t run Visual Studio 2003 – so it’s about time I updated the library to use generics and all that goodness. I’ll keep the .NET 1.1 version available on the web site, but from now on any new code will be 2.0 only.

In the process of updating RandomAccessQueue to use implement the generic collection interfaces, I decided to do the implementation test-first, as is now my habit. It clearly wasn’t habit back when I originally wrote the code (the same day Peramon laid everyone off, incidentally – I remember as I was at home, ill). The new methods use some of the old methods – and unfortunately that’s now exposed some long-standing bugs.

Looking back, I find it hard to understand why I had so much faith in this code: it’s the kind of code which is bound to suffer from off-by-one errors and the like. It’s not terribly hard to test, fortunately (unlike the threadpool stuff, for example). Oh how I wish I’d been using NUnit back then.

This happened the last time I looked at a MiscUtil class, too. It will take a while to add unit tests giving a decent level of coverage to the code – it’s not like I have a lot of spare time – but it’s clearly got to be done.

I wonder how much other code I’ve written over the years is riddled with bugs? To be fair, the MiscUtil stuff was run considerably less than most of the code I wrote professionally at Peramon… but I bet there were quite a few nasty little gotchas waiting there too. And now?
No, I don’t write perfect code, even with unit tests. Even when the code does what I intend it to do, I have to revisit whether the intention was right in the first place. Even with unit tests there can easily be problems which slip through the cracks – but I don’t think I produce nearly as much code which is basically broken as I used to.