Category Archives: Books

Book Review: Effective C# (2nd edition) by Bill Wagner

Resources:

Disclaimer

Just in case you’re unaware, I’m the author of another C# book, C# in Depth. Although Effective C# is somewhat different to my book, they certainly share a target audience. To that extent, Bill and I are competitors. I try hard to stay unbiased in reviews, but it’s probably impossible. Bear this in mind while reading. I should also note that I didn’t buy my copy of Effective C#; it was kindly sent to me by Pearson, for the purpose of reviewing.

Content and target audience

Effective C# is a style guide for C# developers – but not at the low level of "put your braces here, use PascalCase for method names;" instead, it’s at the design level. As far as I can tell, the aim isn’t to be complete, just the most important aspects of style. (Hey, otherwise there wouldn’t be any need for More Effective C#, right?) There are 50 mostly-self-contained items, totalling about 300 pages to digest – which is a nice size of book, in my opinion. It’s not daunting, and the items can definitely be bitten off one at a time.

Looking down the table of contents, the items are divided into six categories: "C# language idioms", ".NET resource management", "Expressing Designs in C#", "Working with the Framework", "Dynamic Programming in C#", and "Miscellaneous". Broadly speaking these contain the sorts of thing you’d expect – although it’s worth pointing out that a significant chunk of "Working with the Framework" is given over to Parallel Extensions, which may not be obvious from the title. (It’s a really good source of information on PFX, by the way.)

This is not a tutorial on C#. If you don’t know C# reasonably well already (including generics, lambda expressions and so on) you should read another book first, and then come back to Effective C# in order to get the most out of it.

Comment from Bill: generics and lambda expressions (and LINQ) are covered in some detail in More Effective C#. It’s a bit strange that as of the 2nd edition, Effective C# covers a newer version of the language than More Effective C#. I tried hard to make sure neither book expects a reader to have read the other, but the organization of both books as a whole does show the hazards of hitting a moving target.

That’s not to say that there’s no explanation of C# – for example, Bill goes into a few details about the "dynamic" type from C# 4, as well as overloading and how optional parameters work. But these are meant to just cover some poorly-understood (or very new) aspects of the language, rather than teaching you from the beginning. The balance here feels just right to me – I believe most professional C# developers will learn details of C# they weren’t aware of before, but won’t be confused by the basics that Bill left out.

Accuracy, opinion and explanation

My copy of Effective C# has plenty of ink annotations now. They broadly fall into five categories:

  • "Ooh, I’d never thought of that" – aspects of C# which were genuinely new to me
  • "Hell, yes!" – things I agree with 100%, and which will help developers a lot
  • "Um, I disagree" – points where Bill and I would go probably different routes, presumably due to different experiences and having worked in different contexts. (It’s possible that when put in the same context, we’d do the same thing, of course.)
  • "No, that’s technically incorrect" – a few areas which are outright wrong, or were correct for previous versions of the framework/CLR, but aren’t correct now
  • "That’s not what that term means" (or "that’s not the right term for the concept you’re trying to get across") – it should come as no surprise to regular readers that I’m a bit of a pedant when it comes to terminology

The majority of my annotations are of the third category – disagreements. That’s not because I disagree with most of the book; it’s just that the second category is reserved for vehement agreement. I haven’t bothered to note every sentence that I’m just fine with.

The good news is that in areas where we disagree, Bill does an admirable job of stating his case. I disagree with some of his arguments – or I can give counter-examples, or merely place different value on some of the pros and cons – but the important thing is that the reasoning is there. If it doesn’t apply to your context, evaluate the advice accordingly.

It’s entirely reasonable for there to be quite a bit of disagreement, as much of the book is opinion. It’s obviously founded in a great deal of experience (and I should note that Bill has spent a lot more time as a professional C# developer than I have), but it’s still opinion. I rather wish that the book was a wiki, so that these items could be debated, amended etc, as per my dream book – I think that would make it even more valuable.

There are relatively few absolutely incorrect statements, and even on the terminology front it’s usually two things which have bugged me repeatedly. Bill uses "implicit properties" for "automatically implemented properties"; I’ve usually heard developers use the abbreviated form "automatic properties" but "implicit" is new to me. Likewise the book talks about "overriding ==" instead of "overloading ==" reasonably frequently. It’s a shame I was too busy with my own book to participate in the technical review for Effective C#, as I suspect that on at least some of these points, Bill would have been happy to amend the text accordingly. I shall, of course, transcribe my comments and send them to him.

Comment from Bill: I’ll make those corrections for the subsequent printings.

What’s missing?

There are some areas which I wish Bill had touched on or emphasized more. Topics such as numbers, text and chronological values could have been given some space as they frequently confuse folks (and are full of pitfalls; see Humanity: Epic Fail for more of my thoughts on this). I would personally have placed more importance on the mantra of "value types should be immutable" – it’s certainly talked about, but in the context of "preferring" atomic, immutable value types – and preferring value types over reference types in rather more situations than I’d personally use. In terms of importance in avoiding shooting yourself in the foot, making sure all structs are immutable comes near the top of the list in my view.

"More Effective C#" doesn’t cover those areas as far as I can tell from the table of contents, but it does go into details about generics and various aspects of C# 3 and LINQ, which are clearly part of any modern C# developer’s toolkit. I certainly intend to get hold of the book to see what else I have to learn from Bill.

I think it might have been nice to have a few sections at an even higher level than the specific items in Effective C#. Topics such as:

  • Don’t trust me: I don’t know your context. Even the smartest folks can only give advice in fairly general terms in books. Don’t apply this advice blindly; weigh up the arguments presented and work out how they apply to your actual code.
  • The importance of testing. It’s possible that this was mentioned, but I don’t recall it. Perhaps it’s a little on the opinionated side (see previous point…) but for significant code bases, testing should be deeply ingrained in whatever process you’re using. Note that although it’s worth trying to keep high standards in test code, it often has a very different look and feel to production code, and different "best practices" may apply.
  • Encouraging "working with the language" – if you find yourself fighting the language, you may discover you can keep winning battles but losing the war. Typically changing your design to represent more idiomatic C# will make life more pleasant for everyone.
  • Performance: how you might decide when and how much to care.

Very few of these would be C#-specific, of course – which may be why Bill left them out. You could easily fill a whole book like that, and it would probably be horrible to read – full of platitudes rather than concrete advice. I personally think there’s room for some discussion of this kind though.

Comment from Bill: The ultimate goal was to have the book be that ‘nice size’ you mention above. I agree that all of those concepts are important. I felt that many of these features were not C# specific (or even .NET specific) that I felt better covered elsewhere.  However, that ‘working with the language’ was one area where I feel that I do cover.  There are only a small number of negative titles (e.g "avoid" something or "do not" do something). In those cases, I tried to recommend alternatives where you would find yourself "working with the language".

Conclusion

I like Effective C# a lot. I like the fact that I’ve disagreed with a number of the points raised, and in disagreeing I’ve found myself thinking about why I disagree and in what situations each point of view may be appropriate. I worry a little about inexperienced readers who may be tempted to treat this (or any other book) as an ultimate truth to be quoted out of context and used to beat other developers into inappropriate solutions… but hopefully most readers won’t be like that.

Comment from Bill: I also hope that most readers avoid that. Thank you for pointing out that I’ve tried very hard to explain when my advice applies, and when it doesn’t.  That is critical.

It’s definitely encouraged me to try to write a somewhat similar book at some point… possibly not with the same organization, and probably dealing with some very different topics – but it’s good to see that it can work. Whether I can pull it off as well as Bill remains to be seen, of course.

I’ll look forward to reading More Effective C# – although my pile of books to review is groaning somewhat, and I should probably go through another of those first :)

You are all individuals! (I’m not…)

I’ve been meaning to post this for a while, but recently a couple of events have coincided, reminding me about the issue.

First, Joe Duffy blogged in defence of premature optimization. Second, I started reading Bill Wagner’s Effective C#, 2nd edition, which contains advice such as "make almost all your types serializable". Now, let’s be clear: I have a great deal of respect for both of these gentlemen… but in both cases I think there’s a problem: to some extent they’re assuming a certain type of development.

In some cases, you really, really want to understand the nuts and bolts of every bit of performance. If, for example, you’re writing a parallelization library to be the part of the .NET framework. For Noda Time I’m pretty obsessed with performance, too – I really want it to be very fast indeed. And to be clear, Joe does give a certain amount of balance in the article… but I think it’s probably still biased due to his background on working on libraries where it really, really matters. For many developers, it’s vastly preferable to have the in-house HR web app used by 100 people take a little bit more time to process each request than to take an extra few days of developer work (cumulative) making sure that every little bit of it is as fast as possible. And many of the questions I’ve seen on Stack Overflow are asking for micro-optimizations which are really, really unlikely to matter. (EDIT: Just to be clear, there’s a lot of stuff I agree with in Joe’s post, but I think enough of us find correctness hard enough to start with, without having to consider every possible performance hit of every statement. At least some of the time.)

Likewise for a certain class of development, it probably does make sense to make most types serializable. If most of your objects are modelling data, serialization really will be a major factor. For other people, it won’t be. Most of my working life has been spent writing code which really doesn’t need to serialize anything… or which uses Protocol Buffers for serialization, in order to preserve portability, compactness and flexible versioning. Very few of my types should really be serializable using the platform-default binary serialization (whether in Java or .NET). Relatively few of them need to be serializable at all.

Finally, I’ll mention another example I’ve probably been guilty of: the assumption that a "public API" really can’t be changed without major hassle. An example of this is making a "public const" in C#, and later wanting to change the value of it. "No," I hear you cry… "Make it a public static readonly field instead, to avoid callers baking the value into their compiled code." Absolutely. If you’re in a situation where you may well not know all of your callers, or can’t recompile them all on every deployment, that’s great advice. But I suspect a lot of developers work in environments where they can recompile everything – where the only code which calls their code is written within the same company, and deployed all in one go.

In short, we’re not all writing system libraries. We’re not all writing data-driven business apps. We’re not all writing the same kind of code at all. Good "one size fits all" advice is pretty rare, and "we" (the community preaching best practices etc) should take that into account more often. I absolutely include myself in that chastisement, too.

Degrees of reality in sample code

Yesterday I tweeted a link to an article about overloading that I’d just finished. In that article, all my examples look a bit like this:

using System;

class Test
{
    static void Foo(int x, int y = 5)
    {
        Console.WriteLine("Foo(int x, int y = 5)");
    }
    
    static void Foo(double x)
    {
        Console.WriteLine("Foo(double x)");
    }

    static void Main()
    {
        Foo(10);
    }
}

Each example is followed by an explanation of the output.

Fairly soon afterwards, I received an email from a reader who disagreed with my choices for sample code. ere are a few extracts from the email exchange. Please read them carefully – they really form the context of the rest of this post.

This is really not proper. When a method can do more than one thing, you might offer what are called ‘convenience overloads’, which make it easier for the consuming developer. When you start swaying away so much that you have wildly different arguments, then it’s probably time to refactor and consider creating a second method. With your example with "Foo", it’s hard to tell which is the case.

My point is, the ‘convenience overloads’ should all directly or indirectly call the one REAL method. I’m not a fan of "test", "foo", and "bar", because they rarely make the point clearer, and often make it more confusing. So let me use something more realistic. So let me use something more realistic. This nonsensical example, but hopefully is clear:

...

The point here was to make you aware of the oversight. I do what I can to try to stop bad ideas from propagating, particularly now that you're writing books. When developers read your book and consider it an "authority" on the topic, they take your example as if it's a model for what they should do. I just hope your more mindful of that in your code samples in the future.

...

Specific to this overload issue, this has come up many times for me. Developers will write 3 overloads that do wildly different things or worse, will have 98% of the same code repeated. We try to catch this in a code review, but sometimes we will get pushback because they read it in a book (hence, my comments).

...

I assume your audience is regular developer, right? In other words, the .NET Framework developers at Microsoft perhaps aren't the ones reading your books, but it's thousands of App Developer I and App Developer II that do business development? I just mean that there are far, far more "regular developers" than seasoned, expert developers who will be able to discern the difference and know what is proper. You are DEFINING what is proper in your book, you become an authority on the matter!

Anyhow, all my point was it to realize how far your influence goes once you become an author. Even the simplest, throwaway example can be seen as a best-practice beacon.

Now, this gave me pause for thought. Indeed, I went back and edited the overloading article - not to change the examples, but to make the article's scope clearer. It's describing the mechanics of overloading, rather than suggesting when it is and isn't appropriate to use overloading at all.

I don't think I'm actually wrong here, but I wanted to explore it a little more in this post, and get feedback. First I'd like to suggest a few categorizations - these aren't the only possible ones, of course, but I think they divide the spectrum reasonably. Here I'll give example examples in another area: overriding and polymorphism. I'll just describe the options first, and then we can talk about the pros and cons afterwards.

Totally abstract - no code being presented at all

Sometimes we talk about code without actually giving any examples at all. In order to override a member, it has to be declared as `virtual` in a base class, and then the overriding member uses the `override `modifier. When the virtual member is called, it is dispatched to the most specific implementation which overrides it, even if the caller is unaware of the existence of the implementation class.

Working but pointless code

This is the level my overloading article worked at. Here, you write code whose sole purpose is to demonstrate the mechanics of the feature you're describing. So in this case we might have:

using System;

public class C1
{
    public virtual void M()
    {
        Console.WriteLine("C1.M");
    }
}

public class C2 : C1
{
    public override void M()
    {
        Console.WriteLine("C2.M");
    }
}

public class C3
{
    static void Main()
    {
        C1 c = new C2();
        c.M();
    }
}

Now this is a reasonably extreme example; as a matter of personal preference I tend to use class names like "Test" or "Program" as the entry point, perhaps "BaseClass" and "DerivedClass" where "C1" and "C2" are used here, and "Foo" instead of "M" for the method name. Obviously "Foo" has no more real meaning than "M" as a name - I just get uncomfortable for some reason around single character identifiers other than for local variables. Arguably "M" is better as it stands for "method" and I could use "P" for a property etc. Whatever we choose, we're talking about metasyntactic variables really.

Complete programs indicative of design in a non-business context

This is the level at which I would probably choose to demonstrate overriding. It's certainly the one I've used for talking about generic variance. Here, the goal is to give the audience a flavour of the purpose of the feature as well as demonstrating the mechanics, but to stay in the simplistic realm of non-business examples. To adapt one of my normal examples - where I'd actually use an interface instead of an abstract class - we might end up with an example like this:

using System;
using System.Collections.Generic;

public abstract class Shape
{
    public abstract double Area { get; }
}

public class Square : Shape
{
    private readonly double side;
    
    public Square(double side)
    {
        this.side = side;
    }
    
    public override double Area { get { return side * side; } }
}

public class Circle : Shape
{
    public readonly double radius;
    
    public Circle(double radius)
    {
        this.radius = radius;
    }
    
    public override double Area { get { return Math.PI * radius * radius; } }
}

public class ShapeDemo
{
    static void Main()
    {
        List<Shape> shapes = new List<Shape>
        {
            new Square(10),
            new Circle(5)
        };
        foreach (Shape shape in shapes)
        {
            Console.WriteLine(shape.Area);
        }
    }
}

Now these are pretty tame shapes - they don't even have a location. If I were really going to demonstrate an abstract class I might try to work out something I could do in the base class to make it sensibly a non-interface... but at least we're demonstrating the property being overridden.

Business-like partial example

Here we'll use classes which sound like they could be in a real business application... but we won't fill in all the useful logic, or worry about any properties that aren't needed for the demonstation.

using System;
using System.Collections.Generic;

public abstract class Employee
{
    private readonly DateTime joinDate;
    private readonly decimal salary;
    
    // Most employees don't get bonuses any more
    public virtual int BonusPercentage { get { return 0; } }
    
    public decimal Salary { get { return salary; } }
    public DateTime JoinDate { get { return joinDate; } }
    
    public int YearsOfService
    {
        // TODO: Real calculation
        get { return DateTime.Now.Year - joinDate.Year; }
    }
    
    public Employee(decimal salary, DateTime joinDate)
    {
        this.salary = salary;
        this.joinDate = joinDate;
    }
}

public abstract class Manager : Employee
{
    // Managers always get a 15% bonus
    public override int BonusPercentage { get { return 15; } }
}

public abstract class PreIpoContract : Employee
{
    // The old style contracts were really generous
    public override int BonusPercentage
    {
        get { return YearsOfService * 2; }
    }
}

Now this particular code sample won't even compile: we haven't provided the necessary constructors in the derived classes. Note how the employees don't have names, and there are no relationships between employees and their managers, either.

Obviously we could have filled in all the rest of the code, ending up with a complete solution to an imaginary business need. Other examples at this level may well include customers and orders. One interesting thing to note here: admittedly I've only been working in the industry for 16 years, and only 12 years full time, but I don't think I've ever written a Customer or Order class as part of my job.

Full application example

No, I'm not going to provide an example of this. Usually this is the sort of thing which a book might work up to over the course of the complete text, and you'll end up with a wiki, or an e-commerce site, or an indexed library of books with complete web site around it. If you think I'm going to spend days or even weeks coding something like that just for this blog post, you'll be disappointed :)

Anyway, the idea of this is that it does something genuinely useful, and you can easily lift whole sections of it into other projects - or at least the design of it.

Which approach is best?

I'm sure you know what's coming here: it depends. In particular, I believe it depends on:

Your readership

Are they likely to copy and paste your example into production code without further thought? Arguably in that case the first option might be the best: they may not understand it, but at least it means your code won't be injuring a project.

Simply put, didactic code is not production code. The parables in the Bible aren't meant to be gripping stories with compelling characterization: they're meant to make a point. Scales aren't meant to sound like wonderful music: they're meant to help you improve your abilities to make a nice sound when you're playing real music.

The point you're trying to put across

If I'm trying to explain the mechanics of a feature, I find the second option to be useful. The reader doesn't need to try to take in the context of what the code is trying to accomplish, because it's explicitly not trying to do anything of any use. It's just demonstrating how the language or platform behaves in a particular scenario.

If, on the other hand, you're trying to explain a design principle, then the third or fourth options are useful. The third option can also be useful for the mechanics of a feature which is particularly abstract - like generic variance, as I mentioned earlier. That goes somewhere between "complete guide to where this feature should be used" and "no guidance whatsoever" - a sort of "here's a hint at the kind of situation where it could be useful."

If you're trying to explore a technology for fun, I find the third option works very well for that situation too. For example, while looking at Reactive Extensions, I've written programs to:

  • Group lines in a file by length
  • Give the results of a UK general election
  • Simulate the 1998 Brazil vs Norway world cup football match
  • Implement drag and drop using event filtering

None of these is likely to be directly useful in a real business app - but they were more appealing than solely demonstrating a sequence of numbers being generated (although with an appropriate marble diagram generator, that can be quite fun too).

The technology you're demonstrating

This is clearly related to the previous point, but I think it bears a certain amount of separation. I believe that language topics are fairly easily demonstrated with the second and third options. Library topics often deserve a slightly higher level of abstraction - and if you're going to try to demonstrate that a whole platform is worth investing time and energy in, it's useful to have something pretty real-world to show off.

Your time and skills

You know what? I suck at the fourth and fifth options here. I can't remember writing any complete, independent systems as a software engineer, and none of them have been in line-of-business applications anyway. The closest I've come is writing standalone tools which certainly have been useful, but often take shortcuts in terms of design which I wouldn't countenance in other applications. (And yes, I'm sure there's some discussion to be had around that as well, but it's not the point of this article.)

You may think my employee example above was lousy - and I'd agree with you. It's not really a great fit for inheritance, in my view - and the bonus calculation is certainly a dubious way of forcing in some polymorphism. But it was the best I could come up with in the time available to me. This wasn't some attempt to make it appear less worthy than the other options; I really am that bad at coming up with business-like examples. Other authors (by which I mean anyone writing at all, not just book authors) may well have found much better examples, either by spending more time on them, being more experienced with line-of-business apps, or having a better imagination. Or all three.

I'm not too proud to admit the things I suck at :) If I spent many extra hours coming up with examples for everything I write about, I would get a lot less written. I'm doing this in notional "spare time" after all. So even if you would prefer the fourth option over the third, would you rather have that but see less of my (ahem) "wisdom"? Personally I think everyone's better off with me braindumping using examples in forms which I'm better at.

How to read examples

Most of this post has been from the point of view of an author. Briefly, I'd like to suggest what this might mean for readers. The onus is on the author to make this clear, of course, but I think it's worth trying to be actively better readers ourselves.

  • Understand what the author is trying to achieve. Don't assume that every example will fit nicely in your application. Example code often doesn't come with any argument validation or error handling - and very rarely does it have an appropriate set of unit tests. If you're reading about how something works, don't assume that the examples are in any way realistic. They may well be simplified to demonstrate the behaviour as clearly as possible without the extra "fluff" of useful functionality.
  • Think about what may be missing, particularly if the context is an evangelical one. If someone is trying to sell you on a particular technology, then of course they'll try to show it in its best possible light. Where are the pitfalls? Where does it not stack up?
  • Don't assume authority means anything. I was quite happy to take Jeffrey Richter to task on boxing for example. Jeffrey Richter is a fabulous author and clearly a smart cookie, but that doesn't mean he's right about everything... and I really, really don't like the idea of anyone appealing to my supposed abilities to justify some bad decision. Judge any argument on its merits... find out what people think and why they think it, but then see how well their reasoning actually hangs together.

Conclusion

This was always going to be a somewhat biased look at this topic, because I hold a certain viewpoint which is clearly contrary to the one held by the chap who emailed me. That's why I included a reasonable chunk of his emails - to give at least some representation to the alternatives. This post has effectively been a longwinded justification of the form my examples have taken... but does it ring true?

I can't guarantee to change my writing style drastically on this front - at least not quickly - but I would very much appreciate your thoughts on this. I'm reluctant to exaggerate, but I think it may be even more important than working out whether "Jedi" was meant to be plural or singular - and I certainly received a lot of feedback on that topic.

How many Jedi?

(There’s no technical content in this post… but you may get a bit of a giggle from it. When I get the second edition web site notes together I’ll include this as well… but I thought it was fun enough to deserve a blog post too.)

The second edition of C# in Depth is nearing the end of its technical review cycle, as performed by the great Eric Lippert. Yesterday I received the comments for chapter 13, which includes this section heading:

The revenge of optional parameters and named arguments

Now, my copy editor (Ben) wasn’t too keen on this. He suggested an alternative form:

I think "have their revenge" has more of a ring to it than "the revenge of"

Personally I’m quite fond of the original, but Eric suggested another alternative, with customary flair:

I’m not buying it Ben. Your way vs Jon’s way:
"The Clones Attack"       / "Attack of the Clones"
"The Sith Have Revenge"   / "The Revenge of the Sith"
"The Empire Strikes Back" / "The Counter-attack of the Empire"
"The Jedi Return"         / "The Return of the Jedi"

I would argue – I have before and I will again – that the proper title for episode two is not the passive, wimpy "Attack of the Clones" but rather the aggressive, dynamic, active "The Clones Attack", preferably with an exclamation point, "The Clones Attack!"

"The Sith Have Revenge" has that awkward verb form. "The Counter-attack of the Empire" is too wordy. And "The Jedi Return" is just… wrong.  So I would score these as the winners being Ben / Jon / Ben / Jon.

I say "the revenge of" is superior to "have their revenge", but that the best would be "Optional and named parameters strike back".

Also, NOOOOOO! You’re not my father!

This intrigued me mightily, so I dashed off an email to Eric:

Hi Eric,

I’m just going through your notes for chapter 13, and they’ve brought up an issue which I think would bother me if I didn’t consult you about it.

You suggested that the alternative to "Return of the Jedi" (1) would be "The Jedi Return." That implies multiple Jedi returning – does this include Anakin returning from the Dark Side? Leia’s nascent ability being revealed? I had always imagine it to only refer to Luke’s return, suggesting "The Jedi Returns" as the parallel title. This could change everything.

Jon

—-

(1) There’s no leading "The" in the English title, as far as I can tell – although in French it’s "Le retour du Jedi." Does this alter your argument at all?

Eric’s reply was as prompt as ever:

First off, you’re right, there’s no leading “The”. I had not realized that.

I had always assumed that the “Jedi” of the title “Return of the Jedi” referred to the beginning of the restoration of the Jedi as an institution. With the downfall of the Emperor and Lord Vader, the Jedi are back. Even though technically there’s only one of them alive in the club right now, there will be more.

However, I must admit that in light of episodes one through three, it now seems plausible that in fact the Jedi referred to in the title is neither the Jedi as a class nor Luke as an individual, but rather the redemption of Anakin.

Beyond the dialogue…

So, that’s the end of that, right? We can’t really tell what Lucas was thinking… except that when I relayed all of this at the office over breakfast, someone suggested that we have a look at some other translations – and that we pay more attention to the French than just the inclusion of "Le" to start with.

The fact that the French version uses "du" suggests it’s the return of a singular Jedi rather than many individual Jedi knights… but apparently the singular form can also be used for a collective institution, in line with Eric’s "Jedi as a class" theory.

The German version is still ambiguous, as far as I can tell interesting: "Die Rückkehr der Jedi-Ritter" – a colleague suggested that this implies knights plural, but "the return of the knight" and "the return of the knights" translate the same way in Google Translate. The fact that "ritter" is both plural and singular (like sheep in English) looks like it foils us. EDIT: As noted in comments, the genetive form would be "des" for a singular knight. So it really is "knights". I was misled by automated translation – I should have trusted my colleague :) But does this mean "the return of several individual Jedi knights" or "the return of an institution of Jedi knights"? Without knowing the subtleties of the German language, I certainly can’t tell for sure.

There’s a whole host of titles of course – if any reader gifted in languages wishes to analyse some more of them, I’d be very grateful. One thing I will point out is the alternative US working title: "Revenge of the Jedi." Who really had their revenge in episode VI? Arguably Luke avenged Han by killing Jabba… and perhaps Anakin himself took revenge on the Emperor? Given that Obi Wan effectively started Luke on the crusade against Vader, perhaps it’s his revenge from beyond the grave?

These are serious matters which I lament I am unable to explore adequately in this post – but comments are more welcome than ever.

Conclusion

So what happened to the heading in the end? Well, for the moment I’ve left it as it is. It may yet change before printing though – we’ll see. Possibly I should take this opportunity to make Eric’s dream change apply in a different context… how about "Attack of the optional parameters and named arguments!" Perhaps not.

Anyway, I’m sure you will all be glad to see that such weighty technical matters are being given the thorough attention they deserve. Yes, the book really will come out sometime reasonably soon.

How do we raise our game?

A couple of weeks ago, I was in the Seattle area for work reasons. I took the opportunity to meet up with a lot of smart folks, including some working on the Reactive Extensions team and the C# team. I asked pretty much the same question of almost everyone:

How is Microsoft intending to make developers smarter?

Let me explain what I mean a bit more clearly…

The problem

Now, let me be quite clear: I only have visibility of a small section of the community. In particular, I get to see:

  • Comments from blog readers
  • Questions and answers on Stack Overflow
  • MSDN forum posts (typically the Code Contracts and Reactive Extensions forums, and then only occasionally)
  • Random emails, often from C# in Depth readers
  • User group and conference attendees

Intuition would suggest that these are some of the more interested developers. Not necessarily the smartest, but ones who are at least engaged with the community. It’s been a while since I’ve worked in a regular company – I don’t think Google engineers are representative of software engineers as a whole – so I’m not going to suggest I have a good grasp of how able the "disengaged" engineers are. I doubt that they’re significantly smarter though.

My concern is that some of the new technologies that I’m getting excited by (in particular Parallel Extensions, Code Contracts and Reactive Extensions) may make it easier for awesome people to write awesome code – but leave everyone else behind. Even though I’m excited about Reactive Extensions, I still regularly get confused by it – this is partly a matter of thin documentation (a natural corollary of an API which is still under development; I’m sure it will improve markedly over time) but partly because a push model is simply harder to think about.

What can we do about it?

Taking Reactive Extensions as an example, how is anyone going to learn about Reactive Extensions to such a degree that they can really use it productively, rather than just for experimentation?

As I see it there are currently five main ways of disseminating information:

User groups and conferences

These are good for getting people interested – but they’re not really great at deep dives. To be absolutely cynical about it, I think they’re better at entertaining than educating. To be slightly more positive, they can be good at inspiring people to look further for themselves… but unless you have a relatively long session or start off with developers who already have a fair amount of experience in the topic, I don’t think they’re a great way of imparting detailed information. (Organisers of such events, please note that this is in no way a condemnation of events in general. I’m still interested in speaking at them – I just question their ability to create experts.)

Training courses

I have little experience of training courses – it could be that they’re highly effective – but they clearly don’t scale in terms of getting information to a wide audience. As training companies make their money by getting people to the courses in person, they’re unlikely to provide videos of the training to let everyone take advantage of an individual tutor’s experience. We can hope for a network effect here and in some of the other options, of course – an expert trains a novice, the novice gains experience and then trains other novices in their company. There’s a risk of truth dilution, but at least this has the possibility of reaching those who would otherwise never voluntarily learn about new technologies.

Blogs

Blogs can be effective, but are difficult to navigate and often outdated. The "navigation" problem is one of picking the right course through the posts in order to try to get a well-rounded knowledge of the subject in an appropriate order.

To give a concrete example, a large amount of the information available in C# in Depth is covered in significantly more detail in Eric Lippert’s blog – which is great for those who already know enough to read it, but it doesn’t provide a 1, 2, 3 experience for learning C# 2-4. (Aside: Over time I’ve been coming to the conclusion that treating a subject "in the round" with a carefully considered ordering is the primary feature of tech books these days – you can almost always find more detailed, accurate and timely information online if you know what you’re looking for.)

I’ve been tremendously impressed at the Microsoft developer blog output in the last few years. There are many well-written, informative and enlightening blogs out there, which can certainly go a long way to filling an educational void. Right at this minute, it feels like they’re probably the most effective teaching tool in this list… but it somehow feels wrong to try to learn a new technology from scratch from a blog. It’s easy to get caught up in details and miss the big picture.

Is there a market for third parties collating blog posts into effective teaching material? I don’t just mean "link blogs" – but well-maintained and organised sequences of posts designed to teach particular topics… possibly with extra explanatory posts to provide extra cohesion. Is there sufficient coverage of basic topics in blogs, or are most bloggers aiming at developers who are already experienced in the topic in question? Answers on a postcard…

Books

Books have problems of freshness and detail, as mentioned before. There’s also a problem of readership – I’m not convinced that books have a wide enough reach these days. I suspect this is largely because blogs do a good enough job in many areas, and also because technology often moves faster than a book market can keep up. Of course this ends up being a cycle to some extent – with fewer and fewer readers, top authors will have more incentive to blog than to write books, so the overall quality could drop, leading to fewer book readers etc. I’m not going to pretend to know enough about the book market to make concrete predictions – but it does concern me.

(I’m not particularly bothered in terms of my own income – I’ve never been writing C# in Depth for the money, although of course that provides more incentive for "polish". I wouldn’t have put in as much editorial work in a purely amateur capacity. There’s a very strange motivational force going on when it comes to tech writing and publishing. That’s possibly a seed for another post some time.)

Will there be a book teaching how to think about push sequences and use Reactive Extensions effectively? Maybe. Heck, I’d potentially be interested in writing one – if only to understand it better myself. Would it reach all the people who could benefit from it though? I’m more dubious about that.

Documentation and tutorials

This is probably the most traditional and widespread route to knowledge of new technologies. Does it cut it? I simply don’t know. I can’t remember the last time I read docs in a "from scratch" mode – my experience is much more in reference mode. I’ve occasionally tried – but found either my patience or the documentation to be lacking. It’s entirely possible that my expectations and methods of learning have changed, and that that’s what’s gone wrong… but this may be a reasonably common phenomenon.

Again there’s a problem of reach – although it’s possible that as the most traditional form of learning, it’s possible that it has a greater reach into the non-community, if you see what I mean.

Non-conclusion

This has been somewhat rambly, partly because I’m demob-happy: I’m about to go to Athens on holiday with Holly for a few days. (Don’t expect any comments to be approved until Sunday.) It’s a real concern though, and one which goes way beyond the Microsoft technologies mentioned here.

The challenges faced in computing are growing, and so are the technologies trying to up us meet those challenges… but we need to grow too, in order to take advantage of what’s available.

I’m not pronouncing doom on the industry – but I’d like your thoughts on how we can keep up, and how we can help others to do likewise.

Just how lazy are you?

I’ve been reviewing chapter 10 of C# in Depth, which is about extension methods. This is where I start introducing some of the methods in System.Linq.Enumerable, such as Where and Reverse. I introduce a few pieces of terminology in callouts – and while I believe I’m using this terminology correctly according to MSDN, I suspect that some of it isn’t quite what many developers expect… in particular, what does it mean for something to be "lazy"?

Let’s start off with a question: is Enumerable.Reverse lazy? Just so we’re all on the same page, here are the interesting bits of the behaviour of Reverse:

  • The call to Reverse doesn’t fetch any items – it merely checks that you’ve not given it a null sequence, stores a reference to that sequence, and returns a new sequence.
  • Once the first item is returned from the returned sequence, all the items in the original sequence are fetched and buffered. Obviously this is required, as the first item in the reversed sequence is the last item in the original sequence.

So, is that lazy or not?

One simple definition of lazy

This morning I tweeted on the somewhat ambiguous notion of something being "lazy" – and the responses I received were along the lines of "it’s deferred execution". You could potentially sum up this notion of laziness as:

An operation is lazy if it defers work until it is required in order to return a result.

By that definition, Reverse is lazy. Assuming we don’t want to perform special optimisations for IList<T> (which change the exact behaviour), Reverse does as little work as it can – it just so happens that when the first item is requested, it has to drain the source sequence.

The MSDN definition of lazy

MSDN describes deferred execution, lazy evaluation and eager evaluation somewhat differently. Admittedly the page I’m taking these definitions from is in the context of LINQ to XML, but that effectively means it’s describing LINQ to Objects. It defines them like this:

Deferred execution means that the evaluation of an expression is delayed until its realized value is actually required.

In lazy evaluation, a single element of the source collection is processed during each call to the iterator. This is the typical way in which iterators are implemented.

In eager evaluation, the first call to the iterator will result in the entire collection being processed. A temporary copy of the source collection might also be required. For example, the OrderBy method has to sort the entire collection before it returns the first element.

Now, I take slight issue with the definition of lazy evaluation here as it specifies that a single element of the source collection is processed on each call. That’s fine for Cast, OfType, Select and a few other operators – but it would preclude Where, which might have to pull several source elements before it finds one which matches the specified filter. I still think of Where as being lazy.

My definition of lazy

Thinking about this more, I believe the following definition of laziness is helpful:

(This space left intentionally blank.)

I don’t believe lazy is a useful term, basically. It’s like "strong typing" – you get some sort of woolly feeling about how something will behave if it’s lazy, but chances are you’ll need something more precise anyway.

For the purposes of LINQ to Objects-style operators which return IEnumerable<T> or any interface derived from it (including IOrderedEnumerable<T>), I propose the following definitions:

An operation uses deferred execution if no elements are fetched from the source sequence until the first element is fetched from the result sequence. This applies to basically all LINQ to Objects operators which return a sequence interface. (It doesn’t apply to ToArray or ToList of course.)

An operation is streaming if it only fetches elements from the source sequence as it requires them, and does not store those elements internally unless otherwise specified.

An operation is buffering if it drains the source sequence (i.e. fetches all the elements) when the first element of the result sequence is fetched, and stores the items internally.

I’m not even entirely comfortable with this: you could claim that Reverse is "streaming but with internal storage" – but that’s against the spirit of the definitions. Why did I mention the internal storage at all? Well, consider Distinct… that streams data in that it will the result sequence will return the first element immediately after reading the first element from the source sequence, and so on – but it has to remember all the elements it’s already returned, for obvious reasons.

Some operations are buffering in one input and streaming in another – for example, Join will read all of the "inner" sequence as soon as it’s asked for its first element, but streams the "outer" sequence.

Why does this matter?

Is this just another example of my pedantry and desire for precision? Not really. Consider my old favourite example of LINQ: processing huge log files. Suppose each log entry contains a user ID, and we’ve got a handy log reader which will iterate over all the log files, yielding one log entry at a time.

  • Using entries.Reverse() would be disastrous if we ever actually used the result. We really, really don’t want to load everything into memory.
  • Using entries.Select(x => x.UserId) would be fine, so long as we used the result without buffering it ourselves.
  • Using entries.Select(x => x.UserId).Distinct() might be fine – it would depend on how many users we have. If you’re processing some company-internal application logs, that’s probably okay even if you’ve generated a huge number of log entries. If you’re processing FaceBook logs, you could have more of a problem.

Basically, you need to know what an operation will do with its input before you can decide whether or not it’s usable for a particular situation.

The best answer for this (IMO) is to document any such methods meticulously. Yes, you then lose the flexibility of changing the behaviour at a later date – but at least you’ve then provided something that can be used with confidence.

Note that Reactive Extensions has a similar problem, but in a slightly different form – in that case, the distinction between "hot" and "cold" observables can make a big difference, along with scheduling etc. Again, documentation is the key in my view.

Optimisations in LINQ to Objects

(Edited on February 11th, 2010 to take account of a few mistakes and changes in the .NET 4.0 release candidate.)

I’ve just been fiddling with the first appendix of C# in Depth, which covers the standard query operators in LINQ, and describes a few details of the LINQ to Objects implementations. As well as specifying which operators stream and which buffer their results, and immediate vs deferred execution, I’ve where LINQ optimises for different collection types – or where it doesn’t, but could. I’m not talking about optimisations which require knowledge of the projection being applied or an overall picture of the query (e.g. seq.Reverse().Any() being optimised to seq.Any()) – this is just about optimisations which can be done on a pretty simple basis.

There are two main operations which can be easily optimised in LINQ: random access to an element by index, and the count of a collection. The tricky thing about optimisations like this is that we do have to make assumptions about implementation: I’m going to assume that any implementation of an interface with a Count property will be able to return the count very efficiently (almost certainly straight from a field) and likewise that random access via an indexer will be speedy. Given that both these operations already have their own LINQ operators (when I say "operator" in this blog post, I mean "LINQ query method" rather than an operator at the level of C# as a language) let’s look at those first.

Count

Count() should be pretty straightforward. Just to be clear, I’m only talking about the overload which doesn’t take a predicate: it’s pretty hard to see how you can do better than iterating through and counting matches when there is a predicate involved.

There are actually two common interfaces which declare a Count property: ICollection<T> and ICollection. While many implemenations of ICollection<T> will also implement ICollection (including List<T> and T[]), it’s not guaranteed: they’re independent interfaces, unrelated other than by the fact that they both extend IEnumerable.

The MSDN documentation for Enumerable.Count() states:

If the type of source implements ICollection<T>, that implementation is used to obtain the count of elements. Otherwise, this method determines the count.

This is accurate for .NET 3.5, but in .NET 4.0 it does optimise for ICollection as well. (In beta 2 it only optimised for ICollection, skipping the generic interface.)

ElementAt

The equivalent of an indexer in LINQ is the ElementAt operator. Note that it can’t really be an indexer as there’s no such thing as an "extension indexer" which is arguably a bit of a pity, but off-topic. Anyway, the obvious interface to look for here is IList<T>… and that’s exactly what ElementAt does. It ignores the possibility that you’ve only implemented the nongeneric IList – but I think that’s fairly reasonable. After all, the extension method extends IEnumerable<T>, so your collection has to be aware of generics – why would you implement IList but not IList<T>? Also, using the implementation of IList would involve a conversion from object to T, which would at the very least be ugly.

So ElementAt doesn’t actually do too badly. Now that we’ve got the core operations, what else could be optimised?

SequenceEqual

If you were going to write a method to compare two lists, you might end up with something like this (ignoring nullity concerns for the sake of brevity):

public static bool ListEqual<T>(List<T> first, List<T> second, IEqualityComparer<T> comparer)
{
    // Optimise for reflexive comparison
    if (first == second)
    {
        return true;
    }
    // If the counts are different we know the lists are different
    if (first.Count != second.Count)
    {
        return false;
    }
    // Compare each pair of elements in turn
    for (int i = 0; i < first.Count; i++)
    {
        if (!comparer.Equals(first[i], second[i]))
        {
            return false;
        }
    }
    return true;
}

Note the two separate optimisations. The first is always applicable, unless you really want to deal with sequences which will yield different results if you call GetEnumerator() on them twice. You could certainly argue that that would be a legitimate implementation, but I’d be interested to see a situation in which it made sense to try to compare such a sequence with itself and return false. SequenceEqual perform this optimisation.

The second optimisation – checking for different counts – is only really applicable in the case where we know that Count is optimised for both lists. In particular, I always make a point of only iterating through each source sequence once when I write a custom LINQ operator – you never know when you’ll be given a sequence which reads a huge log file from disk, yielding one line at a time. (Yes, that is my pet example, but it’s a good one and I’m sticking to it.) But we can certainly tell if both sequences implement ICollection or ICollection<T>, so it would make sense to have an "early negative" in that situation.

Last(predicate)

(All of this applies to LastOrDefault as well, by the way.) The implementation of Last which doesn’t take a predicate is already optimised for the IList<T> case: in that situation the method finds out the count, and returns list[count – 1] as you might expect. We certainly can’t do that when we’ve been given a predicate, as the last value might not match that predicate. However, we could walk backwards from the end of the list… if you have a list which contains a million items, and the last-but-one matches the predicate, you don’t really want to test the first 999998 items, do you? Again, this assumes that we can keep using random access on the list, but I think that’s reasonable for IList<T>.

Reverse

Reverse is an interesting case, because it uses deferred execution and streams data. In reality, it always takes a complete copy of the sequence (which in itself does optimise for the case where it implements ICollection<T>; in that situation you know the count to start with and can use source.CopyTo(destinationArray) to speed things up). You might consider an optimisation which uses random access if the source is an implementation of IList<T> – you could just lazily yield the elements in reverse order using random access. However, that would change behaviour. Admittedly the behaviour of Reverse may not be what people expect in the first place. What would you predict that this code does?

string[] array = { "a", "b", "c", "d" };

var query = array.Reverse();
array[0] = "a1";
        
var iterator = query.GetEnumerator();
array[0] = "a2";
        
// We’ll assume we know when this will stop :)
iterator.MoveNext();
Console.WriteLine(iterator.Current);
array[0] = "a3";
        
iterator.MoveNext();
Console.WriteLine(iterator.Current);
array[0] = "a4";

iterator.MoveNext();
Console.WriteLine(iterator.Current);
array[0] = "a5";
        
iterator.MoveNext();
Console.WriteLine(iterator.Current);

After careful thought, I accurately predicted the result (d, c, b, a2) – but you do need to take deferred execution *and* eager buffering into account. If nothing else, this should be a lesson in not changing the contents of query sources while you’re iterating over the query unless you’re really sure of what you’re doing.

With the candidate "optimisation" in place, we’d see (d, c, b, a5), but only when working on array directly. Working on array.Select(x => x) would have to give the original results, as it would have to iterate through all the initial values before finding the last one.

LongCount

This is an interesting one… LongCount really doesn’t make much sense unless you expect your sequence to have more than 2^31 elements, but there’s no optimisation present. The contract for IList<T> doesn’t state what Count should do if the list has more than Int32.MaxValue elements, so that can’t really be used – but potentially Array.LongValue could be used for large arrays.

A bigger question is when this would actually be useful. I haven’t tried timing Enumerable.Range(0, int.MaxValue) to see how long it would take to become relevant, but I suspect it would be a while. I can see how LongCount could be useful in LINQ to SQL – but does it even make sense in LINQ to Objects? Maybe it will be optimised in a future version with ILongList<T> for large lists…

EDIT: In fact, given comments, it sounds like the time taken to iterate over int.MaxValue items isn’t that high after all. That’ll teach me to make assumptions about running times without benchmarking… I still can’t say I’ve seen LongCount used in anger in LINQ to Objects, but it’s not quite as silly as I thought :)

Conclusion

The optimisations I’ve described here all have the potential to take a long-running operation down to almost instantaneous execution, in the "right" situation. There may well be other opportunities lurking – things I haven’t thought of. The good news is that missing optimisations could be applied in future releases without breaking any sane code. I do wonder whether supporting methods (e.g. TryFastCount and TryFastElementAt) would be useful to encourage other people writing their own LINQ operators to optimise appropriately – but that’s possibly a little too much of a niche case.

Blog post frequency may well change in either direction for the near future – I’m going to be very busy with last-minute changes, fixes, indexing etc for the book, which will give me even less time for blogging. On the other hand, it can be mind-numbingly tedious, so I may resort to blogging as a form of relief…