LINQ to Objects – not just for in-memory collections

I’ve just seen LINQ to Objects described as the LINQ provider for “in-memory collections” again. It’s a fairly frequent occurrence, and I may have done it myself on occasion. It doesn’t do LINQ to Objects justice. An example I’ve used in a few places is a query which runs over log files. Something along the lines of:

var query = from file in Directory.GetFiles(@“c:logs”, “*.log”)
            from line in new LineReader(file)
            let entry = new LogEntry(line)
            where entry.Severity = Severity.Critical
            select entry;

Where’s the in-memory collection here? I suppose there’s the array of log file names, but that’s about it. LINQ to Objects isn’t restricted to datasets which fit comfortably in memory. The above query could process many gigs of data very easily, limited basically by disk speed (and date/time parsing speed in my experience, but that’s a topic for another post).

What LINQ to Objects does is in-process querying of enumerable sequences of data. More of a mouthful than “querying in-memory collections” but more accurate, IMO.

Rant over. Well, for a couple of minutes.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s