Yesterday the Google open source blog announced a new open source project, to release one of the core pieces of Google infrastructure: protocol buffers. Basically protocol buffers are a way of encoding structured data in a language-neutral and versioning-friendly fashion. Yes, there are a lot of similar ways of doing similar things, but this happens to be the one we use.
Currently the protocol buffer compiler (which takes .proto files and converts them into source for various languages) supports C++, Java and Python. I’m hoping to add C# to that list reasonably quickly. To start with this will just be in my spare time rather than as a 20% project, but who knows…
It’s really nice to work in an environment where the question of “is there any reason we can’t open source this?” crops up quite frequently. I’m looking forward to the day when a piece of code I’ve developed at Google ends up as part of an open source project.
Jon Skeet's coding blog 
So, you think it’s OK that Google is competing against an already open standard?
LikeLike
Which open standard are you thinking of in particular?
Google has had protocol buffers for ages – they’re deeply embedded throughout the infrastructure. Changing to another system certainly wouldn’t be an option.
So, given the choice between opening up the code or just not doing so but still using it internally, I see no problem with opening it up.
LikeLike
Would be nice if it was possible to use it in GWT.
What’s the name of that standard Rod?
LikeLike
I took a quick look at the linked site and it looks like this could come in quite handy in a number of scenarios. I wonder how far one could get if trying to integrate it into things like the .net serialization infrastructure and/or WCF. It might be interesting to see just how far that could go.
LikeLike
Interesting; is part of the intent that the binary serialization is compatible between implementations? (this isn’t mentioned in the overview docs). If so, it would have some advantages over BinaryFormatter (being .NET specific, and really, really annoying at times)… Presumably it would require a bespoke formatter implementation [or something more similar to the existing java etc approaches], and a proto parser (presumably a “Custom Tool” from VS’s perspective).
I wonder, however, how it would compare to JSON etc, which combines “simple” and “portable” (although obviously fatter than raw binary).
But if you want any help with the C# version drop me a line ;-p
LikeLike
Of course another thought is how to make it play nicely with other techologies (like LINQ) – whether that means composing the types (the db type and the proto type), or allowing simple conversion between them. Or maybe (as per Jeremy’s note) a WCF plugin would allow proto use in the comms layer without placing restrictions on the types…
If I get time I’ll try and see how it is *actually* used in the java etc scenarios…
LikeLike
Yes, it should be 100% binary compatible between implementations.
Compared with JSON: yes, it’s slimmer and probably faster than JSON – but not human readable in native format, of course. I don’t know much about JSON, but protocol buffers have built in knowledge of optional/required fields and typing so there’s a bit more going on there than in JSON (I think – I could be wrong).
I’m hoping that the C# version will be ready for a first peek in about a week, barring big issues. I’m currently working on the supporting library before going back to the code generation.
Jon
LikeLike
Re JSON; in particular, I was thinking of current .NET implementation, which is based around data-contracts (.NET 3.0). Indeed, since data-contracts themselves know about such things as optional/required, defaults, full type info, etc – and also supports positional layout [which from what I can see [in a very, very brief look, is what proto seems to use]. I don’t know what you’ve cooked up, but something data-contract oriented might be worth consideration, giving a good level of interplay with other formatters. Just a thought.
LikeLike
Okay. I’ll certainly start off by just porting the Java code in a .NET-idiomatic way, but then it’s quite possible that there are nice layers which could be added on top of that.
LikeLike