1

Trying to introduce caching/state-persisting into the middle tier of an application with a complex object layer (and a WCF service layer, not the focus here) running under IIS. Have settled on memcached/enyim as the caching architecture, and now need to get these objects serialized efficiently (speed and space) for it.

The object layer has a lot of pointers and interdependencies among objects, along these lines:

internal class SomeObj
{
    private string attr1;
    private int attr2;
    private OtherObj otherObj;
    private List<OtherOtherObj> otherObjs;
}

internal class OtherObj
{
    //...more attributes
}

internal class OtherOtherObj
{
    // you get the idea
}

Note that all fields are private. Also worth noting, most objects are internal, and many properties are either read-only (no set) or use set from a user perspective (i.e. make an object "dirty"), so cannot be used in rehydration. I have dozens of types that need caching.

I like the looks of both protobuf-net and msgpack, but I need to get the serialization done as fast as possible, with as little alteration of the existing architecture (which works well as is) as possible, and it seems like both of these have limited support for object hierarchies. I understand DTO-type serialization well, but am new to planning the right way to serialize an object for a cache. Can one of these tools work for me? Am I stuck using built-in .NET binary in order to work with a constructor, and repopulate attributes and objects on my own terms?

EDIT: just to clarify that last question--might make more sense if it read "Am I stuck using built-in .NET binary so that I can control the constructor, and repopulate attributes and objects on my own terms?

downwitch
  • 1,362
  • 3
  • 19
  • 40

1 Answers1

1

I can't comment on msgpack, but yes: protobuf-net can do everything you've mentioned, including:

  • serialize non-public types
  • serialize private fields
  • serialize trees
  • constructor-skipping or user-provided factory (or just parameterless constructor)
  • serialize full/cyclic graphs (by explicitly marking the affected as references)
  • attribute usage, or runtime configuration without changing the DTO at all (although frankly, attributes are usually easier)
  • fast and compact output
  • inheritance support

In the case of the example given, the simplest "does it work" test would be to just make the types with [ProtoContract] (there's an optional setting in that attribute for constructor-skipping), and mark the fields as [ProtoMember(n)], for example n=1,2,3,... (unique in each type, but does not need to be unique between types)

Other than the fact that we use Redis+BookSleeve rather than memcached+enyim, this is exactly what we do at Stack Exchange for object caching. Well, we also do a speculative gzip for large objects - if there is lots of string data, gzip can help shave off a few extra bytes.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • thanks, I did a fair amount of reading about Redis too, didn't look as Windows-service simple to spin up to me. I also looked at a few of simple(r) protobuf-net examples, and it seemed very WCF-like, which I'm comfortable with but only as a DTO/POCO object "builder", which seems different. Just to make sure I understand: best-practice protobuf-net in a case like this to serialize the fields, rather than the properties? And then it's as simple as wrapping the calls to add/get to/from the cache in serialize/deserialize methods? – downwitch Nov 25 '12 at 16:25
  • @downwitch it'll work five with either fields or properties. Normally I work with the props, but you mentioned not wanting to do that because of "dirty" checks etc. But: yes, as simple as that – Marc Gravell Nov 25 '12 at 17:04
  • @downwitch protobuf-net can be used with WCF, but that is just one scenario - most of the time I would expect it to be used in isolation – Marc Gravell Nov 25 '12 at 17:22
  • Sorry, one more time, my question wasn't clear... With a DataContract, every DataMember must have get and set accessors, and in the process of rehydration the set accessors all get called. I assume then that a ProtoContract follows the same logic? e.g. if an encapsulated field's property is read-only, the underlying field is not set on rehydration and remains its default value? Then I assume you've worked some other magic to get private fields to work as ProtoMembers, so as you point out may be cleaner in my case anyway. – downwitch Nov 25 '12 at 19:32
  • @downwitch if you mark the field as the [ProtoMember], then it accessed the field directly. If you mark the property, it indeed uses the get and set accessors (note that a set is not always required, for example a list). Don't worry about `readonly` - it bypasses that - that is mainly for compilers. For info, everything I've said applies equally to a DataContract, btw - DataContractSerializer can *also* access private fields, ignore `readonly`, and bypass constructors. – Marc Gravell Nov 25 '12 at 19:49
  • Interesting about DataContractSerializer, I guess I've been systematically making encapsulated fields DataMembers for so long, I decided my own practice was a rule :x. Excited to dig in to pb-n (and loving the ease of integration with enyim). Will leave this open another day in case other ideas materialize, but I really appreciate all the help straight from the source. – downwitch Nov 25 '12 at 19:54
  • @downwitch note that for convenience, I have now created `protobuf-net.Enyim`, which adds a transcoder... see this related question: http://stackoverflow.com/q/13646827/23354 – Marc Gravell Dec 02 '12 at 00:29