"IllegalStateException: Unknown pdx type=2" when all nodes get restarted while client stays up

Hi

I am using a program based on the native client 7.0.1 (.net) to execute functions on a GemFire cluster. I tried to verify a couple of failure scenarios and all works fine except for one scenario, where the minimal case would look like this:

1. Start GF1 (gemfire node 1)
2. Start GF2
3. Start Client
4. Let Client execute a function on GF cluster
5. Kill both GF1 and GF2 (so there's a time period when the GemFire cluster is completely down)
6. Start GF1
7. Start GF2
8. Let Client execute a function on GF cluster

It's at step 8 when I get the exception.

Btw this is an HA function (which is why I use 2 servers even for the minimal case), I have not tried a non-HA function.

I tried several other things. When I e.g. omit step 4 above, everything works fine. Or, if instead of step 8 I do a region.Put on the same region, using as value an instance of the same class as the one I'm passing in to the function as a parameter (and which I assume is causing the exception in GF), it works just fine. And even after a successful region.Put (i.e. proof of a successful serialization/deserialization of the class in question), a function execution will still fail.

However, if I make sure that at least one GemFire node stays up at all times, like for example:

1. Start GF1 (gemfire node 1)
2. Start GF2
3. Start Client
4. Let Client execute a function on GF cluster
5. Kill GF1
6. Start GF1
7. Kill GF2
8. Start GF2
9. Let Client execute a function on GF cluster

everything works flawlessly.

So here's an example of the exception pair I'm getting (it comes in pairs, one pair on each GemFire node):

[warning 2013/04/11 13:28:35.773 JST GF1 <ServerConnection on port 40401 Thread 0> tid=0x45] Server connection from [identity(myhostname(default_GemfireDS:4784:loner):2:GFNative_MGihOBTGfg:d

efault_GemfireDS,connection=1; port=59317]: Unexpected Exception

java.lang.IllegalStateException: Unknown pdx type=2

        at com.gemstone.gemfire.internal.InternalDataSerializer.readPdxSerializable(InternalDataSerializer.java:2976)

        at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2793)

        at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3212)

        at com.gemstone.gemfire.DataSerializer.readArrayList(DataSerializer.java:2232)

        at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2687)

        at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3212)

        at com.gemstone.gemfire.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:81)

        at com.gemstone.gemfire.internal.cache.tier.sockets.CacheServerHelper.deserialize(CacheServerHelper.java:54)

        at com.gemstone.gemfire.internal.cache.tier.sockets.Part.getObject(Part.java:216)

        at com.gemstone.gemfire.internal.cache.tier.sockets.Part.getObject(Part.java:220)

        at com.gemstone.gemfire.internal.cache.tier.sockets.command.ExecuteRegionFunction66.cmdExecute(ExecuteRegionFunction66.java:90)

        at com.gemstone.gemfire.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:173)

        at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:809)

        at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:940)

        at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1189)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at com.gemstone.gemfire.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:532)

        at java.lang.Thread.run(Thread.java:722)

[warning 2013/04/11 13:28:35.781 JST GF1 <ServerConnection on port 40401 Thread 1> tid=0x49] Server connection from [identity(myhostname(default_GemfireDS:4784:loner):2:GFNative_MGihOBTGfg:d

efault_GemfireDS,connection=1; port=59319]: Unexpected Exception

java.lang.IllegalStateException: Unknown pdx type=2

        at com.gemstone.gemfire.internal.InternalDataSerializer.readPdxSerializable(InternalDataSerializer.java:2976)

        at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2793)

        at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3212)

        at com.gemstone.gemfire.DataSerializer.readArrayList(DataSerializer.java:2232)

        at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2687)

        at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3212)

        at com.gemstone.gemfire.internal.util.BlobHelper.deserializeBlob(BlobHelper.java:81)

        at com.gemstone.gemfire.internal.cache.tier.sockets.CacheServerHelper.deserialize(CacheServerHelper.java:54)

        at com.gemstone.gemfire.internal.cache.tier.sockets.Part.getObject(Part.java:216)

        at com.gemstone.gemfire.internal.cache.tier.sockets.Part.getObject(Part.java:220)

        at com.gemstone.gemfire.internal.cache.tier.sockets.command.ExecuteRegionFunction66.cmdExecute(ExecuteRegionFunction66.java:90)

        at com.gemstone.gemfire.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:173)

        at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:809)

        at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:940)

        at com.gemstone.gemfire.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1189)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at com.gemstone.gemfire.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:532)

        at java.lang.Thread.run(Thread.java:722)

 

The relevant parts of my server-side cache.xml:

<cache>    <cache-server port="40401"/>    <pdx>        <pdx-serializer>            <class-name>com.gemstone.gemfire.pdx.ReflectionBasedAutoSerializer</class-name>            <parameter name="classes">                <string>proto.server.data.QueueState,proto.server.data.Session,proto.data.UpstreamMessage,proto.data.DownstreamMessage</string>            </parameter>        </pdx-serializer>    </pdx>    <region name="Queue">        <region-attributes refid="PARTITION">            <partition-attributes redundant-copies="1">                <partition-resolver>                    <class-name>proto.server.partition.QueuePartitionResolver</class-name>                </partition-resolver>            </partition-attributes>        </region-attributes>    </region>

...

And the class that's being serialized in the function executions (and the region.Put) is proto.data.UpstreamMessage.

Any suggestions, workarounds etc. appreciated!

Cheers

Eugen