I've tried turning off reliable session, but the error remains.
I use net tcp binding which is automatically duplex channel.
when I send the message directly (not within the callback on the duplex channel) the message was sent and received with no error.
I think that maybe when the other way of the duplex is sending the message - the size is limited... but it's only a speculation.
The most likely thing to think when using duplex is that both channels would have same message size limitation isn't is?