I solved the problem:
I'll give some background - I work with two different endpoints "A" and "B". "A" is for the calls and the "B" is for events registration. The client asks asynchronously for the object from the server in "A" and subscribes to event on "B". When the server done calculating - it returns it on "B".
First I created a new function at the server that is not async and returns the object directly and it worked. So I figured out the problem is in the definition of the callback channel ("B").
I set the maxReceivedMessageSize to maximum on the client side at the callback endpoint as well. I think that because the channel is duplex I needed to specify it on each way... (reasonable isn't it..?
thank you all for your help!