Sure, but when working in the unsafe arena I tend to prefer working in c++.
Working with 32bpp shouldn't be a big deal, just change your byte[] to a int[]. But I agree that recopying the whole buffer for every frame is probably what’s slowing you down.