I can move an element by setting its Offset but it doesn't know what offset originally was
Of course it know, the original offset is 0. You haven't set the offset when you created the visuals, why would it be anything else?
If I click on a rectangle, is there a way to get the 'System.Windows.Media.RectangleGeometry' out of the DrawingVisual?
Sort of. A DrawingVisual has a Drawing property of type DrawingGroup. This has a Children property and you can go through all the children and on children of type GeometryDrawing you have a Geometry property. The Geometry object has a number of methods that can be used for hit testing.
Can reach its Rect and pen or brush without removing them and recreating them?
Nope, those are read only. But you can do things a bit differently. Start by creating the needed geometries, for example 2 EllipseGeometry. Then draw them on the drawing context by using DrawGeometry instead of DrawEllipse. Now you can modify the properties of those geometries, for example you can apply a transform to them.
And I still haven't figured out how to remove objects.
Nope but the above "tweak" applies here to. You can start by creating the necessary geometries, put them all in a DrawingGroup object and the draw the drawing group on the drawing context by using DrawDrawing. Something like this:
Dim s As New DrawingVisual()
Dim rg As New RectangleGeometry(new Rect(0, 0, 100, 100))
Dim rgd As New GeometryDrawing(Brushes.Pink, Nothing, rg)
Dim dg As New DrawingGroup()
Using dc As DrawingContext = s.RenderOpen()
rgd.Brush = Brushes.Yellow
rg.Transform = new RotateTransform(42)
But now the question raises, why don't you simply use a Canvas and some shapes?