I SOLVED IT !!! here is the event handler:

   Protected Overrides Sub OnMouseLeftButtonDown(e As MouseButtonEventArgs)
      MyBase.OnMouseLeftButtonDown(e)
      Dim location As Point = e.GetPosition(Me)
      Dim result As HitTestResult = VisualTreeHelper.HitTest(Me, location)
      Dim hitVisual As DrawingVisual = TryCast(result.VisualHit, DrawingVisual)
      If result.VisualHit.[GetType]() = GetType(DrawingVisual) And Not hitVisual.Equals(visBackground) Then
         visuals.Remove(hitVisual)
         RemoveVisualChild(hitVisual)
         RemoveLogicalChild(hitVisual)
      End If
   End Sub

Duh: RemoveVisualChild and RemoveLogicalChild. I test for the background to prevent it being deleted. If I want to change the color or other feature of the object, I can delete it and draw again. I don't think the deletion could be noticed. I am working on a generalized example to post. Thanks for helping me on this. Dexter, I'm glad you did that test. Ill have to play around more with this.