The point of the factory pattern is not to restrict access to creating the objects without the factory.  It is simply to allow the creation of objects without specifying the class of the object you want to create.

The implementation you provided was just fine.