It depends how they opened the socket.

For example, in C# you can get either behaviour:

// This form is loopback only and not visible from outside of your machine:

listener = new TcpListener(new IPEndPoint(IPAddress.Loopback, port));

// this form exposes the port:

listener = new TcpListener(new IPEndPoint(IPAddress.Any, port));

Assuming you're running Windows Firewall, you'll get a prompt with the latter to ask you if you want to add an exception to punch a hole through the firewall. If your program gave you a Windows Firewall prompt, then you might want to make a feature request to the product team to allow you disable it.

If you didn't see a prompt, you're probably safe.