It depends how they opened the socket.
For example, in C# you can get either behaviour:
// This form is loopback only and not visible from outside of your machine:
listener = new TcpListener(new IPEndPoint(IPAddress.Loopback, port));
// this form exposes the port:
listener = new TcpListener(new IPEndPoint(IPAddress.Any, port));
Assuming you're running Windows Firewall, you'll get a prompt with the latter to ask you if you want to add an exception to punch a hole through the firewall. If your program gave you a Windows Firewall prompt, then you might want to make a feature request to the product team to allow you disable it.
If you didn't see a prompt, you're probably safe.
