You should be OK

as firewall software makes your PC "invisible" unless you initiate communication or you have a router which does not run firewall software. NAT on router is not good enough .
Firewall software will discard intrusive connections so other side gets no response and usually will think it's an unreachable IP.
If you are pretty sure it's an attack instead of other status polling (maybe you are running gnutella client, etc.), report it to other party's ISP. You can use whois service to find other IP's owner.