For incoming packet on PGW for a UE means, a valid PDP Context must exist on it. When UE attaches to APN, it's PDP context is stored on MME and P-GW. This context info contains several info e.g. UE IP Address, APN , IMSI, MSISDN, Charging ID and peer & local end TEID. This TEID is kept for both local and peer end till the UE is detached from network. So once tunnel is setup between SGW & PGW it's not deleted even when UE goes IDLE.
P-GW is selected by MME during attach process and SGW setups Tunnel to destination PGW. So for PGW, SGW is always known for a UE (as long it remains attached with network)