You might try (depending on platform that you have PBR configured on) to configure
"ip route-cache policy" on the interface with the policy configured on. You might be running into Process switching problems with your PBR setup.
Some more information as to the Hardware you are using, pbr config, IOS version might help more as well.
Hope this helps,
Don