Your cat may be smelling another cat. Cats mark everything and brushing against a bush or sitting at a bench that has been marked can cause your cat to go a bit nuts, especially if your cat is unfixed. Or your cat is smelling your scent on the wet towel. Try using a small blanket for a bit and then let the cat have it. Your scent (or the scent of anyone he trusts) will give him comfort and help him to calm down. I also recommend that you get your cat fixed (if you haven't already). Unfixed cats will start a territorial war if they can smell another unfixed cat on you or your clothes. Also, be sure your cat can see out the window. This will give him a place to safely see the outside.