It is typical of any child, but especially one this age, to be sensitive to changes in his environment, changes in his caregiver, changes in his schedule. Changes. Period. At this age, for instance, he doesn't have the cognitive skills to understand that mom's absence doesn't mean mom is gone forever, yet he is aware enough to know that he's in a different place, with different people, perhaps people he doesn't know or remember, and to wonder: where is mom? Is she coming back? So that's his fear: You are out of sight, you are gone, you may not be coming back. Consider that the the foundation he's working from.
Here are some variables that could be making this even harder for him.
What kind of experiences does he have when he is with dad? Does dad know about and pay attention to his nap and sleep and eating schedule? Is dad patient with him? Does he take care of him himself, or does he drop him off with a relative, introducing yet another change in caregiver?
How does he get to dad's? What is the transition like for him, going from mom to dad? When you and his father see each other, are you friendly and kind to each other, or terse and nasty? And what about when his dad brings him back to you (or you pick him up)? Is there a pleasant exchange of information, or is the vent somewhat uncivil? This is going to be a long haul for your son; the more you can maintain a good relationship with his father, the better it will be for him.
What's even more important to keep in mind is that, short of finding out that his father is abusive or neglectful, none of this means that he shouldn't go with his father; it's important for him to have both his parents in his life. Meanwhile, here are some ways to help him:
Don't push him away when he's clinging to you. Be patient. Give what he needs. Think of each separation as a chance for you to re-establish your credibility with him: Mom is here, mom will give you what you need. Let him get his need for you out of his system, so take your cues from him. If you push him away, thinking that will toughen him up, that's scary for him and will make him even more clingy and dependent.
Talk to him about his daddy during the time they are apart. Have a picture of the two of them in a prominent place so that dad is a constant presence in his life, not just someone who appears out of the blue.
Make a book for him: "My visits with my dad." For instance: "Tomorrow, Mike is going to visit his daddy." On the next page, mommy helps him get his favorite pj's ready, and his favorite stuffed animals, etc. Have a page about the trip to dad's whatever it is. Make other pages about dad's house, the activities they do together, the people he sees there, the trip back to mom, the reunion with mom. End with a goodbye to dad, "See you next time!" Have photos of the people and the places. Bring the book out before he goes, let him take it with him, and read it again and again when he comes home. It can become part of the separation ritual; seeing pictures will help him understand what's happening to him because this is a stage when children are very concrete in their understanding of the world.