When you serve, you dedicate your time. Sometimes that means being away from those you love. You need to understand that it is not a physical walk. There is impartation that happens in the spiritual realm.
2 Kings 2:2 Then Elijah said to Elisha, “Stay here, please, for the LORD has sent me on to Bethel.” But Elisha said, “As the LORD lives, and as your soul lives, I will not leave you!” So they went down to Bethel.
When you understand the spiritual relationship, serving become easy. People can talk but the reward is worth the sleepless nights. 2 Kings 2:9 When they came to the other side, Elijah said to Elisha, “Tell me what I can do for you before I am taken away.” And Elisha replied, “Please let me inherit a double share of your spirit and become your successor.”