If there's one thing that Christians can relate to, it's that sometimes you'll have non-Christian friends. I'm not saying it's a bad thing. In fact, Jesus calls on us to love thy neighbor (Matthew 23:39). He WANTS us to reach out and touch lives. So what if your friends get upset when you say you're going to church and can't go on a trip with them?
Stay calm. It doesn't help if you get upset as well. You just tell them that God comes first in your life and sometimes that means missing out on trips with friends.
"That's a sad way to live. God will always be there, but memories with us won't." That's exactly right. Friends come and go, but God will ALWAYS be there. He is there for the good AND the bad. I love all my friends dearly, but they're a temporary part of my life. Of course, I want to go on trips with them and make memories, but not at the expense of not being able to spend time with God.
Now, you can spend time with God anywhere, because he is everywhere. However, sometimes the atmosphere your friends want to be in is not a place God would want you in. Sometimes you do have to go and make sure your friends will be safe.
Sometimes you have to say no to friends to say yes to God. Your friends might be upset, but the thought that you get to spend time with God will lift your spirits. It's hard seeing your friends go places without you, but the satisfaction that one day you get to see your Father in Heaven makes it all worth it, and NOTHING can take that away.
