You have to think about what is best for YOU. Are you happy?
There has to be a point in time where you recognize that your own happiness is what is most important to you. I would much rather do what I need to do for myself than do something that other people want me to.
There is nothing that is more annoying than when people are extremely worried about what you are doing, but don't let it phase you. Let them waste their time being concerned about your life, but don't let it impact you one bit.
Never forget that YOU are the most important person in your life. You are the only one that should determine what you do. It's your life, not anyone else's. Make sure that you remember that they are your mistakes and your successes rather than giving credit to anyone else.
It sucks to let your friends or family down, but you have to do what is going to be the best for you in the end. When I look back on my life, I want to think about how much I loved each minute of it. I'd never be able to do that if I focused on what everyone else thought about what I was doing or who I was with. If you want something, go and get it. If you love someone, be with them.
Overall, you're the one who gets to make the choices. You call the shots on your future and there is nothing that anyone else is able to do about it. Things will go your way if you do what is best for you and don't worry about what others think. I'd rather be alone than surrounded by people who are unhappy with a decision that brings me joy.
You can never please everyone. People are always going to have something to say about you. What really matters is that you're setting yourself up to have the life that you always dreamed of. The people who truly love and support you will be glad that you are happy no matter the circumstances.