Im so confused

Okay read closely because this iz a long story. About 5 years ago I moved to a new house. about 2 years later this boy I knew from camp moved in up the street me my brother and my nieghbor all became good friends with him and his 2 little brothers. so any way a couple of months later we went together. 3 weeks later we broke up. 2 more years later we got back together. 2 days after that he left me for some girl that left her boyfriend for him. then they broke up so now he's coming back to me trying to get with me again and im slowly fallin for him again. I want to wait a while then get back with him but everyone is going to hate me what do I do

if you continue letting him have you whenever he feels like it and just dropping you whenever someone else comes along he's going to think he has control over you. if people are going to be mad at you its probably because they're worried you'll just get hurt again. I know you probably really want to be with him finally but I dont think you should..I think you should make it loud and clear that he's not the only one who has a say in this matter. do w/e your heart tells you, but if you do date him than just make sure you're prepared in case the same thing happens as the last couple times.

going back with him after what he did to me