When I've slept with his mate

I’ve been seeing a guy (lets call him jeff) casually for about 7 months now. We generally get along really well and always have something to talk about- basically we seem to have very compatable personalities. About 3 or 4 months into our casual relationship I began to develop strong feelings towards him. I admitted that I love him one night while we were in bed cuddling and talking. It was a really hard thing for me to say because im usually quite reserved with how I feel and dont like to let much on. I got quite emotional when saying it and jeff held me and comforted me until I stopped crying. The next morning I dropped him off at work and he got out of the car and said bye and seemed really distant. I decided to think nothing of it and texted him that night to see how his day was. He didnt reply and when I tried calling he didnt pick up. I went away to the beach with one of my good friends and her boyfriend for the weekend and didnt hear from jeff until the second night I was there when he called me late and asked me why I said what I said and basically told me that he wanted nothing to do with me because of what I’d said/how I felt towards him. A few days went by and he ignored any of my attempts to speak to him. At this stage I was feeling really confused and more than a little bit hurt- here was this guy who I’d just poured my heart out to rejecting me without so much as an explaination why. I ended up going out to a pub with some friends at the end of the week and drank a fair bit too much. I was quite drunk and jeff was ignoring me. I was feeling really hurt and rejected and in a way really mad because he couldnt even give me an opportunity to ask what was wrong or what was going through his head. I wanted to make him feel as bad as I did so I ended up taking home a friend of his who he also works with. we slept together and I dont remember much of what happened. The next day I felt disgusted with myself. I drove baz (jeff’s friend who I slept with) back to their work camp. Baz then apparently told some of the guys who they work with and jeff found out that way. Jeff called me and asked me what happened with baz and told me that he had had to hear about it all day at work. I told him the truth as I remembered it and then said that he shouldnt care what I do because he hasnt spoken to me for about 2 weeks and made it quite clear that he wanted nothing to do with me. A few weeks passed and we started talking again more and more and things sort of shifted back to how they were pre baz and we were hanging out again and sleeping together and getting along really well again. Jeff then asked me to be his girlfriend and for about 2 days I was until he decided that he couldnt do it because of the baz thing and that everytime we had sex he thought about what I did with baz. We’ve since gotten back together but he constantly brings up the incident with baz and says that he cant get over it and this makes me feel really terrible. As far as I see it the sex with baz was a one off and not something that had any emotions attached to it. I do not have any feelings towards baz and cant understand why jeff doesnt understand this. I was not with him at the time of this incident so do not see how he is able to justify saying I cheated on him, yet he still maintains that I did. I am aware that the same night that I slept with baz jeff also slept with another girl but in his opinion this is not cheating. Jeff has also slept with a number of other girls while we have been in a casual relationship and this is fine yet I have slept with one guy and im suddenly an awful person. I like him so much as a person and a lover and know that he feels the same. I dont want to let what we have go. We get along incredibly well most of the time, theres that chemistry there that you need to have for a good relationship, we always have something to talk about and because we have a lot of fun together. Id appreciate any advice as to how I can make him understand that I wont do anything like what I’ve done again and to help him work through this so that we can move on in our relationship.