This is an interesting question. I think of 'grown ups' as family people. In my eyes, somebody is a grown up when they have a family with kids at school, a house etc etc. A 'man' to me is a manly guy. I call guys 'guys' unless they are bulky, well built and manly.. then they are a man. But that's just my personal view.
A "man" is a male grown-up ... therefore, a "grown-up" is simply an adult.