Restrict user profile access by groups

I am creating a site wherein users will form groups. I would like to restrict the site wherein users can only view the profiles of other users if they are in the same group.

I will likely use organic groups in order to achieve group functionality, but I'm not sure how to go about restricting user profile access. Any ideas, or should I replace the user profile module with something else entirely? I am willing to write the code if need be.