How do I make sure that only the people in the group access specific forms and not all of my clients?

When you assign a form based on permissions (in WEBSITE > FORMS), the form will show to any client that is assigned either on the left or the right permissions list.

So, if you have a specific group that you want only the people listed in that group to have access, be sure to move only that group into the "Permitted Contacts and/or Groups" box.