You can limit a user to only certain groups (Settings > Users > Select user > Limit to camera groups). Create groups including only that site's cameras and assign users that way. Though I don't know how hard that is, i.e., If there's some way around that if it would matter in your case.