Excluding Specific Group Access to Folders: Options and Solutions

Is it possible to exclude a specific group from seeing a folder and its contents? Wishing there was a "no access" option here. The only other way I could think of is by setting my all folders to no access for the entire org, then assigning access individually. In this case, we don't pay for SCIM provisioning so our groups are manually managed and we are just trying to provide limited repository access to a few of our contractors.