Can I completely remove the Add to cart button for all user groups for specific products in magento?

Yes, on the corresponding Product Information tab you will be able to choose whether Hide Price should be enabled, and if yes, for which customer groups it should work.

Looking for Magento 1 extensions? At Amasty, we offer a wide range of options to enhance your store. Explore our selection here.