Manipulating soft and fragile objects is a challenging task in robotic grasping. The key challenge for robotic grasping is to exert enough grip force to prevent slipping while being gentle enough to prevent damage to an object. Existing grippers used for processes like automatic harvesting of fruits, either apply excessive grip force leading to object damage or react to slip resulting in object release from the gripper. The aim of this study is to develop a grip force controller that uses tactile feedback to maintain a constant frictional safety margin over the minimum required grip force, called Safety Margin Control. Tactile sensors can provide information on friction, which is used to predict slip. An optical tactile sensor is modeled and used in simulations where Safety Margin Control regulates the grip force during interaction with various virtual objects. The deformation of the sensor’s soft viscoelastic membrane is described by local frictional behavior and used to estimate the safety margin. The desired safety margin is set to 30%, based on comparison to the way humans control grip force in their fingertips. The desired value can be tuned to favor release over damage and vice versa. Safety Margin Control is compared to two baseline controllers: React To Slip and Conservative Control. The performance is evaluated based on maximum pressure and total lateral displacement of the object relative to the sensor. Safety Margin Control results in a pressure decrease of 44% on average compared to Conservative Control, and no significant pressure change was observed compared to React To Slip. The total lateral displacement for Safety Margin Control is 0 mm, as opposed to 1.3 mm for React To Slip. Safety Margin Control provides a way forward for automated harvesting as the pressure exerted on an object can be reduced while no slip occurs.