Efficient and safe multi-agent swarm coordination in environments where humans operate, such as warehouses, assistive living rooms, or automated hospitals, is crucial for adopting automation. In this paper, we augment the obstacle avoidance algorithm based on dynamical system modulation for a swarm of heterogeneous holonomic mobile agents. A smooth prioritization is proposed to change the reactivity of the swarm towards the specific agents. Further, a soft decoupling of the initial agent's kinematics is used to design an independent rotation control to ensure the agent reaches the desired position and orientation simultaneously. This decoupling allowed the introduction of a novel heuristic, the virtual drag. It minimizes the disturbance influence an agent has when moving through its surrounding. Additionally, the safety module adapts the velocity commands from the dynamical system modulation to avoid colliding trajectories between agents. The evaluation was performed in simulated assisted living and hospital environments. The prioritization successfully increased the minimum distance relative to a moving agent. The safety module is observed to create collision-free dynamics where alternative methods fail. Additionally, the repulsive nature of the safety module augments the convergence rate, thus making the proposed method better applicable to dense real-world scenarios.