Motivated by the omnipresence of hierarchical structures in many real-world applications, this study delves into the intricate realm of bi-level games, with a specific focus on exploring local Stackelberg equilibria as a solution concept. While existing literature offers various methods tailored to specific game structures featuring one leader and multiple followers, a comprehensive framework providing formal convergence guarantees appears to be lacking. Drawing inspiration from sensitivity results for nonlinear programs and guided by the imperative to maintain scalability and preserve agent privacy, we propose a decentralized approach based on the projected gradient descent with the Armijo stepsize rule. By the virtue of the Implicit Function Theorem, we establish convergence to a local Stackelberg equilibrium for a broad class of bi-level games. Moreover, for quadratic aggregative Stackelberg games, we also introduce a decentralized warm-start procedure based on the consensus alternating direction method of multipliers addressing the initialization issues reported in our previous work. Finally, we provide empirical validation through two case studies in smart mobility, showcasing the effectiveness of our general method in handling general convex constraints, and the effectiveness of its extension in tackling initialization issues.