Friction and wear depend critically on surface roughness and its evolution with time. An accurate control of roughness is essential to the performance and durability of virtually all engineering applications. At geological scales, roughness along tectonic faults is intimately linked to stick-slip behaviour as experienced during earthquakes. While numerous experiments on natural, fractured, and frictional sliding surfaces have shown that roughness has self-affine fractal properties, much less is known about the mechanisms controlling the origins and the evolution of roughness. Here, by performing long-timescale molecular dynamics simulations and tracking the roughness evolution in time, we reveal that the emergence of self-affine surfaces is governed by the interplay between the ductile and brittle mechanisms of adhesive wear in three-body contact, and is independent of the initial state.