A correlation-based (Hebbian'') learning rule at the spike level is formulated, mathematically analyzed, and compared with learning in a firing-rate description. As for spike coding, we take advantage of a learning window'' that describes the effect of timing of pre- and postsynaptic spikes on synaptic weights. A differential equation for the learning dynamics is derived under the assumption that the time scales of learning and spiking dynamics can be separated. Formation of structured synapses is analyzed for a Poissonian neuron model which receives time-dependent stochastic input. It is shown that correlations between input and output spikes tend to stabilize structure formation. With an appropriate choice of parameters, learning leads to an intrinsic normalization of the average weight and the output firing rates. Noise generates diffusion-like spreading of synaptic weights.