Breaking the reciprocity of light propagation in a nanoscale photonic integrated circuit (PIC) is a topic of intense research, fostered by the promises of this technology in areas ranging from experimental research in classical and quantum optics to high-rate telecommunications and data interconnects. In particular, silicon PICs fabricated in processes compatible with the existing complementary metal-oxide-semiconductor (CMOS) infrastructure have attracted remarkable attention. However, a practical solution for integrating optical isolators and circulators within the current CMOS technology remains elusive. Here, we introduce a new non-reciprocal photonic circuit operating with standard single-mode waveguides or optical fibers. Our design exploits a time-dependent index modulation obtained with conventional phase modulators such as the one widely available in silicon photonics platforms. Because it is based on fully balanced interferometers and does not involve resonant structures, our scheme is also intrinsically broadband. Using realistic parameters we calculate an extinction ratio superior to 20dB and insertion loss below 3dB.