Embryonic stem (ES) cells are derived from blastocyst-stage embryos and are thought to be functionally equivalent to the inner cell mass, which lacks the ability to produce all extraembryonic tissues. Here we identify a rare transient cell population within mouse ES and induced pluripotent stem (iPS) cell cultures that expresses high levels of transcripts found in two-cell (2C) embryos in which the blastomeres are totipotent. We genetically tagged these 2C-like ES cells and show that they lack the inner cell mass pluripotency proteins Oct4 (also known as Pou5f1), Sox2 and Nanog, and have acquired the ability to contribute to both embryonic and extraembryonic tissues. We show that nearly all ES cells cycle in and out of this privileged state, which is partially controlled by histone-modifying enzymes. Transcriptome sequencing and bioinformatic analyses showed that many 2C transcripts are initiated from long terminal repeats derived from endogenous retroviruses, suggesting this foreign sequence has helped to drive cell-fate regulation in placental mammals.