The results of a search for Higgs boson pair (HH) production in the (WWWW*)-W-, WWtau tau, and tau tau tau tau decay modes are presented. The search uses 138 fb(-1) of proton-proton collision data recorded by the CMS experiment at the LHC at a center-of-mass energy of 13 TeV from 2016 to 2018. Analyzed events contain two, three, or four reconstructed leptons, including electrons, muons, and hadronically decaying tau leptons. No evidence for a signal is found in the data. Upper limits are set on the cross section for nonresonant HH production, as well as resonant production in which a new heavy particle decays to a pair of Higgs bosons. For nonresonant production, the observed (expected) upper limit on the cross section at 95% confidence level (CL) is 21.3 (19.4) times the standard model (SM) prediction. The observed (expected) ratio of the trilinear Higgs boson self-coupling to its value in the SM is constrained to be within the interval -6.9 to 11.1 (-6.9 to 11.7) at 95% CL, and limits are set on a variety of new-physics models using an effective field theory approach. The observed (expected) limits on the cross section for resonant HH production range from 0.18 to 0.90 (0.08 to 1.06) pb at 95% CL for new heavy-particle masses in the range 250-1000 GeV.