We present a comprehensive analysis of the processes that lead to quasilinear pitch-angle-scattering loss of electrons from the L < 4 region of the Earth’s inner magnetosphere during geomagnetically quiet times. We consider scattering via Coulomb collisions, hiss waves, lightning-generated whistler (LGW) waves, waves from ground-based very-low frequency (VLF) transmitters, and electromagnetic ion cyclotron (EMIC) waves. The amplitude, frequency, and wave normal angle spectra of these waves are parameterized with empirical wave models, which are then used to compute pitch-angle diffusion coefficients. From these coefficients, we estimate the decay timescales, or lifetimes, of 30 keV - 4 MeV electrons and compare the results with timescales obtained from in-situ observations. We demonstrate good quantitative agreement between the two over most of the L and energy range under investigation. Our analysis suggests that the electron decay timescales are very sensitive to the choice of plasmaspheric density model. At L < 2, where our theoretical lifetimes do not agree well with the observations, we show that including Coulomb energy drag (ionization energy loss) in our calculations significantly improves the quantitative agreement with the observed decay timescales. We also use an accurate model of the geomagnetic field to provide an estimate of the effect that the drift-loss cone has on the theoretically-calculated electron lifetimes, which are usually obtained using an axisymmetric dipole field.