We explore in detail the possibility that gravitational wave signals from binary inspirals are affected by a new force that couples only to dark matter particles. We discuss the impact of both the new force acting between the binary partners as well as radiation of the force carrier. We identify numerous constraints on any such scenario, ultimately concluding that observable effects on the dynamics of binary inspirals due to such a force are not possible if the dark matter is accrued during ordinary stellar evolution. Constraints arise from the requirement that the astronomical body be able to collect and bind at small enough radius an adequate number of dark matter particles, from the requirement that the particles thus collected remain bound to neutron stars in the presence of another neutron star, and from the requirement that the theory allows old neutron stars to exist and retain their charge. Thus, we show that any deviation from the predictions of general relativity observed in binary inspirals must be due either to the material properties of the inspiraling objects themselves, such as a tidal deformability, to a true fifth force coupled to baryons, or to a non-standard production mechanism for the dark matter cores of neutron stars. Viable scenarios of the latter type include production of dark matter in exotic neutron decays, or the formation of compact dark matter objects in the early Universe that later seed star formation or are captured by stars.