Density dependence is a fundamental ecological process. In particular, animal habitat selection and social behaviour often affect fitness in a density-dependent manner. The Ideal Free Distribution (IFD) and niche variation hypothesis (NVH) present distinct predictions associated with Optimal Foraging Theory about how the effect of habitat selection on fitness varies with population density. Using caribou (Rangifer tarandus) in Canada as a model system, we test competing hypotheses about how habitat specialization, social behaviour, and annual reproductive success (co)vary across a population density gradient. Within a behavioural reaction norm framework, we estimate repeatability, behavioural plasticity, and covariance among social behaviour and habitat selection to investigate the adaptive value of sociality and habitat selection. In support of NVH, but not the IFD, we find that at high density habitat specialists had higher annual reproductive success than generalists, but were also less social than generalists, suggesting the possibility that specialists were less social to avoid competition. Our study supports niche variation as a mechanism for density-dependent habitat specialization.