Plants and their specialized flower visitors provide valuable insights into the evolutionary consequences of species interactions. In particular, antagonistic interactions between insects and plants have often been invoked as a major driver of diversification. Here we use a tropical community of palms and their specialized insect flower visitors to test whether antagonisms lead to higher population divergence. Interactions between palms and the insects visiting their flowers range from brood pollination to florivory and commensalism, with the latter being species that feed on decaying–and presumably undefended–plant tissues. We test the role of insect-host interactions in the early stages of diversification of nine species of beetles sharing host plants and geographical ranges by first delimiting cryptic species and then using models of genetic isolation by environment. The degree to which insect populations are structured by the genetic divergence of plant populations varies. A hierarchical model reveals that this variation is largely uncorrelated with the kind of interaction, showing that antagonistic interactions are not associated with higher genetic differentiation. Other aspects of host use that affect plant-associated insects regardless of the outcomes of their interactions, such as sensory biases, are likely more general drivers of insect population divergence.