In most developing countries value added per worker is much higher in the non-agricultural sector than in agriculture. Since agriculture accounts for the majority of employment in this countries, this agricultural productivity gap accounts for a large fraction of the aggregate differences in output per capita. We present a two-sector growth model that allows for sector specific income elasticities of demand and elasticities of substitution between capital and labor. As the model economy develops, capital accumulates and income increases. As income increases, the relatively low income elasticity of demand for farming induces a continuous process of reallocation of factors of production out of agriculture. As capital accumulates, the high substitutability agricultural sector becomes capital-intensive, growth in value-added per worker in agriculture outpaces that in non-agriculture, and the agricultural productivity gap declines. We calibrate the model to US time-series data from 1960 to 2007 and evaluate its predictions against a large cross-section of developing and developed countries. Our results suggest that more than two thirds of the agricultural productivity gap can be traced back to cross-country differences in capital per worker.