Market-based policy instruments to reduce greenhouse gas (GHG) emissions are generally considered more appropriate than command and control tools. However, the omission of transaction costs from policy evaluations and decision-making processes may result in inefficiency in public resource allocation and sub-optimal policy choices and outcomes. This paper aims to assess the relative cost-effectiveness of market-based GHG mitigation policy instruments in the agricultural sector by incorporating transaction costs. Assuming that farmers' responses to mitigation policies are economically rationale, an individual-based model is developed to study the relative performances of an emission tax, a nitrogen fertilizer tax, and a carbon trading scheme using farm data from the Scottish farm account survey (FAS) and emissions and transaction cost data from literature metadata survey. Model simulations show that none of the three schemes could be considered the most cost effective in all circumstances. The cost effectiveness depends both on the tax rate and the amount of free permits allocated to farmers. However, the emissions trading scheme appears to outperform both other policies in realistic scenarios. (C) 2012 Elsevier Ltd. All rights reserved.