Dozens of farm owners and conservation groups signed a joint statement on Wednesday, demanding that private agricultural land lying idle is not used for building homes.
Taking over farmland in the New Territories is one of 18 ideas for new land for flats floated in a public consultation exercise underway.
But the farmers and conservationists said instead of using this land for housing, the government should speed up measures to develop the agricultural sector.
Roy Ng from the Conservancy Association, which signed the statement, said it is wrong to sacrifice farming for housing.
"A rural environment is very important and of course, in the rural environment, agriculture is one of the main components. So under this situation, we would say agriculture should be conserved in Hong Kong," Ng told RTHK's Joanne Wong.
"We do think that housing is also a very important matter ... what we are saying is that agriculture and housing development are not contradictory."