Crops are subject to seasonal conditions and infestations.
Infestation, which will destroy the entire crop and generate the icon to the right. Infestation can spread to neighbouring fields bearing the same crop, so it's generally a good idea to use different crops for adjoining fields.
While there has been talk of crop rotation, or lack thereof, influencing infestation rates and soil fertility, no such feature currently exists. It might be added later if a way can be found to make it fun, rather than tedious.
During an infestation, it's best to order your workers to harvest the crop as soon as possible to recover at least some of the food.