To focus on the price one can charge for a food (or anything else ... ) implies giving the economics of its supply chain primacy in differentiating expensive foods from ordinary fare.
My point is that supply chain economics may be as important a factor as the food per se in justifying a high price.
The one factor that seems to be associated with super-expensive foods is rarity or scarcity. In some cases, this rarity / scarcity has as much or more to do with the supply chain as anything else.
In some cases, rarity is imposed by focusing on one particular source or sub-species of a given food item that is generally obtainable from multiple similar sources. Caviar (in general) isn't all that cheap. The most expensive caviars are varieties uniquely derived from a particular fish population or a particular feature (e.g., fishes' age).
At the downstream end of the supply chain there are foodstuffs that are costly because they are the last specimens of their final / processed production cohort. Wine represents the paradigmatic example of pricing driven by end-product rarity / scarcity. There aren't many examples of this sort of end-product specimen scarcity driving price points.
Most of the foods touted as 'most expensive' are costly owing to constraints or overhead costs farther upstream in the supply chain. These influential factors can be of different types, such as:
- Absolute rarity - A very limited overall supply, regardless of any other factor
- Growing / nurturing overhead costs - Lots of expenses in simply generating the stuff, as in exotic Japanese steaks from pampered cattle
- Collecting / harvesting overhead costs - It takes lots of labor in risky settings to gather the birds' nests for birds' nest soup
- Temporal constraints on collecting / harvesting - Both vanilla and saffron can be harvested only during a narrow time window each year
- Production overhead costs - For example, items that must be aged for long periods require storage, etc., investments above and beyond what pertains to most such items
- Distribution / dissemination constraints - Black chicken is readily available at 'common chicken pricing' in Malaysia, but isn't exported worldwide owing to concerns about spreading bird flu. Some caviars are hard to obtain (in part) because of Iran's relative isolation vis a vis trade sanctions / restrictions.
Anyway ... I don't think there's a single clear answer because there's a variety of factors that may be in play.