Insurance's basic principle is simple, spreading risk across a broader pool. When harmed, you get assistance to lessen the impact, when you're not harmed, your payments cover the costs of others who are. But deciding who is harmed is a very time consuming task. Preventing fraud is important to staying competitive, when looking at private enterprise, and important to public trust when dealing with public programs. But preventing fraud places burdens not only on the insurance provider, but the claimants. Having to prove losses, is costly. There's a risk that valid claims are denied, and even when not, there are delays and investments of time as claims are validated. In a dire situation, those delays and investments are a heavy burden.
One innovation to insurance is index based insurance. Index based insurance simplified the claim process by basing it upon an easily verifiable metric. If rainfall in an area exceeds a specified amount, insurance payouts are made automatically. Verification is simple and potentially quick, and claimants shouldn't need to be involved. Cost of validating and preventing fraud are substantially lower.
The downside to index based insurance is that the index will imperfectly map to those impacted. It might be fairly good, and the trade-off in terms of faster payouts can in-total be a better arrangement for claimants who otherwise might have opted for no insurance at all. For the insurer, the reductions in program costs can make up for the unnecessary payments to those in the impacted area who happen to avoid actual damages but receive a payout due to the index. But, for the individual who is impacted, but not covered by the index, the outcome is fairly dire, leaving a large visible hole in value of index based insurance.
It seems to me that there is a not so hard fix to this problem. Layering traditional loss-based insurance on top of index-based insurance. In a well designed system, loss-based claims would be dramatically lower. If you receive a index-based payment, and suffer a loss, there's no need to file a claim. If you receive an index-based payment, without a loss, hopefully you're good fortune will cause you to think about your neighbors. But for the few who suffer a loss without receiving an index-based payment, filing a claim would still be an option. With a lower number of claims, the ability to process these quickly should be more realistic.
It seems somewhat obvious, but the reason it hasn't been done yet is that index-based insurance is not broad enough yet. It's generally fairly specific at certain indexes. A system that accommodates this layering needs to cover a number of indexes. One for low rainfall, one for flooding, one for low forage coverage, etc.
Once enough indexes are bundled together, the chance that a loss-based claim is necessary will have declined to a level where claim handling is no longer expensive.
Inspired by reading the Economist...