Importance sampling remains unbiased because it corrects for the bias introduced by using a different probability distribution than the true one. The key lies in the weighting of the simulation outputs.
Understanding the Bias Correction
The core idea behind importance sampling is to estimate an expectation using a more convenient probability distribution, often called the proposal distribution, instead of the true, often more complex, target distribution. This introduces a bias. However, this bias is systematically corrected by weighting each sample from the proposal distribution by a factor that accounts for the difference between the proposal and target distributions. This weight is the ratio of the probability densities of the target and proposal distributions at the sampled point.
This weighting ensures that samples from regions with higher probability under the target distribution have a larger influence on the final estimate, compensating for the bias induced by sampling from the proposal distribution. The reference explicitly states that: "the simulation outputs are weighted to correct for the use of the biased distribution, and this ensures that the new importance sampling estimator is unbiased."
Think of it like this: if the proposal distribution under-samples a crucial region, the weights will boost the contribution of the few samples that do fall in that region, effectively counteracting the under-sampling. Conversely, if a region is over-sampled, the weights will down-weight the contributions from those samples. This dynamic adjustment ensures that the final estimate converges to the true expectation.
Example
Suppose you want to estimate the expectation of a function under a difficult-to-sample distribution. Importance sampling allows you to sample from an easier distribution and then adjust the results using the weights described above. The weights ensure that despite using a different sampling distribution, the final result remains unbiased.
