There are numerous elements in the answer. The populations in HNBR are typically small, and large numbers are needed to have statistical significance. There may be a genetic background difference between the two populations, especially if the population in the HNBR is well isolated. And then there are all the other traditional factors which confuse the issue like diet, exercise, other carcinogens, etc.
Let's do a simplistic mental exercise, after I first point out that there are about 1E14 cells in your body and EACH one undergoes about 100,000 lesions per day. Your cells do a remarkable job of staying intact, but they are not perfect. Overall then, your body experiences about 1E19 lesions per day, some caused by natural background radiation.
Let's compare that with someone living in a HNBR who is receiving 5 rem per year more than you:
(5 rem) (100 ergs/g per 1 rem) (70,000 g per body) (6.24E11 eV/erg) (1 ion pair/34 eV) = 6E17 ion pairs.
The body weight above is about 150 lbs. The amount of energy required to form ion pairs depends on the bonds being broken, but 34 eV is a traditional average number.
This gives us 6E17 ion pairs all of which we'll consider to be lesions which are proxies for risk.
So over the course of the year there are (365 days/yr) (1E19 lesions/day) = 4E21 lesions.
So your risk is 4E21 and the guy in the HNBR area's risk is 4.0006E21. Both of your risks go up year after year. After 100 years your risk is 4E23 and his is 4.0006E23. Since we're discussing relative risks we can drop the exponents leaving us with 4 versus 4.0006.
This illustrates why it would take huge populations (exposed and unexposed) in order to discern a statistically significant difference in cancer incidence. Recall that when doing epidemiology one has to ensure that there is a 90+% confidence that any measured difference in incidence(between exposed and unexposed populations) isn't due to normal statistical variation (that's in addition to the possibility of the presence of unknown confounding factors). This is also why we don't discern much cancer incidence difference between those with occupational radiation exposures and those without.
We can compare my rough lesion-based estimate to a BEIR VII derived estimated increase of 0.5% per 5 rem.
(4.0006/4) = 1.00015 or a 0.015% increase per 5 rem, so the BEIR estimate is higher by a factor of about 34 (which isn't too bad!). That risk estimate includes all the factors associated with risk, not just the formation of ion pairs (genetic differences, repair, apoptosis, etc.). But hopefully the lesion approach, illustrates simplistically why excess background levels don't show increased cancer incidence.