You should refer to the savings clause that the US puts in (almost?) all of its tax treaties which excludes US citizens from most of the treaty provisions. That includes the interest. See the section 1(3) of the treaty. So form 8833 wouldn't help since treaty provisions don't apply to you here.
So you cannot, unfortunately, avoid double taxation in this case. This is not unusual. The US form 1116 makes an implicit assumption that the whole world taxes income the same way the US does, but it is not true. In your example of New Zealand, capital gains are ignored for the purposes of income tax, and as such you end up with only interest showing up on your NZ tax return. On form 1116 you also have capital losses, which would have offset the interest income, and as such the US doesn't allow credit for the NZ tax paid.
The US is very aggressive when it comes to taxing foreign sourced income of individuals. I'm not asking you what exactly you invested in for the capital gains part, but if you're not familiar with the term PFIC - familiarize yourself.