One is that a few of the most iconic noirs do make use of it, so people who have only seen a handful of the true classics are more likely to associate this technique with noir. The other, and I've never seen this commented on elsewhere, is that the use of a voice-over closely replicates the tone and feeling of the novels on which a lot of noir movies were based.
The Philip Marlowe detective novels by Raymond Chandler are all narrated in the first person by the detective himself, and his world-weary voice and cynical commentary create much of the noir mood in the books. So when a film noir uses a flashback narrated by the detective, it produces basically the same feeling. It makes the movie feel more like the book.
Another aspect is that the flashback in a film noir is usually leading up to a tragedy of some kind. It's the story of one man's fall from grace. So the voice-over has a built-in quality of regret and pathos, as the person speaking has already experienced the tragedy. That's why the theatrical version of the 70s sci-fi noir classic “Bladerunner” feels even more noir than the otherwise superior director's cut, whih dispenses with the voice-over. The studio insisted on adding the voice-over to help the viewers understand a complex story, but in doing so they inadvertently tapped in to a powerful noir trope- the voice of a doomed man narrating the story of his own defeat.