I might be misremembering, but I think that reduced sensation plays the biggest role. Along with some issues with impaired temperature regulation in general. Actual insulation is in large part diminished simply due to larger surface area. Human skin just isn't set up like a water mammal's. The fat can help a bit. But it's a tiny benefit in one area and a significant risk on another.