The short answer is that the rougher the texture, the more slowly the material will absorb heat by touching something else. Heat can be transferred ("traded") between two things in different ways, and texture is not something that matters much unless heat is absorbed directly from one solid thing into another.

Imagine two solid blocks of iron. One of them is very hot, and the other one is at room temperature. Both of them are cube-shaped. If both cubes are smooth (like slippery ice), then as long as the two are touching, the cold cube can absorb heat very quickly from the hot one. However, if the hot cube is smooth, but the cold cube is rough, then there would be less surface where the two cubes are touching. This would make the heat absorption slower. For the other modes of heat transfer, texture will not play as big of a role because the direct contact of objects (what we called touching) is not required.