The specific heat of water is much higher than that of aluminum, meaning that a greater amount of energy must escape the water than the aluminum for them to cool the same amount. Since aluminum needs to lose less heat energy, it can do so more quickly.