Condensation can form when a surface is at the appropriate cold temperature.
If your indoor temperature is, for example, 20°C with an RH level of 35%, the dehumidifier would stop dehumidifying because the air is cold enough.
Hence when the surface temperature of the windows drop to under 5°C overnight, then condensation will occur.
This is normal, and a dehumidifier may not help as it is dependent on your window glazing.