Refactoring takes time. If not done well it creates bugs that you need to fix. So it costs. You refactor when the benefits outweigh the cost of refactoring.
There are good times for refactoring: One is immediately after the code is written. At that time, you know the code. You know exactly what it is supposed to do. You know how to improve it. It hasn’t had the time yet to lose bugs through maturing, so bugs added during refactoring are not a big deal at this point.
The other time is when you need to change the code’s behaviour and the code is in such a bad shape that making changes is hard to do and risky, and it is more effective to first refactor and put the code into a good shape and then make the behaviour changes.
NOT a good time is when you look at the code and don’t like it, and that’s your only reason to change it.