If you do not or cannot indulge your vice during a downtime phase, you take stress equal to your trauma. If you don't have any trauma, you're free to ignore your vice. It doesn't have a hold over you (yet).