Hmm, that's a good question. Guess we need an option to delete. If you don't have the "Always Build" checked and it's not actually on your portal it won't actually be created/updated. I'll need to get an option in to delete one.