, he can hold the deposit, that is what deposit means.
even if you don't have a lease, you two might have an agreement of how long you will stay, you can't leave whenever you want.If you are in his shoe, how would you think?
You roommate might overreact, but you should take the responsiblity.