He didn't have your permission to take the car in the first place, if he doesn't return after you told him to, then you can call the police as theft.
Just tell the police your car is missing, the ex-boyfriend stole it. No need to explain any more. Police doesn't want your life story.
Or you can go to where he is and get the car back when he is not looking. Don't forget to bring some friends with you.
Change the lock on the car later.