Segovia, Toledo, Avila, those all well worth a day trip. For Starter, see the introduction by Lonely Planet
https://www.lonelyplanet.com/articles/day-trips-from-madrid
Unless you plan to return to Spain / Madrid in the future, otherwise, skipping the above daytrip destinations would be a pity.