drive (one) out of (one's) wits
To annoy or frustrate one to the point of exasperation. A week on vacation with my relatives is enough to drive me out of my wits. That loud beeping noise is driving me out of my wits!
drive somebody out of their ˈmind/ˈwits
(informal) make somebody crazy, or very nervous or worried: That noise is driving me out of my mind!