He called me stupid, then said it was all in jest and he felt really bad about it.
in jest
For fun; as a joke; without sincerity. I know your remarks were only in jest, but they still hurt my feelings.
in ˈjest
as a joke: The remark was made half in jest.‘Many a true word is spoken in jest,’ thought Rosie (= people often say things as a joke that are actually true).