I have somewhat two thoughts on that.
Firstly she's effectively the closest thing to that of a mother figure he ever had that's still around today I believe (they showed flashbacks of him having other women around him as a kid which he wanted to call "mom" but he hugged them to death). So I think he has always craved a mother hence it would be hard for him to kill her even though she caused him so much trauma.
To add onto that, I can't recall the exact thing she said but it was something along the lines of having the best psychologists working on him since birth to control him and make him dependent on seeking love/approval. As she was leading all of that and again being the only woman in authority, she probably mind fucked him to the point that he can only ever want to seek validation from her (like a safety mechanism hence why you think she has some bravado against him but she's playing a part because she knows his mental triggers from birth because they put them there deliberately). So because he can't physically hurt her because of that craving of validation from her, he'd do the next best thing which is to show her how much he wants to hurt her by inflicting it on others (ie the other scientists) and left her to watch and contemplate.
