The defeat left Villa 12th in the Championship table, one place lower than Norwich.
Bruce told BBC WM 95.6: "I can understand people's frustrations and disappointments but I think that's only the fifth time in my two years that we've been beaten here [at Villa Park].
"It's difficult at the moment. No matter what I do, in some people's eyes I'm not going to be the right fit.
"There's nothing I can do about that. I will stay on the ropes and stay with it. That will never change with me. I'm never going to walk away from a challenge - and it's certainly been a challenge.
"In some people's eyes, I will never be the answer, that's obvious but the one thing that changes it is results so we need a few results. I do believe we've got the nucleus of a very decent team again, given time."
Bruce has now been in charge at the Birmingham club for 100 matches.