It depends  on their mental capabilities. If they are high functioning and understand what's happening, then I do hold them responsible. To not hold them responsible if they're capable is insulting them. Otherwise, no I blame the caretakers too. However, I'm not a doctor and I have no clue who is or isn't high functioning, so (unless it's obvious) I just stay away from the subject all together mostly.