I've always imagined it is because castling is intended as a development move for breaking board symmetry, not as an escape move to get out of a serious attack. If your opponent has already launched an attack on the King's current position, or if (say) he is using a Rook to cut you off from that side of the board, it's considered unfair to get away or to cross the line of control using a special move.
Kind of like how en passant capture was added because the pawn's double-step was intended as a development move to speed up the game, and people didn't like that it was being used to leap past enemy pawns without giving them a chance to intercept, thus altering the game's strategy.
But I could be wrong.