Can't I just push them in?

I was sneaking through Annúminas last night - well, only sneaking around groups of 3+ warriors/etc. and happened upon this group. I did not expect to see a group of them fishing. This is GREAT! Hey, if we can chillax in Middle-Earth, I imagine the evil forces NEED to have some time off as well... but no, usually the evil doers are ALWAYS just doing evil. Perhaps they were hoping to catch fish to torture them, then release - rinse-repeat. In any event, I snuck up close enough to get a decent pic and thought "man, why can't I just push them in?"
Later, while swimming below them, they dropped poles and raised crossbows taking shots at me while I swam. That wasn't nice.