If I sat idly and watched you drown, knowing that I could easily save you, according to a strict interpretation of the word, I would not have committed aggression. I would not have violated the Non-aggression Principle and I could very easily conclude that I did not act immorally. What should be clear, however, is that I acted in a manner which I believed would result in harm to another person without their consent, which is immoral.
If the Non-aggression Principle indeed fails to recognize idleness as a potential source of “aggression”, it fails ( in my opinion ) as a universal ethical standard. Doing nothing is not the same as not acting. Ludwig von Mises says in his book, Human Action, “Action is not only doing but no less omitting to do what possibly could be done.”
If NAP would not permit me to willfully allow you to drown, then I think the Libertarian community has done a horrible job accurately describing exactly what “aggression” is as it pertains to the Non-Aggression Principle. If NAP would permit aggression by idleness, then I see it as an incomplete ethical standard, needing desperately a more robust definition of aggression plugged into it for there to be any hope of it being taken seriously.