To wit, so as to destroy thee, as the next verse limits and expounds it. For surely this promise is not made to all that dwell nigh to his children and servants, who may possibly be wicked men, and so strangers from God's covenant and promises. How far this secures his own person, See Poole on "Psalms 91:7".
Related Commentaries of Psalms 91:10
Psalms 91:10
10 There shall no evil befall thee, neither shall any plague come nigh thy dwelling.