For our heart shall rejoice in him - See the notes at Psalms 13:5.
Because we have trusted in his holy name - In “him,” the “name” often being put for the person himself. See the notes at Psalms 20:1. The idea is:
(a) that the fact of our having put our trust in God is in itself an occasion of joy or rejoicing;
(b) that the result will be joy, for we shall never be disappointed.
It will always, and in all circumstances, be a source of joy to anyone that he HAS put his trust in the name of God.