It should seem they said it in Jacob's hearing. The last chapter began with Rachel's envying Leah; this begins with Laban's sons envying Jacob. He has gotten all his glory — And what was this glory? It was a parcel of brown sheep and speckled goats, and some camels and asses. Jacob has taken away all that was our fathers — Not all, sure; what was become of those cattle which were committed to the custody of Laban's sons, and sent three days journey off?
Related Commentaries of Genesis 31:1
Genesis 31:1
1 And he heard the words of Laban's sons, saying, Jacob hath taken away all that was our father's; and of that which was our father's hath he gotten all this glory.