The “fathers” are, as in Deuteronomy 4:37, the patriarchs, Abraham, Isaac, and Jacob. With them God did indeed make a covenant, but not the particular covenant now in question. The responsibilites of this later covenant, made at Sinai by the nation as a nation, attached in their day and generation to those whom Moses was addressing.
Related Commentaries of Deuteronomy 5:3
Deuteronomy 5:3
3 The LORD made not this covenant with our fathers, but with us, even us, who are all of us here alive this day.