'Nor be you idolaters, as were some of them. As it is written, The people sat down to eat and drink, and rose up to play.'
Reference here is to worship of the molten calf and its accompanying immoral rites (Exodus 32:6), again paralleling entering idol temples, and the danger of participation in their immoral behaviour. Note the stress on the fact that they ate in the presence of the idol which resulted in sin as a consequence. That is precisely what the danger was for the Corinthians.
Note also the change from 'we' (1 Corinthians 10:6) to 'you' (1 Corinthians 10:7) to 'us' (1 Corinthians 10:8). Paul could not link himself to idolatry because he had never been involved with it. But he recognised his ever present, (although held under by his walking with the Spirit), propensity for sins of the flesh.