Question: Why is it sin if you are not serving? What is serving?
Answer:

but to each one is given the manifestation of the Spirit for the common good.
All Christians have been gifted by God for the good of others. 1 Corinthians 12:4-30 and Romans 12:3-8 both describe how the Spirit of God gives a variety of spiritual gifts to believers so that the church would grow (spiritually & numerically) as believers utilize their gifts. Ephesians 4:16 emphasizes that the growth of the church body is contigent upon “the proper working of each individual part.”
God has gifted every Christian SO THAT they would minister to others. That is why Peter writes,
As each one has received a gift, employ it in serving one another, as good stewards of the manifold grace of God. (1 Pet 4:10)
Your ability to minister is a stewardship from God, for which you will give account to Him. Just like the money, food and clothing which He provides to you, He calls you to use your spiritual gift in a way that brings Him glory (1 Pet 4:11).
Is it a sin not to serve? YES! You’re commanded to. Jesus calls you to be a servant, “for even the Son of Man did not come to be served, but to serve, and to give His life a ransom for many” (Mark 10:42-45).