Please help transmission or clutch!!

I have a 91 240 it is a 5 speed and has a spec clutch put in. Recently i was driving and ive been hearing this noise its like a draging or grinding noise. Can sound like a chain draging or 2 metal wheels hitting each other like maybe 2 gear wheels if it makes sense. This happens usually if i shift into like 2nd or 3rd and if i let off the gas. Usually only happens when i let off the gas so if im in 2nd or 3rd gear give it gas and start going but then start slowing up by letting off the gas i hear this noise. Not sure if it is transmission or clutch or how serious this is. Ive put so much money into this car already just want to know how much of a problem this is. Note it does not effect the driving of this car at all still drives and shifts perfect so far. Note the noise can also be heard if im in neutral and rev the engine then simply move the shifter left right or up but not into a gear just with that little play you have. Thanks to everyone and anyone who can help.