Members of the public have raised their concerns over HGV traffic and noise pollution should the partial demolition of Fold Mill go ahead.