It ultimately depends how loud the soundboard is, what is playing and how frequently is being used.
As long as it's not overly loud, playing NSFW content, hateful slurs etc. and isn't being spammed every 5 seconds, you should be fine.
However, if people find it annoying and ask you to stop, stop.
Let me know if you have any further questions on this topic.