Is your Joomla website slow? You should definitely work on that: users hate slow websites, and they may leave your site earlier then they otherwise would, resulting in low conversion. Any site that needs more then 2 seconds to load could be at risk. Also, Google even says that site speed is used to determine your place in their rankings.
What is nice to know: it should not be that hard to improve the speed of your Joomla website. While sometimes you can hear stories that Joomla is slow, this will usually be because of incorrect setup or bad hosting companies, and you can work on both. Especially if you know what to look at, improving the speed of your Joomla website may not be too hard, and making it load in less then a second should be possible.