Shouldn't have to clear it daily and it'll result in much longer loading times if you do. Guess it depends on your PC (up to a point), but it should just.. work. Even with a huge inventory.
Granted I've had issues with my shared room more recently and a large freeze before it 'properly' loads that wasn't the case a few weeks ago. Hardware definitely isn't an issue and neither myself nor the person I share it with added much more to the room, so it's something on IMVU's end. I know today and yesterday the servers have been chugging a little with slow webpage loading - happens every so often it seems so things might ease in a few days. Likely to be an update soon anyway if it's anything to go by and the timeline for AP+ releasing is any indicator.
Something I noted that can help if you're a long-time user is deleting some of your unused outfits if you have a ton. It made a somewhat noticeable difference for me in regards to general loading and of course how long it takes for the outfits list to load. Especially after you next clear the cache. Could try hiding any unused items in your inventory as well, although I don't think that would help in the same way.
