We have 6 gadgets. One is often blank (major incidents) but two are lists so they might take more. In general it's my main three that seem to take much longer to show up. "My incidents", "My Group Incidents" and "Unassigned incidents". The other stuff does not have a report template and come up much quicker.
Migrating to SQL 2012 helped immensely. It's still sluggish but much better than when I was running SQL 2008. I realized my issues was not database related, it must be something else. My Web server runs at 8GB ram, about 75% used and my 2 processors average about 12% capacity.