Load times are slightly reduced by some modchips and some softmod loading techniques. However the loading time should not be siginficantly increased such that you'd notice a massive difference. Most people only notice that FMV sequences are stuttered.
The type of media you use is somewhat important in the speed that the Wii can read. I tried some DVD+RW on my Wiikey2 and noticed the speeds were a lot slower and the drive made lots of seek noises as it struggled to read in data. If you are also getting this symptom then looks towards good quality DVD-R media such as Verbatim.
Of course you could always ditch disc loading and go USB like mostly everyone else as that offers the ultimate speed of transfer.