Well mine is set to 2MB which is the default setting in php.ini
I could edit it and make it larger but I don't have much need for the gallery and 2MB is more than ample for my users.
Your users may need larger files depending on the content of the gallery.
You could also set Max file sizes per group and promote users to each higher group based on post count automatically.
As far as slowing down the server I guess it would really depend on how much activity your site has and if your server can handle the bandwidth.
