Yes, you will need to upload all files to the server for a new install. For an upgrade you'd upload all but the cache_builders/custom, styles, and includes folders (though you'd also upload styles/common.css).
Most FTP clients will allow you to just select the files to upload and move them so they all upload one after another, without having to do them one at a time.
