The responsive page display takes priority over any image dimensions as a smart fallback; so mobile users can see a huge image on their tiny screens (otherwise they'd be bombarded by something that'd require them to scroll).
The images are processed with the compression settings you setup via the attachments settings; allowing an 8MB upload will allow a user to upload a huge image, that'll be compressed down.
The thumb max height/width affects the display of the thumbnail, medium I don't believe is used anymore, full is the max size the image can be stored as on the server.
If you look at the image information of my attached image you'll see that its scaled to 200x200 (thumb width) as a 30.73KB image (it was 49KB, so it was compressed); since the image was only 600x600 it didn't reach the full size max dimensions, so it retains the dimensions of the original.
If you allow 8MB-12MB uploads (which I recommend, since it's about the highest I've seen from cellphones), you'll end up with images around 400k-700k (depending on compression and full size dimension). They'll be resized to the full size dimensions only if the image exceeds that dimension (I allow 1920 as the max).
The FILESIZE is determined based on the image dimensions and the compression; it'll vary by image, especially using an image with a lot of detail.
When you click on an image and it opens in the lightbox, or you display inline, it should display the image with the css "max-width: 100%;"; which means as wide as the screen allows before trying to stretch the page.
I've been doing some testing, and I have some questions.
I uploaded a 3.8 MB image as an attachment. Instead of being displayed inline, it displayed as a link. On the hard drive, it was 2.2MB, which is 60% of its original size, so the compression is working as expected. But I *thought* that the file's size would be reduced to 650 pixels (the setting we have in attachments), and it was not.
When I deleted the post, the attachment was deleted as well, although its record was retained in the database.
The end filesize depends on the image quality you allow (defaults are thumb 200, medium 480, full 1920) and the max dimension you allow (if an image doesn't meet the dimension it will not be resized; the defaults are thumb 80%, medium 70%, full 60%), the level of detail the image has, and the Image Library you're utilizing to process images.
ImageMagick will process images more efficiently, and will more aggressively compress images; the downside is that it doesn't come pre-bundled with PHP like GD (it has to be installed to the local filesystem). Your host likely has it installed, you just have to populate the paths in the UBB.threads "Paths" configuration. I believe there was previously an issue with IM processing image dimensions (the system was rewritten in v7.6.2, whereas the following link is a patch for v184.108.40.206), that discussion can be found here.
The "Max File Size for Displaying Attached Images Inline" setting in the Attachments Settings will allow you to limit the images being displayed to the browser based on how large the image is; I wouldn't have this setting much higher than about a meg as your mobile users will have some pretty large images to push through their data plan.