You're on the right track with using the scaling to change their size (eventually I want all the components to be able to be easily resized independently, but for now, they're mostly at a fixed size, but can be globally scaled up/down using the Scaler).
I'd recommend scaling with screen size, and using a reference resolution of 360x640 for portrait, or 640x360 for landscape (it doesn't *really* matter which one you pick, as the scaler swaps the x and y values upon screen rotation). This will give you a good 'normal' size for any phones and small tablets. For larger screens, you may want to use something larger like 540x960.
Let me know if that helps!