What are my GPU upgrade options?
Your system will handle the most current RTX 2000 series graphics cards.
You would need to simply measure the inside available space and make sure its enough room to handle what ever card you choose. You also need to be aware of your power requirements. You currently have a 500w PSU. Make sure what ever new card you get will not require more than that. Also make sure the available power cable will work. I dont know if yours has a 6 pin or 8 pin but if you don't have the correct plug or enough power, you would have to up grade the PSU to match.
I assume you need more video memory than what the current 3GB 1060 has available. I would look at something with 6GB or better.
Would a RAM upgrade be worthwhile?
You already have 16 GB of memory. You shouldn't need more than that for gaming.