I would highly recommend you think of an alternative plan before you dump that kind of money on a old, weak outdated system like that one. The AMD A8 5500 cpu was not a good gaming processor. You also have a minimal amount of ram installed.
IF you insist on installing a new gpu, I would not go higher than a GTX 1050.
There are mini versions that will not take up much room. Measure the distance inside the case to make sure the dimensions of the card will fit.