I have a residential proxy located in the U.S. The problem is I can only use these on browsers supporting proxy.
What I want is to setup the proxy on my VPS and have my phone connect to it using wireguard.
Is this possible?
This is not possible. A socks proxy forwards tcp connections over another TCP connection. A wireguard vpn sends encrypted generic IP packets over udp. A socks proxy can't understand the types of things wireguard sends.
However, you could just install wireguard on both your vps and your phone, and you probably wouldn't even need the proxy. If your VPS is hosted by a big public cloud provider, be aware that many sites restrict incoming traffic from known up ranges because public cloud vms often used by spammers.
Iptables has support for forwarding network traffic via a proxy, the keyword is "tproxy".
I don't think you can use socks5 directly, but you can translate tproxy to socks5 by installing squid listening on localhost for tproxy connections and setting the socks5 as an upstream proxy.
If it reads like a bunch of obsolete tech, it's because it is, but it gets the job done.