You can't. RTX is only supported on the Win10 version of Minecraft, not the Java version. And the Java version is the only one mods run on.
That said, to get shaders of any description back in the 1.7.10 era, you need Optifine. However, in 1.7.10, it did NOT play very well with how most mods handle their shaders internally, and would frequently cause graphical bugs when attempting to use it in conjunction with a mod pack of any kind. You could try to get SUSE Ultra Shaders running, but it is a whole process back in 1.7.10 (much easier these days), and is almost certain to cause graphical errors when attempting to render modded content.