First and foremost, your gauges shouldn't really change until your tank is almost totally empty. I am talking a few days to a day, depending on how much CO2 you use daily. The high pressure one should really stay about the same all the time. The only thing that would effect this would be the temperature. The other gauge shows the working pressure. This should only change if you adjust the working pressure. If it's not adjustable, it won't change.
It also sounds like you have a bad check valve. Just replace it.
Lastly, depending on what type of diffuser you use, it can take awhile to get things started. I actually have 2 different diffusers in my tank right now, an inline that requires high working pressure, and a regular glass one. My reason for that is just an experiment but the inline one can take more than a few minutes to get going, the glass one only takes like 30 seconds. It just has to build enough pressure to start working. So if it takes awhile, but works, it likely isn't a problem. It may take longer if you have a small leak though so you may want to check for leaks. If it never starts, you most definitely have a leak. Reactors shouldn't take more than a second to start working however.
It sounds like you may have everything under control, outside of the check valve and possible leaks. It's hard to tell if you have a leak so I am not assuming you do, but it's good to test. Soapy water is good to test, do all joints/elbows/check valves, etc.