If your test numbers are right, the short answer is that you've got too much CO2 in your tank, which is dropping the pH and is probably somewhat toxic even without the lowered pH causing additional stress.
You say your tap water is KH 4, your tank is KH 6, and the low pH you observed was 6.4. From the chart, it looks like you had up to 50 - 75 ppm of CO2 in your tank, which is definitely too much.
What volume is your tank? Maybe you'd be better off with a bigger one!
Or a smaller yeast reactor... You can also add a pinch of baking soda to your yeast / sugar mix to make it go slower, but for a longer period of time.
Another possibility is contamination from the yeast reactor. If the yeast has been going for a while, there's a good bit of alcohol in there, and if the reactor is frothing up and overflowing into the tank at all, that could be another problem. Don't fill your reactor too close to the top when you make the yeast / sugar mix.
Baking soda (sodium bicarbonate, Na.HCO3) increases your KH. KH test kits measure how hard it is to drop the pH of your water. More drops means it takes more acid to reach a low pH. CO2 is an acid when you put it in your tank, so having a higher KH will prevent reasonable amounts of CO2 from dropping your pH off the chart.
Another way of looking at this is that the pH in your tank is determined by the balance of carbonate and carbonic acid (CO2 + water). Baking soda adds carbonate.
GH is a measure of the magnesium and calcium in your water. Those have no effect on pH, so don't worry about it in the context of this problem.
Get familiar with your KH kit until you're confident with your ability to get consistent results, and then keep your KH where you want it by adding baking soda. Don't go nuts - I recommend boosting your tap water by a maximum of 4 KH. If you need to go higher, you'll turn to other carbonates.
1/4 tsp will raise 50 liters by ~1 degree KH. Just add the desired amount at every partial change. eg. if you change 50 l at a time, and you want to go from KH 2 to KH 4, add 1/2 tsp. at each change. Dissolve it in 500 ml of water and pour it slowly into the tank. I don't consider this 'chasing the dragon' any more than adding CO2, or adding some salt to a brackish tank. Carbonates are the buffers that are in natural waters, so you're not creating some artificial system with a bunch of side effects.