Batteries have internal resistance themselves. When a current is flowing that causes a battery voltage drop. Even a brand new battery will exhibit this. The question is how much voltage drop over how much time.
Any load will drain a battery but depending upon the battery capacity , load draw, battery age and charging regimen it may draw down quickly or not quickly.
How old are the batteries used for the fridge? What is the battery size? Maybe they are just getting to old which means capacity has been lost so the voltage will drop faster than when the batts. were new.
What current does the fridge draw, name and specification plate, when the batteries are fully charged?
There have been lots of threads here about how to help minimize fridge draw, power useage, by ensuring adequate exhausting of the heated air the fridge compressor pump removes from the fridge inside and the food. If the fridge cannot eliminate heat easily it will run and run and run pulling battery power.