Gravity will act on both aircraft equally, so if weight was the only force acting on the aircraft with no air resistance or other forces, they would both hit the ground at the same time. But the wings act against weight and hold the aircraft in the air for longer.
More weight = a lower resultant force of the weight against the lift. Less lift = in the air a shorter time.
So I would say the lighter aircraft would be able to glide for longer.
You're forgetting that we can change how the wing deals with gravity, by changing the A of A. The wing does not fight gravity by creating drag, like an old-fashioned round parachute, it does it by moving the air a certain way. That way is variable, thanks to the elevator.
If the pilot of the heavier glider cannot change A of A and thus airspeed, yes, he will not cover the same distance from the same altitude. His A of a will be too high, to the point where the lift gained by more A of A will not overcome the lift lost by the lower airspeed (remember, speed and angle work together to produce lift). Pitching up more will just make things worse, just as they do if you try to maintain a climb with pitch only in any aircraft. So he has to lower the nose- just a little- to pick up a little more speed. The vertical speed will increase, but it will go farther in the time it takes to run out of altitude, because it is moving forward faster, too.
On the other hand, let's say the competition is to stay aloft the longest period of time, in still air (no lifting air of any kind). This is completely different from trying to achieve a given distance from a given altitude!! Best-glide speed is not the same as minimum sink speed!!
As long as the heavier glider can adjust its airspeed (by changing a of A), it can maintain the same vertical speed as the lighter glider, and stay up just as long.
Keep adding weight to it, and yes, eventually the wing won't lift so well regardless of A of A, but with a 2-seater,
designed so that the same rate of vertical speed can be achieved with one or two people aboard, if the weight is within limits, the same endurance can be achieved... IF the airspeed is changed. The heavy glider, gliding at a higher indicated airspeed, will not necessarily be sinking faster, IF that indicated airspeed is yielding "minimum sink". Minimum sink is achieved by assigning a specific A of A, and thus airspeed, to the wing... it is a V-speed. And V-speeds change with load factor, including load increased because of a higher payload.