wish helps you I'll summarize Paul R's comments as the answer, which has solved my problem: 50 ms corresponds to a total buffer size of around 2048 at a 44.1 kHz sample rate, which doesn't seem unreasonable given that you have both a record and a playback path.
this one helps. If you have control over the code of the page you're trying to measure the requests from, you could use the navigation timing API (see MDN and a useful blog post), to determine the time to first byte, which sounds like the latency you're after. You can also measure other factors, some that might affect overall latency, like DNS lookup, too.
how to show latency information in summary report in Jmeter