The 23 msec RTT suggests that you may be pinging e.g. from Minneapolis or Dallas to a San Francisco datacenter.
There's more than one explanation for missing packets, including
buffer bloat,
ordinary congestion, and good old power fail.
Some of the difficulty may be local.
I encourage you to also routinely ping a LAN-attached host,
plus a "nearby" host that is at least a kilometer away.
This will help you separate "near" from "far" effects,
such as episodes that your WAN link experiences.
Fundamentally ICMP is not an "app to app" protocol.
Rather, it is a "host to host" protocol; a randomized
payload, reflected back to the sender, helps us to
distinguish what one or another ping process sent.
Embrace that aspect of it.
In addition to logging stdout of ping, also run
tcpdump -s 1500 -w ping.pkts icmp.
This will capture all inbound and outbound pings, plus
some other things that routers occasionally send.
If possible, run that wireshark / tcpdump command
on a distant WAN-attached host that you have sudo on.
If possible, do periodic large file transfers to
that WAN-attached host, to assess how packet loss rates
and available bandwidth vary over the course of the day.
Failing that, do file transfers over your WAN link to a host you control.
Periodically record netstat -s output on the source host,
and if possible on the target host.
Pay close attention to the "Icmp" and "IcmpMsg" sections.
Your goal in all of this is to tease apart
effects happening near the source host,
near the target host,
or "internet weather" effects on the amorphous cloud in between.
For that last part, you might choose to ping
router IPs that show up in traceroute output.
However, 100 msec is much too fast to be pinging ISP / NSP infrastructure.
Send just a handful of pings each minute,
lest your source IP be blacklisted.