Wed, Nov 12 · 6:30 PM CST

•

Online

"Round Robin DB" (RRD) tutorial for troubleshooting.

Online

by Grant Taylor. I've been recording stats in Round Robin Databases (RRD files), doing graphs as I troubleshoot the Nagios server at work.

• This morning I spent some time writing some Perl to parse the
  output of `netstat --all --numeric --tcp --udp` to calculate the number
  of connections in various states, both inbound and outbound.
• I have something like 27 different data points that I'm graphing.
• As has been the case with three other (sets of) graphs that I've
  created for this, I've asked questions within a few minutes of looking
  at -> consuming -> using the graphs.
• I first started collecting, recording, and graphing the number of
  Nagios tests that were in the `Unknown` (3) state.
• That turned out to be quite enlightening.
• We were seeing periodic rapid swings from a few hundred to nearly
  two thousand tests in the `Unknown` (3) state.
• The visual nature of the graph showed that the number of tests in
  the `Unknown` (3) state was ebbing and flowing.
• It also showed some atypical / jittery / not-smooth spikes as
  opposed to the ramp-up / ramp-down that one might expect.
• As in nearly triple in a matter of a few minutes and then taper
  back down a few minutes later.
• It turns out that Gearman / gearmand -- which we're using behind
  Nagios for ${QUESTIONABLE_REASONS} -- was running out of file descriptors.
• Once I raised the max file handles in the systemd unit file, things
  were much smoother.
• So I started collecting, recording, and graphing the number of open
  file descriptors.
• That too was quite telling.
• Gearman / gearmand was averaging about 1200 file descriptors
  ranging from a few hundred to upwards of 1800.
• While working on the file descriptor problem I noticed that
  connections to Gearman / gearmand would block until there was a free
  file descriptor to accept the connection.
• Remember, TCP connections use a file descriptor.
• Knowing that slow connections to Gearman / gearmand were a symptom, I started collecting, recording, and graphing connection latency. That graph didn't show much of a symptom to chase. But it did provide a LOT of data in that I now know that the vast majority of the time, connections to Gearman / gearmand take < 20 ms.
• So I have statistics / data to back monitoring -> alerting thresholds.
• I noticed that the number of open file descriptors drops from
  nominally 1200 to below 500 during times that we're having problems.
• So, this morning I wrote about 210 lines of Perl to parse the
  output of `netstat --all --numeric --tcp --udp` to collect and record
  connection data.
• Graphing of said connection data yielded questions in a matter of
  minutes. Not the least of which is that the vast majority of the connections are inbound to Gearman / gearmand, not outbound like I thought they were.
• I determined inbound vs outbound by looking at the listening
  sockets in the `netstat` output. I defined listening as the remote IP being 0.0.0.0 (IPv4) or ::(IPv6) and the ports thereon to be `*`.
• Knowing what is listening allowed me to compare the local IP & port
  connections use with the list of listeners.
• If the local IP & port is a listener, then the connection is an
  inbound connection.
• If the local IP & port isn't a listener, then the connection is an
  outbound connection.
• So now I have more data and am effectively waiting for a problem
  event to analyze the data that I'm collecting during the event for a
  problematic time-frame after-the-fact.

BASIC Tutorial: Troubleshooting with a Round Robin DB by Grant Taylor
The above Basic Tutorial is at 6:30pm.

~7:00pm Announcements & short Q&A

~7:15pm MAIN Topic: Evolving "ifconfig & route" to "ip route" by Lee_Lammert.

Learn how to replace deprecated "ifconfig, route, & netstat" commands with the modern "ip & ss" commands. Clearer output, better troubleshooting, full IPv6 support, and future-proof networking.

"ip: Because Real Admins Don't Use ifconfig Anymore"

The above are both Wed 12 Nov 2025 =≠=≠=≠=≠≠≠====≠

EVERY MONTH:http://www.sluug.org/

The URL link to Zoom or Jitsi connection instructions for this meetng is posted earlier on the day of the meetng, at the above home page. It is the link called "linked here".

ONLINE MEETINGS ONLY until further notice.* ONLINE sessions will use a remote video meeting service.* HOW TO CONNECT instructions will be on the https://www.sluug.org/ web page and on our mailing lists. Note that your browser cache may need to be refreshed each time you check the web page for the instructions.

We will open the remote session at about 6:00 PM, so that you can join early to test sharing your microphone, screen and video camera.

Meetings are every 2nd Wednesday of the month from 6:30 PM to 9:00 PM.

EVERY mtng:

www.sluug.org/next

6:30pm
BASE Tutorial

~7:00pm Announcements & general Q&A

~7:15pm
MAIN:

The URL link to instructions at the www.sluug.org home page, are posted the day prior to this remote ZOOM meeting. The "next" link is called "linked here". You may have to refresh your browser after each time you check.

ONLINE MEETINGS ONLY until further notice.
ONLINE session will use remote video software.
CONNECTION instructions will be on the www.sluug.org web page and our mailing lists. Note that your browser cache may need to be refreshed each time you check the above web page for the instructions. We will open the remote session at about 6:00 PM, so that you can join early to test your microphone, screen and video sharing.

Details will be published on https://www.sluug.org/ when available.

Wed, Nov 12 · 7:00 PM CST

•

Online

Evolve "ifconfig & route"cmds into modern "ip route" cmd.

Online

Learn how to replace deprecated "ifconfig, route, & netstat" commands with the modern "ip & ss" commands. Clearer output, better troubleshooting, full IPv6 support, and future-proof networking.

"ip: Because Real Admins Don't Use ifconfig Anymore"
MAIN Topic: Evolving "ifconfig & route" to "ip route"
Speaker: Lee_Lammert.

The above MAIN is at ~7:15pm. It is immediately PRECEDED at
~7:00pm by Announcements & short Q&A which is immediately preceeded at
6:30pm by

BASIC Tutorial: Troubleshooting with a Round Robin DB by Grant Taylor

• For the last week or so I've been spending a lot of time recording
  statistics in Round Robin Databases (RRD files) and generating graphs of things as I try to troubleshoot some problems with the Nagios server at work.
• This morning I spent some time writing some Perl to parse the
  output of `netstat --all --numeric --tcp --udp` to calculate the number
  of connections in various states, both inbound and outbound.
• I have something like 27 different data points that I'm graphing.
• As has been the case with three other (sets of) graphs that I've
  created for this, I've asked questions within a few minutes of looking
  at -> consuming -> using the graphs.
• I first started collecting, recording, and graphing the number of
  Nagios tests that were in the `Unknown` (3) state.
• That turned out to be quite enlightening.
• We were seeing periodic rapid swings from a few hundred to nearly
  two thousand tests in the `Unknown` (3) state.
• The visual nature of the graph showed that the number of tests in
  the `Unknown` (3) state was ebbing and flowing.
• It also showed some atypical / jittery / not-smooth spikes as
  opposed to the ramp-up / ramp-down that one might expect.
• As in nearly triple in a matter of a few minutes and then taper
  back down a few minutes later.
• It turns out that Gearman / gearmand -- which we're using behind
  Nagios for ${QUESTIONABLE_REASONS} -- was running out of file descriptors.
• Once I raised the max file handles in the systemd unit file, things
  were much smoother.
• So I started collecting, recording, and graphing the number of open
  file descriptors.
• That too was quite telling.
• Gearman / gearmand was averaging about 1200 file descriptors
  ranging from a few hundred to upwards of 1800.
• While working on the file descriptor problem I noticed that
  connections to Gearman / gearmand would block until there was a free
  file descriptor to accept the connection.
• Remember, TCP connections use a file descriptor.
• Knowing that slow connections to Gearman / gearmand were a symptom, I started collecting, recording, and graphing connection latency. That graph didn't show much of a symptom to chase. But it did provide a LOT of data in that I now know that the vast majority of the time, connections to Gearman / gearmand take < 20 ms.
• So I have statistics / data to back monitoring -> alerting thresholds.
• I noticed that the number of open file descriptors drops from
  nominally 1200 to below 500 during times that we're having problems.
• So, this morning I wrote about 210 lines of Perl to parse the
  output of `netstat --all --numeric --tcp --udp` to collect and record
  connection data.
• Graphing of said connection data yielded questions in a matter of
  minutes. Not the least of which is that the vast majority of the connections are inbound to Gearman / gearmand, not outbound like I thought they were.
• I determined inbound vs outbound by looking at the listening
  sockets in the `netstat` output. I defined listening as the remote IP being 0.0.0.0 (IPv4) or ::(IPv6) and the ports thereon to be `*`.
• Knowing what is listening allowed me to compare the local IP & port
  connections use with the list of listeners.
• If the local IP & port is a listener, then the connection is an
  inbound connection.
• If the local IP & port isn't a listener, then the connection is an
  outbound connection.
• So now I have more data and am effectively waiting for a problem
  event to analyze the data that I'm collecting during the event for a
  problematic time-frame after-the-fact.

The above are both Wed 12 Nov 2025 =≠=≠=≠=≠≠≠====≠

EVERY MONTH:http://www.sluug.org/

The URL link to Zoom or Jitsi connection instructions for this meetng is posted earlier on the day of the meetng, at the above home page. It is the link called "linked here".

ONLINE MEETINGS ONLY until further notice.* ONLINE sessions will use a remote video meeting service.* HOW TO CONNECT instructions will be on the https://www.sluug.org/ web page and on our mailing lists. Note that your browser cache may need to be refreshed each time you check the web page for the instructions.

We will open the remote session at about 6:00 PM, so that you can join early to test sharing your microphone, screen and video camera.

Meetings are every 2nd Wednesday of the month from 6:30 PM to 9:00 PM.

EVERY mtng:

www.sluug.org/next

6:30pm
BASE Tutorial

~7:00pm Announcements & general Q&A

~7:15pm
MAIN:

The URL link to instructions at the www.sluug.org home page, are posted the day prior to this remote ZOOM meeting. The "next" link is called "linked here". You may have to refresh your browser after each time you check.

ONLINE MEETINGS ONLY until further notice.
ONLINE session will use remote video software.
CONNECTION instructions will be on the www.sluug.org web page and our mailing lists. Note that your browser cache may need to be refreshed each time you check the above web page for the instructions. We will open the remote session at about 6:00 PM, so that you can join early to test your microphone, screen and video sharing.

Details will be published on https://www.sluug.org/ when available.

Thu, Nov 20 · 6:30 PM CST

•

Online

STLLINUX - St. Louis Linux Users Group

Online

www.stllinux.org/next

The url link to this ZOOM meeting is posted earlier on the day of the meeting at the www.stllinux.org home page. It is the "next" link called "linked here". You may have to refresh your browser after each time you check.

TOPIC: To Be Determined (TBD)
Presenter: TBD
Normally an Abstract/Background/Details/History/Outline/Summary/Walk-through will be listed when available.

ONLINE MEETINGS ONLY until further notice.
ONLINE session will use remote video software.
HOW TO CONNECT instructions on https://stllinux.org/ web page and our mailing lists. Note that your browser cache may need to be refreshed each time you check the above web page for the instructions. We will open the remote session at about 6:00 PM Central Standard Time ( CST ), so that you can join early to test your microphone, screen and video sharing.

The Saint Louis MO, STL Linux Users Group (STLLUG) meets monthly to talk about Linux. This GNU/Linux Users Group usually holds its meetings on the third or fourth Thursday of every month. Meetings are free and open to everyone.

At 6:30 PM CST we start with introductions, announcements, current events of interest, and a general CALL FOR HELP segment. Then we will go into the presentation of our main topic, sometime around or after 7:00 PM CST.

Tue, Nov 25 · 6:00 PM CST

•

Online

NEWLINUX ~ New Linux Users Group for beginners

Online

https://newlug.sluug.org/next.txt
The URL link instructions to this Jitsi meeting are posted at the above home page on the day prior. It is the link called "linked here" on the home page above. You may have to refresh you browser each time you check.

ONLINE MEETINGS ONLY until further notice.
ONLINE session testing will use JITSI remote video meeting software.
CONNECT INSTRUCTIONS to be posted on the above web page and our mailing lists. Note that your browser cache may need to be refreshed each time you check the above web page for the instructions.

A Linux Users Group for the newcomers to Linux.

Topics begin with whatever the newest user needs to know.

Meetings are free and open to the public. You are invited to attend our meetings. They are usually held on the fourth Tuesday of every month from 6:30 PM until 9:00 PM. The December meetings might be on the third Tuesday.

We will open the remote session at about 6:00 PM, so that you can join early to test your microphone, screen and video sharing.

We have no membership dues or other charges for the meetings. Any room or service fee is paid for by SLUUG. Please join us at our next meeting or event for answers, fun, knowledge and networking.