Autofs: too many open files issue
Hi, We are running AlmaLinux 9.3 on several nodes that normally use autofs to mount different kinds of remote directories. Since the last week, we are seeing these errors: automount[2150866]: open_pipe:161: failed to open pipe: Too many open files And automount stops working. It does not seem that any user has arrived at the maximum number of open files. Do you have any idea where we can look to find a solution? Thank you in advance. Best regards, Carles
Il 17/04/24 17:48, Carles Acosta Silva ha scritto:
Hi,
We are running AlmaLinux 9.3 on several nodes that normally use autofs to mount different kinds of remote directories. Since the last week, we are seeing these errors:
automount[2150866]: open_pipe:161: failed to open pipe: Too many open files
And automount stops working. It does not seem that any user has arrived at the maximum number of open files.
Do you have any idea where we can look to find a solution?
Thank you in advance.
Best regards,
Carles
_______________________________________________ AlmaLinux Users mailing list -- users@lists.almalinux.org To unsubscribe send an email to users-leave@lists.almalinux.org
Hi Carles, You should check how many files are opened on the system and check what is the maximum number limit of open file for the system. You should check also what is the user that receive the "too many open files" error (I suppose root?) and check how many files can be opened for that specified user (ulimit -n could help). You could run also lsof to check how many files are opened and what is opened. Could you consider something program that does not close a resource on file? I don't know how works automount but maybe some resource is not umounted? Hope that helps. Alessandro.
We've just hit this issue on some of our alma 9.3 hosts The triage was to look at the number of file descriptors that automount had open. ls -1 /proc/$(pgrep automount)/fd | wc -l Most clients are well below 100, but a few are in the thousands, and when they hit the default soft limit of 20480 (cat /proc/($pgrep automount)/limits) autofs breaks - starting with too many open files. systemctl restart autofs resolves it, and seems to 'clear' the problem on any clients straying up past 1000, but it seems there's got to be some sort of file descriptor 'leaking' occurring here.
Hi Ed,
We found this bug:
https://www.findbugzero.com/operational-defect-database/vendors/rh/defects/R...
https://issues.redhat.com/browse/RHEL-33796
After updating the packages autofs and sssd to versions:
*autofs-5.1.7-58.el9.x86_64* and *sssd-2.9.4-6.el9_4.x86_64* respectively,
it does not happen again for us. We are still monitoring the behaviour but
we have at least one month without seeing the error.
Cheers
Carles
Missatge de Ed Rolison
We've just hit this issue on some of our alma 9.3 hosts The triage was to look at the number of file descriptors that automount had open.
ls -1 /proc/$(pgrep automount)/fd | wc -l
Most clients are well below 100, but a few are in the thousands, and when they hit the default soft limit of 20480 (cat /proc/($pgrep automount)/limits) autofs breaks - starting with too many open files.
systemctl restart autofs resolves it, and seems to 'clear' the problem on any clients straying up past 1000, but it seems there's got to be some sort of file descriptor 'leaking' occurring here. _______________________________________________ AlmaLinux Users mailing list -- users@lists.almalinux.org To unsubscribe send an email to users-leave@lists.almalinux.org
We discovered this bug. https://www.findbugzero.com/operational-defect-database/vendors/rh/defects/R... https://issues.redhat.com/browse/RHEL-33796 https://bitlife-online.io After updating autofs and sssd to versions *autofs-5.1.7-58.el9.x86_64* and *sssd-2.9.4-6.el9_4.x86_64*, we no longer have this issue. We are still monitoring the behavior, but have not seen the issue in over a month.
participants (4)
-
Alessandro Baggi -
Carles Acosta Silva -
Daniel Howard -
Ed Rolison