Troubleshooting Custom audispd Plugin

[Osama Elnaggar]

Hi,

I'm working on a custom audispd plugin written in Python 3. It’s a work in
progress and I’ve successfully run it numerous times as an audispd plugin.
However, I sometimes make modifications that result in the audispd plugin
failing and I end up with the following in /var/log/syslog

Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
terminated unexpectedly
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
restarted
...

This is repeated several times until audispd gives up and I see the
following message:

Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
exceeded max_restarts

To troubleshoot, I modify my code to read from /var/log/audit/audit.log
instead. I modify a single line (with fileinput.input() to read from
myfile as shown in the commented line below).

Here is the code snippet (a colorized easier to read version is available
here - https://pastebin.com/84Nxu3Rp):

# let us initialize the AuParser
aup = auparse.AuParser(auparse.AUSOURCE_FEED)

# we initalize the callback to be fn_process_event
aup.add_callback(fn_process_event, None, None)

myfile = "/var/log/audit/audit.log"

while True:
try:
# we read in line by line from stdin
for line in fileinput.input():
#for line in fileinput.input(myfile):
aup.feed(line)
except:
logger.error("Fatal error in while loop", exc_info=True)

# we flush the feed when we quit
aup.flush_feed()

Any suggestions on how to troubleshoot these types of issues when reading
from a file works fine without issue but running it as a plugin fails as
shown in /var/log/syslog? Thanks.

[Steve Grubb]

Hi,
I'm working on a custom audispd plugin written in Python 3. It’s a work in
progress and I’ve successfully run it numerous times as an audispd plugin.
However, I sometimes make modifications that result in the audispd plugin
failing and I end up with the following in /var/log/syslog
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
terminated unexpectedly
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
restarted
...
This is repeated several times until audispd gives up and I see the
Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
exceeded max_restarts
To troubleshoot, I modify my code to read from /var/log/audit/audit.log
instead. I modify a single line (with fileinput.input() to read from
myfile as shown in the commented line below).
Here is the code snippet (a colorized easier to read version is available
# let us initialize the AuParser
aup = auparse.AuParser(auparse.AUSOURCE_FEED)
# we initalize the callback to be fn_process_event
aup.add_callback(fn_process_event, None, None)
myfile = "/var/log/audit/audit.log"
# we read in line by line from stdin
aup.feed(line)
logger.error("Fatal error in while loop", exc_info=True)
# we flush the feed when we quit
aup.flush_feed()
Any suggestions on how to troubleshoot these types of issues when reading
from a file works fine without issue but running it as a plugin fails as
shown in /var/log/syslog? Thanks.

All plugins have a requirement to take events from stdin. As long as it
expects strings (which is the way that auparse wants them), then all you have
to do is:

ausearch --start boot --raw | ./plugin

You can also save raw logs with ausearch and cat them into the plugin. This
is helpful when you get a problem down to a certain series of events and you
don't want to go through a thousand events before the problem sequence.

-Steve

[Osama Elnaggar]

Hi Steve,

I tried it but the problem still only shows up when it runs as a plugin.
Also, the script basically does some processing on the records and extracts
certain data from records of interest, so it should run fine regardless of
the input source. It seems to fail immediately when run as a plugin. Any
other suggestions on troubleshooting the discepancy?

PS. I also read your very useful auditd tutorials over here -
https://security-plus-data-science.blogspot.com/ Thanks.
--
Osama Elnaggar

Hi,
I'm working on a custom audispd plugin written in Python 3. It’s a work

in

progress and I’ve successfully run it numerous times as an audispd plugin.
However, I sometimes make modifications that result in the audispd plugin
failing and I end up with the following in /var/log/syslog
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
terminated unexpectedly
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
restarted
...
This is repeated several times until audispd gives up and I see the
Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
exceeded max_restarts
To troubleshoot, I modify my code to read from /var/log/audit/audit.log
instead. I modify a single line (with fileinput.input() to read from
myfile as shown in the commented line below).
Here is the code snippet (a colorized easier to read version is available
# let us initialize the AuParser
aup = auparse.AuParser(auparse.AUSOURCE_FEED)
# we initalize the callback to be fn_process_event
aup.add_callback(fn_process_event, None, None)
myfile = "/var/log/audit/audit.log"
# we read in line by line from stdin
aup.feed(line)
logger.error("Fatal error in while loop", exc_info=True)
# we flush the feed when we quit
aup.flush_feed()
Any suggestions on how to troubleshoot these types of issues when reading
from a file works fine without issue but running it as a plugin fails as
shown in /var/log/syslog? Thanks.

All plugins have a requirement to take events from stdin. As long as it
expects strings (which is the way that auparse wants them), then all you
have
to do is:

ausearch --start boot --raw | ./plugin

You can also save raw logs with ausearch and cat them into the plugin. This
is helpful when you get a problem down to a certain series of events and
you
don't want to go through a thousand events before the problem sequence.

-Steve

[Steve Grubb]

Hello,

Post by Osama Elnaggar
I tried it but the problem still only shows up when it runs as a plugin.
Also, the script basically does some processing on the records and extracts
certain data from records of interest, so it should run fine regardless of
the input source. It seems to fail immediately when run as a plugin. Any
other suggestions on troubleshooting the discepancy?

I don't know if there are any permission restrictions by AppArmor or SE Linux
if you have those running. I don't know if you are logging errors when they
occur. But my guess would be something is throwing an uncaught exception.
Which might be caused by MAC permissions. Just a guess.

-Steve

Post by Osama Elnaggar
PS. I also read your very useful auditd tutorials over here -
https://security-plus-data-science.blogspot.com/ Thanks.

Hi,
I'm working on a custom audispd plugin written in Python 3. It’s a work

in

progress and I’ve successfully run it numerous times as an audispd

plugin.

However, I sometimes make modifications that result in the audispd plugin
failing and I end up with the following in /var/log/syslog
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
terminated unexpectedly
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
restarted
...
This is repeated several times until audispd gives up and I see the
Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
exceeded max_restarts
To troubleshoot, I modify my code to read from /var/log/audit/audit.log
instead. I modify a single line (with fileinput.input() to read from
myfile as shown in the commented line below).
Here is the code snippet (a colorized easier to read version is available
# let us initialize the AuParser
aup = auparse.AuParser(auparse.AUSOURCE_FEED)
# we initalize the callback to be fn_process_event
aup.add_callback(fn_process_event, None, None)
myfile = "/var/log/audit/audit.log"
# we read in line by line from stdin
aup.feed(line)
logger.error("Fatal error in while loop", exc_info=True)
# we flush the feed when we quit
aup.flush_feed()
Any suggestions on how to troubleshoot these types of issues when reading
from a file works fine without issue but running it as a plugin fails as
shown in /var/log/syslog? Thanks.

All plugins have a requirement to take events from stdin. As long as it
expects strings (which is the way that auparse wants them), then all you have
ausearch --start boot --raw | ./plugin
You can also save raw logs with ausearch and cat them into the plugin. This
is helpful when you get a problem down to a certain series of events and you
don't want to go through a thousand events before the problem sequence.
-Steve

[Osama Elnaggar]

Hi,

Just in case anyone runs into something similar in the future, it was due
to a Python library that I had added to my script and was installed was in
my normal user's environment (+ in sudo -s) but not in the system’s
environment used by audisp. I discovered this by stracing the audisp
process. Thank you for your help.
--
Osama Elnaggar

On September 7, 2018 at 11:42:27 PM, Steve Grubb (***@redhat.com) wrote:

Hello,

Post by Osama Elnaggar
I tried it but the problem still only shows up when it runs as a plugin.
Also, the script basically does some processing on the records and extracts
certain data from records of interest, so it should run fine regardless of
the input source. It seems to fail immediately when run as a plugin. Any
other suggestions on troubleshooting the discepancy?

I don't know if there are any permission restrictions by AppArmor or SE
Linux
if you have those running. I don't know if you are logging errors when they
occur. But my guess would be something is throwing an uncaught exception.
Which might be caused by MAC permissions. Just a guess.

-Steve

Post by Osama Elnaggar
PS. I also read your very useful auditd tutorials over here -
https://security-plus-data-science.blogspot.com/ Thanks.

Hi,
I'm working on a custom audispd plugin written in Python 3. It’s a work

in

progress and I’ve successfully run it numerous times as an audispd

plugin.

However, I sometimes make modifications that result in the audispd plugin
failing and I end up with the following in /var/log/syslog
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3
terminated unexpectedly
Sep 6 20:52:05 ubuntu-hypervisor audispd: plugin /usr/bin/python3 was
restarted
...
This is repeated several times until audispd gives up and I see the
Sep 6 20:52:14 ubuntu-hypervisor audispd: plugin /usr/bin/python3 has
exceeded max_restarts
To troubleshoot, I modify my code to read from /var/log/audit/audit.log
instead. I modify a single line (with fileinput.input() to read from
myfile as shown in the commented line below).
Here is the code snippet (a colorized easier to read version is available
# let us initialize the AuParser
aup = auparse.AuParser(auparse.AUSOURCE_FEED)
# we initalize the callback to be fn_process_event
aup.add_callback(fn_process_event, None, None)
myfile = "/var/log/audit/audit.log"
# we read in line by line from stdin
aup.feed(line)
logger.error("Fatal error in while loop", exc_info=True)
# we flush the feed when we quit
aup.flush_feed()
Any suggestions on how to troubleshoot these types of issues when reading
from a file works fine without issue but running it as a plugin fails as
shown in /var/log/syslog? Thanks.

All plugins have a requirement to take events from stdin. As long as it
expects strings (which is the way that auparse wants them), then all you have
ausearch --start boot --raw | ./plugin
You can also save raw logs with ausearch and cat them into the plugin. This
is helpful when you get a problem down to a certain series of events and you
don't want to go through a thousand events before the problem sequence.
-Steve