Discussion:
Kernel 4.9.51 null pointer dereference add_rule with null key
Preston Bennes
2018-08-17 20:04:11 UTC
Permalink
Hey Audit team,

I've got a kernel null pointer deref oops on Amazon Linux
kernel 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes
that spawned were stuck in D wait on some audit related syscall. This
exhausted system file handles and the box ended up needing to be rebooted
out-of-bounds. The audit handle was held by
https://github.com/facebook/osquery/.

It looks like there was an issue with audit sending to osquery
(netlink_unicast sending to pid 21233, error -111) followed by something
(presumably also osquery) attempting to 'op="add_rule" key=(null)'.

Here's from /var/log/messages.
netlink_unicast sending to audit_pid=21233 returned error: -111
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982]
audit_log_lost: 12 callbacks suppressed
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983]
audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985]
audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233
auid=501 ses=102936 res=0
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986]
audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1
auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987]
audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1
old=1 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989]
audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096
old=4096 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990]
audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0
auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991]
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
unable to handle kernel NULL pointer dereference at 00000000000001e0
[<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336] PGD 0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
0000 [#1] SMP
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736]
Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E)
nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E)
isofs(E) ipv6(E) crc_
0 PID: 21372 Comm: osqueryd Tainted: G E
4.9.51-10.52.amzn1.x86_64 #1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620]
Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
ffff8807fa1c3b00 task.stack: ffffc900079a4000
0010:[<ffffffff814a76da>] [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
0018:ffffc900079a7c38 EFLAGS: 00010246
0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
0010 DS: 0000 ES: 0000 CR0: 0000000080050033
00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680]
00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540]
ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379]
ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201] Call
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137]
[<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561]
[<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642]
[<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718]
[<ffffffff81110ab2>] audit_receive+0x52/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713]
[<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943]
[<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236]
[<ffffffff81455818>] sock_sendmsg+0x38/0x50
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219]
[<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142]
[<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853]
[<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464]
[<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189]
[<ffffffff8145673e>] SyS_sendto+0xe/0x10
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952]
[<ffffffff81003854>] do_syscall_64+0x54/0xc0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995]
[<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00 81 c6
20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24 e0 01
00 00 48 89 45
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625] RIP
[<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253] RSP
<ffffc900079a7c38>
00000000000001e0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307] ---[
end trace 6a41b2274729ba2a ]---
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029]
audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59
success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2
ppid=4556 pid=2407
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741]
audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s"
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585]
audit: type=1307 audit(1534170895.379:117649235): cwd="/"
Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1
kernel oopses to Abrt
Paul Moore
2018-08-17 21:12:06 UTC
Permalink
On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes
Post by Preston Bennes
Hey Audit team,
Hey Audit user!
Post by Preston Bennes
I've got a kernel null pointer deref oops on Amazon Linux kernel 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that spawned were stuck in D wait on some audit related syscall. This exhausted system file handles and the box ended up needing to be rebooted out-of-bounds. The audit handle was held by https://github.com/facebook/osquery/.
It looks like there was an issue with audit sending to osquery (netlink_unicast sending to pid 21233, error -111) followed by something (presumably also osquery) attempting to 'op="add_rule" key=(null)'.
The first thing I need to comment on is that since this is a distro
specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel
there is a limit in the amount of help we can provide. Generally the
upstream list focuses on the upstream code and we leave support for
the distro specific builds/packages up to the distro. If you can
reproduce your problem on an upstream kernel, with a strong preference
for a *recent* upstream kernel, we can probably be of more help. I
simply don't know what patches Amazon merges into it's amzn1 kernels.

With that out of the way, I will add that v4.9.x is a rather old
kernel, and we've made a number of relevant improvements since the
v4.9 release and it is unlikely all of them have been backported to
your kernel. If possible, I would suggest upgrading to a newer kernel
as it may resolve your problem.

There is also a possible issue of osquery as an auditd replacement;
it's unlikely this is being caused by osquery, but we don't test with
osquery to that save level as we do with Steve's audit userspace
(auditd).
Post by Preston Bennes
Here's from /var/log/messages.
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904] audit: netlink_unicast sending to audit_pid=21233 returned error: -111
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982] audit_log_lost: 12 callbacks suppressed
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983] audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985] audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233 auid=501 ses=102936 res=0
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986] audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987] audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1 old=1 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989] audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096 old=4096 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990] audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991] audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 op="add_rule" key=(null) list=4 res=0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007] BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872] IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336] PGD 0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230] Oops: 0000 [#1] SMP
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736] Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E) nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E) isofs(E) ipv6(E) crc_
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036] CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E 4.9.51-10.52.amzn1.x86_64 #1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513] task: ffff8807fa1c3b00 task.stack: ffffc900079a4000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179] RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822] RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779] RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473] RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167] R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738] R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999] FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555] CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680] 00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540] ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379] ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137] [<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561] [<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642] [<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718] [<ffffffff81110ab2>] audit_receive+0x52/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713] [<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943] [<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236] [<ffffffff81455818>] sock_sendmsg+0x38/0x50
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219] [<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142] [<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853] [<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464] [<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189] [<ffffffff8145673e>] SyS_sendto+0xe/0x10
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952] [<ffffffff81003854>] do_syscall_64+0x54/0xc0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995] [<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840] Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00 81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24 e0 01 00 00 48 89 45
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625] RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253] RSP <ffffc900079a7c38>
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013] CR2: 00000000000001e0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307] ---[ end trace 6a41b2274729ba2a ]---
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029] audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59 success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2 ppid=4556 pid=2407
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741] audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s"
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585] audit: type=1307 audit(1534170895.379:117649235): cwd="/"
Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1 kernel oopses to Abrt
--
Linux-audit mailing list
https://www.redhat.com/mailman/listinfo/linux-audit
--
paul moore
www.paul-moore.com
Preston Bennes
2018-08-17 21:24:05 UTC
Permalink
Hey Paul,

My expectations are aligned to what you've pointed out already (not-vanilla
old kernel, user space client other than auditd).

Primarily what I'm looking for is a sanity check on the null pointer deref
on current vanilla kernel code because I think the rest of the issues stem
from that. My C isn't sharp enough to try my hand at a reproduction case,
though I would hope someone with the requisite expertise could put one
together relatively quickly. specifically this message:

audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes
Post by Preston Bennes
Hey Audit team,
Hey Audit user!
Post by Preston Bennes
I've got a kernel null pointer deref oops on Amazon Linux kernel
4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that
spawned were stuck in D wait on some audit related syscall. This exhausted
system file handles and the box ended up needing to be rebooted
out-of-bounds. The audit handle was held by
https://github.com/facebook/osquery/.
Post by Preston Bennes
It looks like there was an issue with audit sending to osquery
(netlink_unicast sending to pid 21233, error -111) followed by something
(presumably also osquery) attempting to 'op="add_rule" key=(null)'.
The first thing I need to comment on is that since this is a distro
specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel
there is a limit in the amount of help we can provide. Generally the
upstream list focuses on the upstream code and we leave support for
the distro specific builds/packages up to the distro. If you can
reproduce your problem on an upstream kernel, with a strong preference
for a *recent* upstream kernel, we can probably be of more help. I
simply don't know what patches Amazon merges into it's amzn1 kernels.
With that out of the way, I will add that v4.9.x is a rather old
kernel, and we've made a number of relevant improvements since the
v4.9 release and it is unlikely all of them have been backported to
your kernel. If possible, I would suggest upgrading to a newer kernel
as it may resolve your problem.
There is also a possible issue of osquery as an auditd replacement;
it's unlikely this is being caused by osquery, but we don't test with
osquery to that save level as we do with Steve's audit userspace
(auditd).
Post by Preston Bennes
Here's from /var/log/messages.
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904]
audit: netlink_unicast sending to audit_pid=21233 returned error: -111
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982]
audit_log_lost: 12 callbacks suppressed
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983]
audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985]
audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233
auid=501 ses=102936 res=0
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986]
audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1
auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987]
audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1
old=1 auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989]
audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096
old=4096 auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990]
audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0
auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991]
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007]
BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872]
IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336]
PGD 0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230]
Oops: 0000 [#1] SMP
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736]
Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E)
nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E)
isofs(E) ipv6(E) crc_
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036]
CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E
4.9.51-10.52.amzn1.x86_64 #1
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620]
Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513]
task: ffff8807fa1c3b00 task.stack: ffffc900079a4000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179]
RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>]
netlink_unicast+0x4a/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822]
RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065]
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779]
RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473]
RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167]
R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738]
R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999]
FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796]
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555]
CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401]
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725]
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680]
00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540]
ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379]
ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137]
[<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561]
[<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642]
[<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718]
[<ffffffff81110ab2>] audit_receive+0x52/0xa0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713]
[<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943]
[<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236]
[<ffffffff81455818>] sock_sendmsg+0x38/0x50
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219]
[<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142]
[<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853]
[<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464]
[<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189]
[<ffffffff8145673e>] SyS_sendto+0xe/0x10
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952]
[<ffffffff81003854>] do_syscall_64+0x54/0xc0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995]
[<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840]
Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00
81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24
e0 01 00 00 48 89 45
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625]
RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253]
RSP <ffffc900079a7c38>
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013]
CR2: 00000000000001e0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307]
---[ end trace 6a41b2274729ba2a ]---
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029]
audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59
success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2
ppid=4556 pid=2407
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741]
audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s"
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585]
audit: type=1307 audit(1534170895.379:117649235): cwd="/"
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1
kernel oopses to Abrt
Post by Preston Bennes
--
Linux-audit mailing list
https://www.redhat.com/mailman/listinfo/linux-audit
--
paul moore
www.paul-moore.com
Richard Guy Briggs
2018-08-17 21:46:23 UTC
Permalink
Post by Preston Bennes
Hey Paul,
My expectations are aligned to what you've pointed out already (not-vanilla
old kernel, user space client other than auditd).
Primarily what I'm looking for is a sanity check on the null pointer deref
on current vanilla kernel code because I think the rest of the issues stem
from that. My C isn't sharp enough to try my hand at a reproduction case,
though I would hope someone with the requisite expertise could put one
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
Not digging particularly deeply here, this fix just went upstream which
may fix the problem:

https://www.redhat.com/archives/linux-audit/2018-July/msg00037.html

As Paul has already pointed out, the kernel you are using is a pretty
old vendor kernel, so it is difficult to speculate more than that.
Post by Preston Bennes
On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes
Post by Preston Bennes
Hey Audit team,
Hey Audit user!
Post by Preston Bennes
I've got a kernel null pointer deref oops on Amazon Linux kernel
4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that
spawned were stuck in D wait on some audit related syscall. This exhausted
system file handles and the box ended up needing to be rebooted
out-of-bounds. The audit handle was held by
https://github.com/facebook/osquery/.
Post by Preston Bennes
It looks like there was an issue with audit sending to osquery
(netlink_unicast sending to pid 21233, error -111) followed by something
(presumably also osquery) attempting to 'op="add_rule" key=(null)'.
The first thing I need to comment on is that since this is a distro
specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel
there is a limit in the amount of help we can provide. Generally the
upstream list focuses on the upstream code and we leave support for
the distro specific builds/packages up to the distro. If you can
reproduce your problem on an upstream kernel, with a strong preference
for a *recent* upstream kernel, we can probably be of more help. I
simply don't know what patches Amazon merges into it's amzn1 kernels.
With that out of the way, I will add that v4.9.x is a rather old
kernel, and we've made a number of relevant improvements since the
v4.9 release and it is unlikely all of them have been backported to
your kernel. If possible, I would suggest upgrading to a newer kernel
as it may resolve your problem.
There is also a possible issue of osquery as an auditd replacement;
it's unlikely this is being caused by osquery, but we don't test with
osquery to that save level as we do with Steve's audit userspace
(auditd).
Post by Preston Bennes
Here's from /var/log/messages.
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904]
audit: netlink_unicast sending to audit_pid=21233 returned error: -111
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982]
audit_log_lost: 12 callbacks suppressed
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983]
audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985]
audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233
auid=501 ses=102936 res=0
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986]
audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1
auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987]
audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1
old=1 auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989]
audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096
old=4096 auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990]
audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0
auid=501 ses=102936 res=1
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991]
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007]
BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872]
IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336]
PGD 0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230]
Oops: 0000 [#1] SMP
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736]
Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E)
nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E)
isofs(E) ipv6(E) crc_
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036]
CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E
4.9.51-10.52.amzn1.x86_64 #1
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620]
Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513]
task: ffff8807fa1c3b00 task.stack: ffffc900079a4000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179]
RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>]
netlink_unicast+0x4a/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822]
RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065]
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779]
RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473]
RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167]
R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738]
R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999]
FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796]
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555]
CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401]
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725]
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680]
00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540]
ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379]
ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137]
[<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561]
[<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642]
[<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718]
[<ffffffff81110ab2>] audit_receive+0x52/0xa0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713]
[<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943]
[<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236]
[<ffffffff81455818>] sock_sendmsg+0x38/0x50
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219]
[<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142]
[<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853]
[<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464]
[<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189]
[<ffffffff8145673e>] SyS_sendto+0xe/0x10
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952]
[<ffffffff81003854>] do_syscall_64+0x54/0xc0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995]
[<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840]
Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00
81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24
e0 01 00 00 48 89 45
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625]
RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253]
RSP <ffffc900079a7c38>
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013]
CR2: 00000000000001e0
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307]
---[ end trace 6a41b2274729ba2a ]---
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029]
audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59
success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2
ppid=4556 pid=2407
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741]
audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s"
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585]
audit: type=1307 audit(1534170895.379:117649235): cwd="/"
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1
kernel oopses to Abrt
paul moore
- RGB

--
Richard Guy Briggs <***@redhat.com>
Sr. S/W Engineer, Kernel Security, Base Operating Systems
Remote, Ottawa, Red Hat Canada
IRC: rgb, SunRaycer
Voice: +1.647.777.2635, Internal: (81) 32635
Paul Moore
2018-08-17 21:52:40 UTC
Permalink
On Fri, Aug 17, 2018 at 5:24 PM Preston Bennes
Post by Preston Bennes
Hey Paul,
My expectations are aligned to what you've pointed out already (not-vanilla old kernel, user space client other than auditd).
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 op="add_rule" key=(null) list=4 res=0
Unfortunately without digging into Amazon's source for that kernel
build it is difficult to come to any strong conclusion.

The audit message you are seeing above is an AUDIT_CONFIG_CHANGE (type
== 1305), and based on the 'op="add_rule"' field it appears as though
the system is adding an exit (list == 4) filter rule to the kernel.
The operation failed (res == 0), which shouldn't be surprising given
the kernel oops.

I'm not very familiar with osquery, does it load it's own audit filter
rules? Are you loading some of your own?
Post by Preston Bennes
On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes
Post by Preston Bennes
Hey Audit team,
Hey Audit user!
Post by Preston Bennes
I've got a kernel null pointer deref oops on Amazon Linux kernel 4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that spawned were stuck in D wait on some audit related syscall. This exhausted system file handles and the box ended up needing to be rebooted out-of-bounds. The audit handle was held by https://github.com/facebook/osquery/.
It looks like there was an issue with audit sending to osquery (netlink_unicast sending to pid 21233, error -111) followed by something (presumably also osquery) attempting to 'op="add_rule" key=(null)'.
The first thing I need to comment on is that since this is a distro
specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel
there is a limit in the amount of help we can provide. Generally the
upstream list focuses on the upstream code and we leave support for
the distro specific builds/packages up to the distro. If you can
reproduce your problem on an upstream kernel, with a strong preference
for a *recent* upstream kernel, we can probably be of more help. I
simply don't know what patches Amazon merges into it's amzn1 kernels.
With that out of the way, I will add that v4.9.x is a rather old
kernel, and we've made a number of relevant improvements since the
v4.9 release and it is unlikely all of them have been backported to
your kernel. If possible, I would suggest upgrading to a newer kernel
as it may resolve your problem.
There is also a possible issue of osquery as an auditd replacement;
it's unlikely this is being caused by osquery, but we don't test with
osquery to that save level as we do with Steve's audit userspace
(auditd).
Post by Preston Bennes
Here's from /var/log/messages.
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904] audit: netlink_unicast sending to audit_pid=21233 returned error: -111
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982] audit_log_lost: 12 callbacks suppressed
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983] audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985] audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233 auid=501 ses=102936 res=0
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986] audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987] audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1 old=1 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989] audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096 old=4096 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990] audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0 auid=501 ses=102936 res=1
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991] audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936 op="add_rule" key=(null) list=4 res=0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007] BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872] IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336] PGD 0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230] Oops: 0000 [#1] SMP
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736] Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E) nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E) isofs(E) ipv6(E) crc_
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036] CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E 4.9.51-10.52.amzn1.x86_64 #1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513] task: ffff8807fa1c3b00 task.stack: ffffc900079a4000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179] RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822] RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779] RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473] RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167] R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738] R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999] FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555] CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.267680] 00000000079a7c58 00000000000052f1 00000000000052f1 00000000ffffffa1
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.273540] ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8 ffffffff811107d2
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.279379] ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8 ffffffff8145a08c
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.287137] [<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.291561] [<ffffffff811e2f85>] ? __kmalloc_node_track_caller+0x35/0x260
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.296642] [<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.300718] [<ffffffff81110ab2>] audit_receive+0x52/0xa0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.304713] [<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.308943] [<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.313236] [<ffffffff81455818>] sock_sendmsg+0x38/0x50
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.317219] [<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.321142] [<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.325853] [<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.330464] [<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.335189] [<ffffffff8145673e>] SyS_sendto+0xe/0x10
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.338952] [<ffffffff81003854>] do_syscall_64+0x54/0xc0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.342995] [<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840] Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00 81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24 e0 01 00 00 48 89 45
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625] RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.373253] RSP <ffffc900079a7c38>
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013] CR2: 00000000000001e0
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307] ---[ end trace 6a41b2274729ba2a ]---
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029] audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59 success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2 ppid=4556 pid=2407
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741] audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s"
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585] audit: type=1307 audit(1534170895.379:117649235): cwd="/"
Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported 1 kernel oopses to Abrt
--
Linux-audit mailing list
https://www.redhat.com/mailman/listinfo/linux-audit
--
paul moore
www.paul-moore.com
--
paul moore
www.paul-moore.com
Preston Bennes
2018-08-17 23:28:31 UTC
Permalink
osquery loads a fairly minimal set of audit rules depending on its
configuration. In our case we only have a single rule for execve:

-a always,exit -S execve


I'll try finding that patch in the LTS kernel series upon which the Amazon
kernel is based and try upgrading past that. Thanks all.
Post by Paul Moore
On Fri, Aug 17, 2018 at 5:24 PM Preston Bennes
Post by Preston Bennes
Hey Paul,
My expectations are aligned to what you've pointed out already
(not-vanilla old kernel, user space client other than auditd).
Post by Preston Bennes
Primarily what I'm looking for is a sanity check on the null pointer
deref on current vanilla kernel code because I think the rest of the issues
stem from that. My C isn't sharp enough to try my hand at a reproduction
case, though I would hope someone with the requisite expertise could put
Post by Preston Bennes
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
Unfortunately without digging into Amazon's source for that kernel
build it is difficult to come to any strong conclusion.
The audit message you are seeing above is an AUDIT_CONFIG_CHANGE (type
== 1305), and based on the 'op="add_rule"' field it appears as though
the system is adding an exit (list == 4) filter rule to the kernel.
The operation failed (res == 0), which shouldn't be surprising given
the kernel oops.
I'm not very familiar with osquery, does it load it's own audit filter
rules? Are you loading some of your own?
Post by Preston Bennes
On Fri, Aug 17, 2018 at 4:26 PM Preston Bennes
Post by Preston Bennes
Hey Audit team,
Hey Audit user!
Post by Preston Bennes
I've got a kernel null pointer deref oops on Amazon Linux kernel
4.9.51-10.52.amzn1.x86_64. After the oops all new cron processes that
spawned were stuck in D wait on some audit related syscall. This exhausted
system file handles and the box ended up needing to be rebooted
out-of-bounds. The audit handle was held by
https://github.com/facebook/osquery/.
Post by Preston Bennes
Post by Preston Bennes
It looks like there was an issue with audit sending to osquery
(netlink_unicast sending to pid 21233, error -111) followed by something
(presumably also osquery) attempting to 'op="add_rule" key=(null)'.
Post by Preston Bennes
The first thing I need to comment on is that since this is a distro
specific kernel (4.9.51-10.52.amzn1.x86_64) and not an upstream kernel
there is a limit in the amount of help we can provide. Generally the
upstream list focuses on the upstream code and we leave support for
the distro specific builds/packages up to the distro. If you can
reproduce your problem on an upstream kernel, with a strong preference
for a *recent* upstream kernel, we can probably be of more help. I
simply don't know what patches Amazon merges into it's amzn1 kernels.
With that out of the way, I will add that v4.9.x is a rather old
kernel, and we've made a number of relevant improvements since the
v4.9 release and it is unlikely all of them have been backported to
your kernel. If possible, I would suggest upgrading to a newer kernel
as it may resolve your problem.
There is also a possible issue of osquery as an auditd replacement;
it's unlikely this is being caused by osquery, but we don't test with
osquery to that save level as we do with Steve's audit userspace
(auditd).
Post by Preston Bennes
Here's from /var/log/messages.
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.133904]
audit: netlink_unicast sending to audit_pid=21233 returned error: -111
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137982]
audit_log_lost: 12 callbacks suppressed
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137983]
audit: audit_lost=36081041 audit_rate_limit=8192 audit_backlog_limit=4096
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137985]
audit: type=1305 audit(1534170894.852:117649229): audit_pid=21233 old=21233
auid=501 ses=102936 res=0
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137986]
audit: type=1305 audit(1534170894.852:117649230): audit_enabled=1 old=1
auid=501 ses=102936 res=1
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137987]
audit: type=1305 audit(1534170894.852:117649231): audit_backlog_wait_time=1
old=1 auid=501 ses=102936 res=1
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137989]
audit: type=1305 audit(1534170894.852:117649232): audit_backlog_limit=4096
old=4096 auid=501 ses=102936 res=1
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137990]
audit: type=1305 audit(1534170894.852:117649233): audit_failure=0 old=0
auid=501 ses=102936 res=1
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:54 packer_default-10-180-21-24 kernel: [4749520.137991]
audit: type=1305 audit(1534170894.852:117649234): auid=501 ses=102936
op="add_rule" key=(null) list=4 res=0
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.138007]
BUG: unable to handle kernel NULL pointer dereference at 00000000000001e0
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.143872]
IP: [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.148336]
PGD 0
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.149822]
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.151230]
Oops: 0000 [#1] SMP
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.153736]
Modules linked in: iptable_filter(E) ip_tables(E) x_tables(E) udp_diag(E)
nfnetlink_queue(E) nfnetlink_log(E) nfnetlink(E) tcp_diag(E) inet_diag(E)
isofs(E) ipv6(E) crc_
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.184036]
CPU: 0 PID: 21372 Comm: osqueryd Tainted: G E
4.9.51-10.52.amzn1.x86_64 #1
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.190620]
Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.195513]
task: ffff8807fa1c3b00 task.stack: ffffc900079a4000
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.200179]
RIP: 0010:[<ffffffff814a76da>] [<ffffffff814a76da>]
netlink_unicast+0x4a/0x230
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.206822]
RSP: 0018:ffffc900079a7c38 EFLAGS: 00010246
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.211065]
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.216779]
RDX: 00001ffffffffffe RSI: 00000000024000c0 RDI: 0000000000000014
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.222473]
RBP: ffffc900079a7c68 R08: 0000000000000004 R09: ffff88062a375414
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.228167]
R10: ffff8808004032c0 R11: ffff88062a375400 R12: 0000000000000000
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.233738]
R13: ffff8804200a0100 R14: 00000000000052f1 R15: 0000000000016d42
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.238999]
FS: 00007f3d40cc4700(0000) GS:ffff880800a00000(0000) knlGS:0000000000000000
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.244796]
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.249555]
CR2: 00000000000001e0 CR3: 000000059e63d000 CR4: 00000000001406f0
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.255401]
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.260725]
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.266038]
[4749520.267680] 00000000079a7c58 00000000000052f1 00000000000052f1
00000000ffffffa1
[4749520.273540] ffff88062a376800 ffff8804200a1d00 ffffc900079a7cf8
ffffffff811107d2
[4749520.279379] ffffc90000000004 ffffffff811e2f85 ffffc900079a7ca8
ffffffff8145a08c
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.285201]
[4749520.287137] [<ffffffff811107d2>] audit_receive_msg+0x992/0xc20
[4749520.291561] [<ffffffff811e2f85>] ?
__kmalloc_node_track_caller+0x35/0x260
[4749520.296642] [<ffffffff8145a08c>] ? release_sock+0x8c/0xa0
[4749520.300718] [<ffffffff81110ab2>] audit_receive+0x52/0xa0
[4749520.304713] [<ffffffff814a77ef>] netlink_unicast+0x15f/0x230
[4749520.308943] [<ffffffff814a7bdc>] netlink_sendmsg+0x31c/0x390
[4749520.313236] [<ffffffff81455818>] sock_sendmsg+0x38/0x50
[4749520.317219] [<ffffffff81455c4f>] SYSC_sendto+0xef/0x170
[4749520.321142] [<ffffffff811156ab>] ? __audit_syscall_entry+0xeb/0x100
[4749520.325853] [<ffffffff81003437>] ? syscall_trace_enter+0x1b7/0x290
[4749520.330464] [<ffffffff811158b3>] ? __audit_syscall_exit+0x1f3/0x280
[4749520.335189] [<ffffffff8145673e>] SyS_sendto+0xe/0x10
[4749520.338952] [<ffffffff81003854>] do_syscall_64+0x54/0xc0
[4749520.342995] [<ffffffff8153d32b>] entry_SYSCALL64_slow_path+0x25/0x25
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.347840]
Code: 65 8b 05 a2 5b b6 7e 25 00 ff 00 00 83 f8 01 19 f6 81 e6 a0 00 38 00
81 c6 20 00 08 02 e8 af cf ff ff 49 89 c5 31 c0 85 db 75 08 <49> 8b 84 24
e0 01 00 00 48 89 45
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.368625]
RIP [<ffffffff814a76da>] netlink_unicast+0x4a/0x230
[4749520.373253] RSP <ffffc900079a7c38>
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.376013]
CR2: 00000000000001e0
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.379307]
---[ end trace 6a41b2274729ba2a ]---
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.663029]
audit: type=1300 audit(1534170895.379:117649235): arch=c000003e syscall=59
success=yes exit=0 a0=271cce0 a1=2741dd0 a2=26dc3b0 a3=7ffdd7f1ea80 items=2
ppid=4556 pid=2407
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.682741]
audit: type=1309 audit(1534170895.379:117649235): argc=2 a0="date" a1="+%s"
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 kernel: [4749520.688585]
audit: type=1307 audit(1534170895.379:117649235): cwd="/"
Post by Preston Bennes
Post by Preston Bennes
Aug 13 14:34:55 packer_default-10-180-21-24 abrt-dump-oops: Reported
1 kernel oopses to Abrt
Post by Preston Bennes
Post by Preston Bennes
--
Linux-audit mailing list
https://www.redhat.com/mailman/listinfo/linux-audit
--
paul moore
www.paul-moore.com
--
paul moore
www.paul-moore.com
Loading...