Topics

Incorrect "invalid indirect read" error from verifier?

Paul Chaignon
 

Hi,

I'm getting an "invalid indirect read from stack" when trying to run the
following bcc script:

paul$ cat tmp.py
from bcc import BPF
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

BPF_HASH(currsock, u32, struct sock **);

int trace_entry(struct pt_regs *ctx, struct sock *sk) {
u32 zero = 0;
struct sock **skp = &sk;
currsock.update(&zero, &skp);
return 0;
};
"""
b = BPF(text=bpf_text)
b.load_func("trace_entry", BPF.KPROBE)

paul$ sudo python tmp.py
bpf: Failed to load program: Permission denied
0: (79) r1 = *(u64 *)(r1 +112)
1: (7b) *(u64 *)(r10 -8) = r1
2: (b7) r1 = 0
3: (63) *(u32 *)(r10 -12) = r1
4: (bf) r1 = r10
5: (07) r1 += -8
6: (7b) *(u64 *)(r10 -24) = r1
7: (18) r1 = 0xffff8801ed41b300
9: (bf) r2 = r10
10: (07) r2 += -12
11: (bf) r3 = r10
12: (07) r3 += -24
13: (b7) r4 = 0
14: (85) call 2
invalid indirect read from stack off -24+0 size 8

As far as I can see, that program is valid; the stack offset r10-24 was
initialized on line 6 with the value read from context on line 1. Is this
an error on the verifier's side or am I missing something?

I found the issue on a 4.4 kernel and can reproduce with a 4.16.

Thanks,

Paul

Alexei Starovoitov
 

On Wed, May 9, 2018 at 4:48 AM, Paul Chaignon via iovisor-dev
<iovisor-dev@...> wrote:
Hi,

I'm getting an "invalid indirect read from stack" when trying to run the
following bcc script:

paul$ cat tmp.py
from bcc import BPF
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

BPF_HASH(currsock, u32, struct sock **);

int trace_entry(struct pt_regs *ctx, struct sock *sk) {
u32 zero = 0;
struct sock **skp = &sk;
currsock.update(&zero, &skp);
return 0;
};
"""
b = BPF(text=bpf_text)
b.load_func("trace_entry", BPF.KPROBE)

paul$ sudo python tmp.py
bpf: Failed to load program: Permission denied
0: (79) r1 = *(u64 *)(r1 +112)
1: (7b) *(u64 *)(r10 -8) = r1
2: (b7) r1 = 0
3: (63) *(u32 *)(r10 -12) = r1
4: (bf) r1 = r10
5: (07) r1 += -8
6: (7b) *(u64 *)(r10 -24) = r1
7: (18) r1 = 0xffff8801ed41b300
9: (bf) r2 = r10
10: (07) r2 += -12
11: (bf) r3 = r10
12: (07) r3 += -24
13: (b7) r4 = 0
14: (85) call 2
invalid indirect read from stack off -24+0 size 8

As far as I can see, that program is valid; the stack offset r10-24 was
initialized on line 6 with the value read from context on line 1. Is this
an error on the verifier's side or am I missing something?
verifier complains that fp-24 was initialized with
pointer to stack.
I don't think you really meant to take local stack pointer
and store it into the map:
struct sock **skp = &sk;

Paul Chaignon <paul.chaignon@...>
 

On Wed, May 09, 2018 at 07:26:17AM -0700, Alexei Starovoitov wrote:
On Wed, May 9, 2018 at 4:48 AM, Paul Chaignon via iovisor-dev
<iovisor-dev@...> wrote:
Hi,

I'm getting an "invalid indirect read from stack" when trying to run the
following bcc script:

paul$ cat tmp.py
from bcc import BPF
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

BPF_HASH(currsock, u32, struct sock **);

int trace_entry(struct pt_regs *ctx, struct sock *sk) {
u32 zero = 0;
struct sock **skp = &sk;
currsock.update(&zero, &skp);
return 0;
};
"""
b = BPF(text=bpf_text)
b.load_func("trace_entry", BPF.KPROBE)

paul$ sudo python tmp.py
bpf: Failed to load program: Permission denied
0: (79) r1 = *(u64 *)(r1 +112)
1: (7b) *(u64 *)(r10 -8) = r1
2: (b7) r1 = 0
3: (63) *(u32 *)(r10 -12) = r1
4: (bf) r1 = r10
5: (07) r1 += -8
6: (7b) *(u64 *)(r10 -24) = r1
7: (18) r1 = 0xffff8801ed41b300
9: (bf) r2 = r10
10: (07) r2 += -12
11: (bf) r3 = r10
12: (07) r3 += -24
13: (b7) r4 = 0
14: (85) call 2
invalid indirect read from stack off -24+0 size 8

As far as I can see, that program is valid; the stack offset r10-24 was
initialized on line 6 with the value read from context on line 1. Is this
an error on the verifier's side or am I missing something?
verifier complains that fp-24 was initialized with
pointer to stack.
I see. I had no idea kprobe BPF programs couldn't store stack pointers in
maps.

I don't think you really meant to take local stack pointer
and store it into the map:
struct sock **skp = &sk;
Actually, I did. This is just a dummy example to test a fix for the bcc
rewriter I'm working on. But if BPF programs can't save stack pointers to
maps, that will actually ease my work :)

Paul Chaignon
 


On Wed, May 9, 2018 at 4:38 PM, Paul Chaignon <paul.chaignon@...> wrote:
On Wed, May 09, 2018 at 07:26:17AM -0700, Alexei Starovoitov wrote:
> On Wed, May 9, 2018 at 4:48 AM, Paul Chaignon via iovisor-dev
> <iovisor-dev@...> wrote:
> > Hi,
> >
> > I'm getting an "invalid indirect read from stack" when trying to run the
> > following bcc script:
> >
> > paul$ cat tmp.py
> > from bcc import BPF
> > bpf_text = """
> > #include <uapi/linux/ptrace.h>
> > #include <net/sock.h>
> > #include <bcc/proto.h>
> >
> > BPF_HASH(currsock, u32, struct sock **);
> >
> > int trace_entry(struct pt_regs *ctx, struct sock *sk) {
> >     u32 zero = 0;
> >     struct sock **skp = &sk;
> >     currsock.update(&zero, &skp);
> >     return 0;
> > };
> > """
> > b = BPF(text=bpf_text)
> > b.load_func("trace_entry", BPF.KPROBE)
> >
> > paul$ sudo python tmp.py
> > bpf: Failed to load program: Permission denied
> > 0: (79) r1 = *(u64 *)(r1 +112)
> > 1: (7b) *(u64 *)(r10 -8) = r1
> > 2: (b7) r1 = 0
> > 3: (63) *(u32 *)(r10 -12) = r1
> > 4: (bf) r1 = r10
> > 5: (07) r1 += -8
> > 6: (7b) *(u64 *)(r10 -24) = r1
> > 7: (18) r1 = 0xffff8801ed41b300
> > 9: (bf) r2 = r10
> > 10: (07) r2 += -12
> > 11: (bf) r3 = r10
> > 12: (07) r3 += -24
> > 13: (b7) r4 = 0
> > 14: (85) call 2
> > invalid indirect read from stack off -24+0 size 8
> >
> > As far as I can see, that program is valid; the stack offset r10-24 was
> > initialized on line 6 with the value read from context on line 1. Is this
> > an error on the verifier's side or am I missing something?
>
> verifier complains that fp-24 was initialized with
> pointer to stack.

I see.  I had no idea kprobe BPF programs couldn't store stack pointers in
maps.

Never mind.  I see why that's not possible.  Not sure what I was thinking.
 


> I don't think you really meant to take local stack pointer
> and store it into the map:
> struct sock **skp = &sk;

Actually, I did.  This is just a dummy example to test a fix for the bcc
rewriter I'm working on.  But if BPF programs can't save stack pointers to
maps, that will actually ease my work :)

Alexei Starovoitov
 

On Wed, May 9, 2018 at 7:43 AM, Paul Chaignon <paul.chaignon@...> wrote:

On Wed, May 9, 2018 at 4:38 PM, Paul Chaignon <paul.chaignon@...>
wrote:

On Wed, May 09, 2018 at 07:26:17AM -0700, Alexei Starovoitov wrote:
On Wed, May 9, 2018 at 4:48 AM, Paul Chaignon via iovisor-dev
<iovisor-dev@...> wrote:
Hi,

I'm getting an "invalid indirect read from stack" when trying to run
the
following bcc script:

paul$ cat tmp.py
from bcc import BPF
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

BPF_HASH(currsock, u32, struct sock **);

int trace_entry(struct pt_regs *ctx, struct sock *sk) {
u32 zero = 0;
struct sock **skp = &sk;
currsock.update(&zero, &skp);
return 0;
};
"""
b = BPF(text=bpf_text)
b.load_func("trace_entry", BPF.KPROBE)

paul$ sudo python tmp.py
bpf: Failed to load program: Permission denied
0: (79) r1 = *(u64 *)(r1 +112)
1: (7b) *(u64 *)(r10 -8) = r1
2: (b7) r1 = 0
3: (63) *(u32 *)(r10 -12) = r1
4: (bf) r1 = r10
5: (07) r1 += -8
6: (7b) *(u64 *)(r10 -24) = r1
7: (18) r1 = 0xffff8801ed41b300
9: (bf) r2 = r10
10: (07) r2 += -12
11: (bf) r3 = r10
12: (07) r3 += -24
13: (b7) r4 = 0
14: (85) call 2
invalid indirect read from stack off -24+0 size 8

As far as I can see, that program is valid; the stack offset r10-24
was
initialized on line 6 with the value read from context on line 1. Is
this
an error on the verifier's side or am I missing something?
verifier complains that fp-24 was initialized with
pointer to stack.
I see. I had no idea kprobe BPF programs couldn't store stack pointers in
maps.

Never mind. I see why that's not possible. Not sure what I was thinking.
the programs can write pointers to stack and
can fetch them back. That's normal spill/fill that
compilers do all the time.
That part is fine.
The verifier disallows to write a pointer to stack
and pass that whole area into helpers, since
it's either a pointer leak or more likely a bug in the program.
And by 'pointer' I mean a pointer to something that
verifier knows about. Like pointer to map, pointer to ctx.
If it's a kernel pointer that the program got from bpf_probe_read,
it's seen as 8-byte unknown value to the verifier and can be
passed in stack to helpers.