Topics

#bcc - skb_network_header crashes in a BPF Kernel trace function #bcc

vigs.prof@...
 

Hello - I am looking to trace ip_forward_finish. The intent is to trace latency of all TCP connections going through a linux based gateway router.  Hence thought of tracing ip_forward_finish kernel function. And capture the time-stamp of SYN, SYN-ACK and ACK messages at the router. 
 
The issue is accessing iphdr inside the trace function crashes with the below error:
 
bpf: Failed to load program: Permission denied
0: (79) r6 = *(u64 *)(r1 +96)
1: (b7) r1 = 0
2: (6b) *(u16 *)(r10 -24) = r1
3: (bf) r3 = r6
4: (07) r3 += 192
5: (bf) r1 = r10
6: (07) r1 += -24
7: (b7) r2 = 2
8: (85) call bpf_probe_read#4
9: (69) r1 = *(u16 *)(r10 -24)
10: (55) if r1 != 0x8 goto pc+7
 R0=inv(id=0) R1=inv8 R6=inv(id=0) R10=fp0
11: (69) r1 = *(u16 *)(r6 +196)
R6 invalid mem access 'inv'
 
HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read() to copy it to the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times you'll need to be explicit.
 
The code fragment I originally had was as below and the crash occurs when an access to ip_Hdr->protocol is made. And I also checked that ip_Hdr is not null. 
 
int trace_forward_finish(struct pt_regs *ctx,struct net *net, struct sock *sk, struct sk_buff *skb)
{
 
    if (skb->protocol != htons(ETH_P_IP)) return 0;
 
    struct iphdr* ip_Hdr = (struct iphdr *) skb_network_header(skb);
 
    if (ip_Hdr->protocol != IPPROTO_TCP)
         return 0;
 
 
    /// Other code
 
  }
 
Per the HINT in the message, I did try to change to bpf_probe_read but still the same outcome
 
int trace_forward_finish(struct pt_regs *ctx,struct net *net, struct sock *sk, struct sk_buff *skb)
{
    if (skb->protocol != htons(ETH_P_IP)) return 0;
 
    struct iphdr ip_Hdr;
    bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void*)ip_hdr(skb)); 
    
    if ( (ip_Hdr.protocol != IPPROTO_TCP))
         return 0;
 
    return 0;
}
 
Any help would be appreciated. 

Yonghong Song
 

On Wed, May 6, 2020 at 9:26 AM <vigs.prof@...> wrote:

Hello - I am looking to trace ip_forward_finish. The intent is to trace latency of all TCP connections going through a linux based gateway router. Hence thought of tracing ip_forward_finish kernel function. And capture the time-stamp of SYN, SYN-ACK and ACK messages at the router.

The issue is accessing iphdr inside the trace function crashes with the below error:

bpf: Failed to load program: Permission denied
0: (79) r6 = *(u64 *)(r1 +96)
1: (b7) r1 = 0
2: (6b) *(u16 *)(r10 -24) = r1
3: (bf) r3 = r6
4: (07) r3 += 192
5: (bf) r1 = r10
6: (07) r1 += -24
7: (b7) r2 = 2
8: (85) call bpf_probe_read#4
9: (69) r1 = *(u16 *)(r10 -24)
10: (55) if r1 != 0x8 goto pc+7
R0=inv(id=0) R1=inv8 R6=inv(id=0) R10=fp0
11: (69) r1 = *(u16 *)(r6 +196)
R6 invalid mem access 'inv'
You did not show the code which actually caused the problem.

bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void*)ip_hdr(skb));

if ( (ip_Hdr.protocol != IPPROTO_TCP))
return 0;

return 0;
}
There must be code after "if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0;" .
You may need bpf_probe_read() for memory accesses there.


HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read() to copy it to the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times you'll need to be explicit.

The code fragment I originally had was as below and the crash occurs when an access to ip_Hdr->protocol is made. And I also checked that ip_Hdr is not null.

int trace_forward_finish(struct pt_regs *ctx,struct net *net, struct sock *sk, struct sk_buff *skb)
{

if (skb->protocol != htons(ETH_P_IP)) return 0;

struct iphdr* ip_Hdr = (struct iphdr *) skb_network_header(skb);

if (ip_Hdr->protocol != IPPROTO_TCP)
return 0;


/// Other code

}

Per the HINT in the message, I did try to change to bpf_probe_read but still the same outcome

int trace_forward_finish(struct pt_regs *ctx,struct net *net, struct sock *sk, struct sk_buff *skb)
{
if (skb->protocol != htons(ETH_P_IP)) return 0;

struct iphdr ip_Hdr;
bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void*)ip_hdr(skb));

if ( (ip_Hdr.protocol != IPPROTO_TCP))
return 0;

return 0;
}

Any help would be appreciated.

Yonghong Song
 

On Wed, May 6, 2020 at 11:00 PM Yonghong Song via lists.iovisor.org
<ys114321=gmail.com@...> wrote:

On Wed, May 6, 2020 at 9:26 AM <vigs.prof@...> wrote:

Hello - I am looking to trace ip_forward_finish. The intent is to trace latency of all TCP connections going through a linux based gateway router. Hence thought of tracing ip_forward_finish kernel function. And capture the time-stamp of SYN, SYN-ACK and ACK messages at the router.

The issue is accessing iphdr inside the trace function crashes with the below error:

bpf: Failed to load program: Permission denied
0: (79) r6 = *(u64 *)(r1 +96)
1: (b7) r1 = 0
2: (6b) *(u16 *)(r10 -24) = r1
3: (bf) r3 = r6
4: (07) r3 += 192
5: (bf) r1 = r10
6: (07) r1 += -24
7: (b7) r2 = 2
8: (85) call bpf_probe_read#4
9: (69) r1 = *(u16 *)(r10 -24)
10: (55) if r1 != 0x8 goto pc+7
R0=inv(id=0) R1=inv8 R6=inv(id=0) R10=fp0
11: (69) r1 = *(u16 *)(r6 +196)
R6 invalid mem access 'inv'
You did not show the code which actually caused the problem.

bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void*)ip_hdr(skb));

if ( (ip_Hdr.protocol != IPPROTO_TCP))
return 0;

return 0;
}
There must be code after "if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0;" .
You may need bpf_probe_read() for memory accesses there.


HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read() to copy it to the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times you'll need to be explicit.

The code fragment I originally had was as below and the crash occurs when an access to ip_Hdr->protocol is made. And I also checked that ip_Hdr is not null.

int trace_forward_finish(struct pt_regs *ctx,struct net *net, struct sock *sk, struct sk_buff *skb)
{

if (skb->protocol != htons(ETH_P_IP)) return 0;

struct iphdr* ip_Hdr = (struct iphdr *) skb_network_header(skb);

if (ip_Hdr->protocol != IPPROTO_TCP)
return 0;


/// Other code

}

Per the HINT in the message, I did try to change to bpf_probe_read but still the same outcome

int trace_forward_finish(struct pt_regs *ctx,struct net *net, struct sock *sk, struct sk_buff *skb)
{
if (skb->protocol != htons(ETH_P_IP)) return 0;

struct iphdr ip_Hdr;
bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void*)ip_hdr(skb));
I see the issue now. ip_hdr(skb) eventually transforms to

static inline unsigned char *skb_network_header(const struct sk_buff *skb)
{
return skb->head + skb->network_header;
}

The above two pointer dereferences need bpf probe read.
Unfortunately, you may need put the above function in your bpf program
so you could use bpf_probe_read to access skb->head and skb->network_header.


if ( (ip_Hdr.protocol != IPPROTO_TCP))
return 0;

return 0;
}

Any help would be appreciated.