iovisor-dev@lists.iovisor.org | #bcc - skb_network_header crashes in a BPF Kernel trace function

Home Messages Hashtags Subgroups

This group is locked. No changes can be made to the group while it is locked.

Date Date 1 - 3 of 3

#bcc - skb_network_header crashes in a BPF Kernel trace function

#bcc

vigs.prof@... #1842 Hello - I am looking to trace ip_forward_finish. The intent is to trace latency of all TCP connections going through a linux based gateway router. Hence thought of tracing ip_forward_finish kernel function. And capture the time-stamp of SYN, SYN-ACK and ACK messages at the router. The issue is accessing iphdr inside the trace function crashes with the below error: bpf: Failed to load program: Permission denied 0: (79) r6 = (u64 )(r1 +96) 1: (b7) r1 = 0 2: (6b) (u16 )(r10 -24) = r1 3: (bf) r3 = r6 4: (07) r3 += 192 5: (bf) r1 = r10 6: (07) r1 += -24 7: (b7) r2 = 2 8: (85) call bpf_probe_read#4 9: (69) r1 = (u16 )(r10 -24) 10: (55) if r1 != 0x8 goto pc+7 R0=inv(id=0) R1=inv8 R6=inv(id=0) R10=fp0 11: (69) r1 = (u16 )(r6 +196) R6 invalid mem access 'inv' HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read() to copy it to the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times you'll need to be explicit. The code fragment I originally had was as below and the crash occurs when an access to ip_Hdr->protocol is made. And I also checked that ip_Hdr is not null. int trace_forward_finish(struct pt_regs ctx,struct net net, struct sock sk, struct sk_buff skb) { if (skb->protocol != htons(ETH_P_IP)) return 0; struct iphdr* ip_Hdr = (struct iphdr ) skb_network_header(skb); if (ip_Hdr->protocol != IPPROTO_TCP) return 0; /// Other code } Per the HINT in the message, I did try to change to bpf_probe_read but still the same outcome int trace_forward_finish(struct pt_regs ctx,struct net net, struct sock sk, struct sk_buff skb) { if (skb->protocol != htons(ETH_P_IP)) return 0; struct iphdr ip_Hdr; bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void)ip_hdr(skb)); if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0; return 0; } Any help would be appreciated.
Yonghong Song #1843 On Wed, May 6, 2020 at 9:26 AM <vigs.prof@...> wrote: Hello - I am looking to trace ip_forward_finish. The intent is to trace latency of all TCP connections going through a linux based gateway router. Hence thought of tracing ip_forward_finish kernel function. And capture the time-stamp of SYN, SYN-ACK and ACK messages at the router. The issue is accessing iphdr inside the trace function crashes with the below error: bpf: Failed to load program: Permission denied 0: (79) r6 = (u64 )(r1 +96) 1: (b7) r1 = 0 2: (6b) (u16 )(r10 -24) = r1 3: (bf) r3 = r6 4: (07) r3 += 192 5: (bf) r1 = r10 6: (07) r1 += -24 7: (b7) r2 = 2 8: (85) call bpf_probe_read#4 9: (69) r1 = (u16 )(r10 -24) 10: (55) if r1 != 0x8 goto pc+7 R0=inv(id=0) R1=inv8 R6=inv(id=0) R10=fp0 11: (69) r1 = (u16 )(r6 +196) R6 invalid mem access 'inv' You did not show the code which actually caused the problem. bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void)ip_hdr(skb)); if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0; return 0; } There must be code after "if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0;" . You may need bpf_probe_read() for memory accesses there. HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read() to copy it to the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times you'll need to be explicit. The code fragment I originally had was as below and the crash occurs when an access to ip_Hdr->protocol is made. And I also checked that ip_Hdr is not null. int trace_forward_finish(struct pt_regs ctx,struct net net, struct sock sk, struct sk_buff skb) { if (skb->protocol != htons(ETH_P_IP)) return 0; struct iphdr ip_Hdr = (struct iphdr ) skb_network_header(skb); if (ip_Hdr->protocol != IPPROTO_TCP) return 0; /// Other code } Per the HINT in the message, I did try to change to bpf_probe_read but still the same outcome int trace_forward_finish(struct pt_regs ctx,struct net net, struct sock sk, struct sk_buff skb) { if (skb->protocol != htons(ETH_P_IP)) return 0; struct iphdr ip_Hdr; bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void)ip_hdr(skb)); if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0; return 0; } Any help would be appreciated.
Yonghong Song #1844 On Wed, May 6, 2020 at 11:00 PM Yonghong Song via lists.iovisor.org <ys114321=gmail.com@...> wrote: On Wed, May 6, 2020 at 9:26 AM <vigs.prof@...> wrote: Hello - I am looking to trace ip_forward_finish. The intent is to trace latency of all TCP connections going through a linux based gateway router. Hence thought of tracing ip_forward_finish kernel function. And capture the time-stamp of SYN, SYN-ACK and ACK messages at the router. The issue is accessing iphdr inside the trace function crashes with the below error: bpf: Failed to load program: Permission denied 0: (79) r6 = (u64 )(r1 +96) 1: (b7) r1 = 0 2: (6b) (u16 )(r10 -24) = r1 3: (bf) r3 = r6 4: (07) r3 += 192 5: (bf) r1 = r10 6: (07) r1 += -24 7: (b7) r2 = 2 8: (85) call bpf_probe_read#4 9: (69) r1 = (u16 )(r10 -24) 10: (55) if r1 != 0x8 goto pc+7 R0=inv(id=0) R1=inv8 R6=inv(id=0) R10=fp0 11: (69) r1 = (u16 )(r6 +196) R6 invalid mem access 'inv' You did not show the code which actually caused the problem. bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void)ip_hdr(skb)); if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0; return 0; } There must be code after "if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0;" . You may need bpf_probe_read() for memory accesses there. HINT: The invalid mem access 'inv' error can happen if you try to dereference memory without first using bpf_probe_read() to copy it to the BPF stack. Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times you'll need to be explicit. The code fragment I originally had was as below and the crash occurs when an access to ip_Hdr->protocol is made. And I also checked that ip_Hdr is not null. int trace_forward_finish(struct pt_regs ctx,struct net net, struct sock sk, struct sk_buff skb) { if (skb->protocol != htons(ETH_P_IP)) return 0; struct iphdr ip_Hdr = (struct iphdr ) skb_network_header(skb); if (ip_Hdr->protocol != IPPROTO_TCP) return 0; /// Other code } Per the HINT in the message, I did try to change to bpf_probe_read but still the same outcome int trace_forward_finish(struct pt_regs ctx,struct net net, struct sock sk, struct sk_buff skb) { if (skb->protocol != htons(ETH_P_IP)) return 0; struct iphdr ip_Hdr; bpf_probe_read(&ip_Hdr, sizeof(ip_Hdr), (void)ip_hdr(skb)); I see the issue now. ip_hdr(skb) eventually transforms to static inline unsigned char skb_network_header(const struct sk_buff skb) { return skb->head + skb->network_header; } The above two pointer dereferences need bpf probe read. Unfortunately, you may need put the above function in your bpf program so you could use bpf_probe_read to access skb->head and skb->network_header. if ( (ip_Hdr.protocol != IPPROTO_TCP)) return 0; return 0; } Any help would be appreciated.

1 - 3 of 3

1

Previous Topic Next Topic

Home Hashtags Subgroups Terms