tc filter show tcp_flags wrong mask value

Bug #1873961 reported by bugproxy
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Fix Released
Medium
Ubuntu on IBM Power Systems Bug Triage
iproute2 (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Medium
Stefan Bader

Bug Description

[SRU Justification]

Impact: The tc command does not show the correct values for tcp_flags (and ip_tos) on filter rules. This might break other scripts parsing that output but at least confuses users.

Fix: Backport of "tc: fix bugs for tcp_flags and ip_attr hex output" from upstream iproute2.

Testcase:
tc qdisc add dev lo ingress
tc filter add dev lo parent ffff: prio 3 proto ip flower ip_tos 0x8/32
tc filter add dev lo parent ffff: prio 5 proto ip flower ip_proto tcp \
     tcp_flags 0x909/f00

tc filter show dev lo parent ffff:

filter protocol ip pref 3 flower chain 0
filter protocol ip pref 3 flower chain 0 handle 0x1
  eth_type ipv4
  ip_tos a9606c10 <-- bad, should be 0x8/32
  not_in_hw
filter protocol ip pref 5 flower chain 0
filter protocol ip pref 5 flower chain 0 handle 0x1
  eth_type ipv4
  ip_proto tcp
  tcp_flags 909909 <-- bad, should be 0x909/f00
  not_in_hw

Note that the ip_tos value in the -j[son] output is correct, while the tcp_flags value is
is incorrect in both cases.

Risk of Regression:
Low: Usually scripts would use the json output and that has at least the ip output correct. And the values shown in the bad case seem to be little useful. So it seems unlikely anybody relied on them. But cannot completely be ruled out.

=== Original description ===

---Problem Description---
Problem Descriptions
"tc" utility does not show correct TC rule's tcp_flags mask correctly in current "iproute2" package shipped on Genesis.
# dpkg -l |grep iproute2
ii iproute2 4.15.0-2ubuntu1 ppc64el networking and traffic control tools

---Steps to Reproduce---
 Steps to reproduce the problem:
1) Add a tc rule to the testing VF (i.e. p0v2_r):
# tc filter add dev p0v2 protocol ip parent ffff: pref 5 chain 1 handle 0x1 flower src_mac 00:00:00:00:4e:2f/00:00:00:ff:ff:ff ip_proto tcp tcp_flags 2 skip_sw action mirred egress redirect dev p0v0_r

2) Validate the added TC rule:
# tc filter show dev p0v2_r root
filter protocol ip pref 5 flower chain 1
filter protocol ip pref 5 flower chain 1 handle 0x1
  src_mac 00:00:00:00:4e:2f/00:00:00:ff:ff:ff
  eth_type ipv4
  ip_proto tcp
  tcp_flags 22 /* <--- Wrong */
  skip_sw
  in_hw
        action order 1: mirred (Egress Redirect to device p0v0_r) stolen

3) If we add the tcp_flags using explicit mask 0x7:
# tc filter add dev p0v2 protocol ip parent ffff: pref 5 chain 1 handle 0x1 flower src_mac 00:00:00:00:4e:2f/00:00:00:ff:ff:ff ip_proto tcp tcp_flags 0x2/7 skip_sw action mirred egress redirect dev p0v0_r

After that, using "tc filter show dev p0v2_r root" to verify, we still see the same output (tcp_flags 22) as shown in 2) above, which is wrong.

Userspace tool common name: tc

The userspace tool has the following bit modes: 64-bit

Userspace package: iproute2

==
Fixes:
There are 2 patches to fix the issue:
patch 1:
commit b85076cd74e77538918d35992b1a9cd17ff86af8
Author: Stephen Hemminger <email address hidden>
Date: Tue Sep 11 08:29:33 2018 -0700
    lib: introduce print_nl
    Common pattern in iproute commands is to print a line seperator
    in non-json mode. Make that a simple function.
/* This patch declares global variable "const char *_SL_ = "\n";" in lib/utils.c to be used by 2nd patch */

patch 2:
commit e8bd395508cead5a81c2bebd9d3705a9e41ea8bc
Author: Keara Leibovitz <email address hidden>
Date: Thu Jul 26 09:45:30 2018 -0400
    tc: fix bugs for tcp_flags and ip_attr hex output
    Fix hex output for both the ip_attr and tcp_flags print functions.

With the above 2 patches pull in, the new "tc" utility will show the correct tcp_flags mask:
# tc filter show dev p0v2 root
filter protocol ip pref 5 flower chain 1
filter protocol ip pref 5 flower chain 1 handle 0x1
  src_mac 00:00:00:00:4e:2f/00:00:00:ff:ff:ff
  eth_type ipv4
  ip_proto tcp
  tcp_flags 0x2/7 /* <--- Correct */
  skip_sw
  in_hw
        action order 1: mirred (Egress Redirect to device p0v0_r) stolen

====
This bug affects tc in Ubuntu 18.04.1 stock image.

Related branches

bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-185386 severity-medium targetmilestone-inin18041
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → iproute2 (Ubuntu)
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
importance: Undecided → Medium
assignee: nobody → Canonical Server Team (canonical-server)
Frank Heimes (fheimes)
Changed in iproute2 (Ubuntu):
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Canonical Kernel Team (canonical-kernel-team)
Changed in ubuntu-power-systems:
assignee: Canonical Server Team (canonical-server) → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Both fixes are in v4.19.0 and later

Changed in iproute2 (Ubuntu):
status: New → Fix Released
Changed in iproute2 (Ubuntu Bionic):
status: New → Triaged
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Changed in iproute2 (Ubuntu):
assignee: Canonical Kernel Team (canonical-kernel-team) → nobody
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-05-29 11:58 EDT-------
(In reply to comment #8)
> Both fixes are in v4.19.0 and later

Do you know the link of Ubuntu that we can get iproute2 v4.19.0 or later version?

Revision history for this message
Frank Heimes (fheimes) wrote :

The Lauchpad package site for iproute2 is this one:
https://launchpad.net/ubuntu/+source/iproute2

Changed in ubuntu-power-systems:
status: New → Triaged
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-05-29 12:54 EDT-------
From the link:
https://launchpad.net/ubuntu/+source/iproute2
For Ubuntu Bionic release, the latest supported version is 4.18.0-1ubuntu2~ubuntu18.04.1.

The iproute2 5.x versions are supported in other releases (Eoan, Focal).

Genesis OS is based on Bionic kernel (18.04.1).

Question: How can Bionic based kernel get the iproute2 version v4.19.0 or later?

Stefan Bader (smb)
Changed in iproute2 (Ubuntu Bionic):
importance: Undecided → Medium
assignee: Canonical Kernel Team (canonical-kernel-team) → Stefan Bader (smb)
Revision history for this message
Stefan Bader (smb) wrote :

I believe the mention pre-req patch is actually wrong (or not quite the patch we would be looking for). Instead

commit 6e8634eb13f30c58d1e28d975d99509680e1abc3
Author: Roman Mashak <email address hidden>
Date: Thu Mar 29 18:12:35 2018 -0400

    tc: add oneline mode

    Add initial support for oneline mode in tc; actions, filters and qdiscs
    will be gradually updated in the follow-up patches.

    Signed-off-by: Roman Mashak <email address hidden>
    Signed-off-by: David Ahern <email address hidden>

Adds the local declaration and assignment (the assignment is still in there after moving the declaration with the other patch). The magic there is that _SL_ is "\\" if set into oneline mode and otherwise "\n". However since oneline is not yet supported in the 18.04 version, I would propose to just adjust the fix with a declaration of _SL_ = "\n" in tc/tc.c.

Revision history for this message
Stefan Bader (smb) wrote :

I would like to verify any changes before moving ahead with SRU. But the given instructions are incomplete or at least not generically usable. Could the instructions be updated with steps that can be performed in a generic VM.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-07-20 17:09 EDT-------
I download iproute2-4.19.0.tar.gz from this upstream url:
https://git.kernel.org/pub/scm/network/iproute2/iproute2.git

Build and tested on my existing ppc64 Genesis OS (4.15.0-1039-ibm-gt),
I do see it now reports correct tcp_flags mask.

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi David, since Ubuntu 18.04 / bionic is a versions that is already released and in service, we cannot simply bump it's iproute2 version to the latest from upstream (this is only possible for Ubuntu releases that are in development - aot 20.10).
Instead we need to follow the pretty strict SRU process (https://wiki.ubuntu.com/StableReleaseUpdates) and according to this we would need the patch/commit (or in this case the minimal list of patches/commits) that fix a certain issue.
Hence the approach in the bug description - listing the needed commits - is the right way.
But out engineer assumes in comment #5 (https://bugs.launchpad.net/ubuntu/+source/iproute2/+bug/1873961/comments/5) that the pre-req. patch is probably a different one - and before going further we want you to confirm.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-07-21 10:54 EDT-------
I download iproute2-4.18.0.tar.gz from this upstream url:
https://git.kernel.org/pub/scm/network/iproute2/iproute2.git

Then pulled in 2 patches documented in comment 1 on top of iproute2-4.18.0.tar.gz, build and test the functionality.
It builds successfully, and it does reports correct tcp_flags mask with these 2 patches.

Attach these 2 patches in the bug for reference.

The first patch declares global variable "const char *_SL_ = "\n";" in lib/utils.c to be used by 2nd patch to fix the build break issue.

Revision history for this message
bugproxy (bugproxy) wrote : Fix 1

------- Comment (attachment only) From <email address hidden> 2020-07-21 10:55 EDT-------

Revision history for this message
bugproxy (bugproxy) wrote : Fix 2

------- Comment (attachment only) From <email address hidden> 2020-07-21 10:56 EDT-------

Revision history for this message
Stefan Bader (smb) wrote :

Sorry this seems to have gotten list in translation. I already got the one patch fixed up. What I need is some generic instructions to check the result. Can one just use any pair of NICs or do they have to be special?

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-07-21 14:57 EDT-------
It can be any NIC, or even lo device.
The fix2.txt has the example like this:
$ $TC qdisc add dev lo ingress
$ $TC filter add dev lo parent ffff: prio 3 proto ip flower ip_tos 0x8/32
$ $TC fitler add dev lo parent ffff: prio 5 proto ip flower ip_proto tcp \
tcp_flags 0x909/f00

$ $TC filter show dev lo parent ffff:
filter protocol ip pref 3 flower chain 0
filter protocol ip pref 3 flower chain 0 handle 0x1
eth_type ipv4
ip_tos 0x8/32
not_in_hw
filter protocol ip pref 5 flower chain 0
filter protocol ip pref 5 flower chain 0 handle 0x1
eth_type ipv4
ip_proto tcp
tcp_flags 0x909/f00 /* <--- */

Revision history for this message
Stefan Bader (smb) wrote :

Oh, so the upstream patch had the test case all the time and I just blanked it, looking at the content there and visible comments in this bug report only. :/ Thanks for the pointer. That should be all needed and I can go on with the SRU process.

Changed in iproute2 (Ubuntu Bionic):
status: Triaged → In Progress
Stefan Bader (smb)
description: updated
description: updated
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: Triaged → In Progress
Revision history for this message
Stefan Bader (smb) wrote :

Test build of proposed change at: https://launchpad.net/~smb/+archive/ubuntu/bionic

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-07-22 10:07 EDT-------
Download "iproute2_4.15.0-2ubuntu1.2_ppc64el.deb" from:
https://launchpad.net/~smb/+archive/ubuntu/bionic/+packages

Installed it on Genesis OS (4.15.0-1039-ibm-gt). New "tc" command sum value:
root@ltcgen4:~# sum /sbin/tc
45845 729

I am able to see the correct tcp_flags mask with this new /sbin/tc".
The new package works.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted iproute2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/iproute2/4.15.0-2ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in iproute2 (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: In Progress → Fix Committed

All autopkgtests for the newly accepted iproute2 (4.15.0-2ubuntu1.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

libreswan/unknown (armhf)
ubuntu-fan/0.12.10 (amd64)
systemd/237-3ubuntu10.41 (amd64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#iproute2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Stefan Bader (smb) wrote :

The ADT failures related to lubreswan and ubuntu-fan were resolved by re-try. The systemd test fails somewhere in the upstream test but does not give any hints about which part of that section was considered a failure. The amd64 re-try has now succeeded and from the timing it feels like the failure was rather due to some timeout and the test-case is rather to blame.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-08-03 13:45 EDT-------
Follow the link Waiki Wright provided today:
"https://bugs.launchpad.net/ubuntu-power-systems/+bug/1873961 see comment 17"

Get the binary from:
https://launchpad.net/ubuntu/bionic/ppc64el/iproute2/4.15.0-2ubuntu1.2
iproute2_4.15.0-2ubuntu1.2_ppc64el.deb (783.3 KiB)

After installation:
root@ltcgen4:~# ls -al /sbin/tc
-rwxr-xr-x 1 root root 746392 Jul 15 10:16 /sbin/tc

root@ltcgen4:~# sum /sbin/tc
45845 729

root@ltcgen4:~# dpkg -l |grep iproute
ii iproute2 4.15.0-2ubuntu1.2 ppc64el networking and traffic control tools

It works with this iproute2 package.

# /sbin/tc qdisc add dev lo ingress
# /sbin/tc filter add dev lo parent ffff: prio 3 proto ip flower ip_tos 0x8/32
# /sbin/tc filter add dev lo parent ffff: prio 5 proto ip flower ip_proto tcp tcp_flags 0x909/f00

# /sbin/tc filter show dev lo parent ffff:
filter protocol ip pref 3 flower chain 0
filter protocol ip pref 3 flower chain 0 handle 0x1
eth_type ipv4
ip_tos 0x8/32
not_in_hw
filter protocol ip pref 5 flower chain 0
filter protocol ip pref 5 flower chain 0 handle 0x1
eth_type ipv4
ip_proto tcp
tcp_flags 0x909/f00 /* <--- */
not_in_hw

Frank Heimes (fheimes)
tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package iproute2 - 4.15.0-2ubuntu1.2

---------------
iproute2 (4.15.0-2ubuntu1.2) bionic; urgency=medium

  * tc filter show tcp_flags wrong mask value (LP: #1873961)
    - d/p/lp1873961-tc-fix-bugs-for-tcp_flags-and-ip_attr-hex-output.patch

 -- Stefan Bader <email address hidden> Wed, 15 Jul 2020 17:16:31 +0200

Changed in iproute2 (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for iproute2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in ubuntu-power-systems:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.