Add integration tests for .NET

[eyalb181]

We should add integration tests for common .NET workflows.

[IsaacCloos]

✋🏻

I'd like to take this issue 😄

[eyalb181]

Good luck!

[IsaacCloos]

Preamble

Sorry for delay! I have most of my free time on the weekends, and was only just able to sit down to get a lay-of-the-land on this issue. If at any point this issue or related works need to be escalated please reassign 👍🏻.

TLDR

Stage

Stage	Status
Get Current Tests Working	In Progress
Define Test Expectations
Replicate Tests in .NET

Issues

System details

macOS Monterey Version 12.5 | Arch : arm64

test>README.md

mirrord/tests/README.md

Lines 1 to 11 in 3795e75

	# How to Run the E2E tests
	To run the tests locally with the latest mirrord-agent image, run the following in the mirrord directory:
	- `docker build -t test . --file mirrord-agent/Dockerfile`
	- `minikube image load test` (you might have to specify `-p <PROFILE-NAME>` as well)
	- `cargo test --package tests --lib -- tests --nocapture`
	The name "test" is hardcoded for the CI, and the tests will fail with an `Elapsed` error if the image named "test" is not found.
	To use a different image change the environment variable `MIRRORD_AGENT_IMAGE` at `test_server_init` in `tests/src/utils.rs`.
	To use the latest release of mirrord-agent, comment out the line adding the `MIRRORD_AGENT_IMAGE` environment.

Following these instructions was more difficult (but doable) than the excellent online resource you have hosted. I think it would be an easy win to point to that resource from this file.

I had mixed results getting the minikube deployment to execute the tests consistently, but after following the online guide it was smooth sailing (minus the points to follow). A possibly helpful upgrade to the doc site would be extending the examples to include minikube (since it's specifically recommended in the test readme file above).

Flakey tests

#473 will be helpful on my end absolutely. I've had mixed results from this test regardless of configuration or local kube solution. Typically related to www.live.com (which forwards to outlook), but often others as well.

mirrord/tests/src/sanity.rs

Lines 1049 to 1062 in 3795e75

	#[rstest]
	#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
	pub async fn test_outgoing_traffic_many_requests_enabled(#[future] service: KubeService) {
	let service = service.await;
	let node_command = vec![
	"node",
	"node-e2e/outgoing/test_outgoing_traffic_many_requests.mjs",
	];
	let mut process = run(node_command, &service.pod_name, None, None).await;
	let res = process.child.wait().await.unwrap();
	assert!(res.success());
	process.assert_stderr();
	}

Failing tests

All Golang related e2e tests fail on my local machine with similar errors to this:

Launching "go-e2e-dns/19" with arguments []

Extracted library file to "/var/folders/xx/5c4wz4613s780646p68l6g380000gn/T/test.ut6bK01O1C3G/x0slmsfwds-libmirrord_layer.dylib"

Couldn't execute Errno(Errno { code: 2, description: Some("No such file or directory") })

I have spent this morning trying to figure out if these errors are a fault of how I followed the test environment instructions with no luck. I have found related work #248 #260, but I'm not entirely sure these are justifications for me to ignore these failing on my machine, so I thought I'd ask. One other piece of evidence I can point to is:

mirrord/.github/workflows/ci.yaml

Lines 152 to 279 in 3795e75

	e2e:
	runs-on: ubuntu-latest
	name: e2e
	needs: [build_mirrord, test_agent_image]
	env:
	MIRRORD_AGENT_RUST_LOG: "warn,mirrord=debug"
	steps:
	- uses: actions/checkout@v2
	- uses: actions-rs/toolchain@v1
	with:
	profile: minimal
	toolchain: nightly
	components: rustfmt
	- uses: Swatinem/rust-cache@v1
	with:
	key: ubuntu-latest
	cache-on-failure: true
	- uses: actions/setup-node@v3
	with:
	node-version: 14
	- run: npm install express
	- run: |
	sudo apt-get update -y
	sudo apt-get install -y libpcap-dev cmake
	- uses: actions/setup-python@v3
	- run: pip3 install flask
	- run: pip3 install fastapi
	- run: pip3 install uvicorn[standard]
	- uses: actions/setup-go@v3
	with:
	go-version: "1.18.0"
	- run: |
	cd tests/go-e2e
	go build -o 18
	- run: |
	cd tests/go-e2e-fileops
	go build -o 18
	- run: |
	cd tests/go-e2e-env
	go build -o 18
	- run: |
	cd tests/go-e2e-outgoing
	go build -o 18
	- run: |
	cd tests/go-e2e-dns
	go build -o 18
	- uses: actions/setup-go@v3
	with:
	go-version: "1.19.0"
	- run: |
	cd tests/go-e2e
	go build -o 19
	- run: |
	cd tests/go-e2e-fileops
	go build -o 19
	- run: |
	cd tests/go-e2e-env
	go build -o 19
	- run: |
	cd tests/go-e2e-outgoing
	go build -o 19
	- run: |
	cd tests/go-e2e-dns
	go build -o 19
	- name: start minikube
	uses: medyagh/setup-minikube@master
	with:
	container-runtime: containerd
	- name: download image
	uses: actions/download-artifact@v2
	with:
	name: test
	path: /tmp
	- run: minikube image load /tmp/test.tar
	- name: cargo test
	run: cargo test -p tests
	- name: switch minikube runtime
	run: |
	minikube delete
	minikube start --container-runtime=docker
	minikube image load /tmp/test.tar
	- name: test with docker runtime
	run: cargo test -p tests
	- name: Collect logs
	if: ${{ failure() }}
	run: |
	kubectl describe pods
	docker exec minikube find /var/log/pods -print -exec cat {} \;
	e2e_macos:
	runs-on: macos-12
	name: e2e macOS
	needs: [build_mirrord, test_agent_image]
	env:
	MIRRORD_AGENT_RUST_LOG: "warn,mirrord=debug"
	steps:
	- uses: actions/checkout@v2
	- uses: actions-rs/toolchain@v1
	with:
	profile: minimal
	toolchain: nightly
	components: rustfmt
	- uses: Swatinem/rust-cache@v1
	with:
	key: macos-latest
	cache-on-failure: true
	- run: npm install express
	- uses: actions/setup-python@v3
	- run: pip3 install flask
	- run: pip3 install fastapi
	- run: pip3 install uvicorn[standard]
	- name: setup cluster requirements
	run: brew install docker colima kubectl
	- name: create cluster
	run: colima start --kubernetes --network-address --cpu 3 --memory 8
	- name: download image
	uses: actions/download-artifact@v2
	with:
	name: test
	path: /tmp
	- name: load image
	run: docker load --input /tmp/test.tar
	- name: cargo test
	run: cargo test -p tests test_mirror_http_traffic
	- name: Collect container logs
	if: ${{ failure() }}
	run: for CONTAINER in $(docker ps --format "{{.ID}}"); do echo $CONTAINER && docker logs $CONTAINER; done

All of the go-e2e tests are omitted from the macOS ci test suite.

Question

Having followed the developer testing guide:
Should I be concerned that my local environment is not set up correctly if any of these tests fail¹? I want to make sure these things are all in order before I add new tests and believe them to be consistent, passable, and properly fail-able when real problems arise.

Thanks so much for reading all my blathering! Any feedback/advice would mean the world to me 😄

Footnotes

1. Besides the 6 Go tests in question, the other sanity tests are passing consistently ↩

[infiniteregrets]

Launching "go-e2e-dns/19" with arguments []
Extracted library file to "/var/folders/xx/5c4wz4613s780646p68l6g380000gn/T/test.ut6bK01O1C3G/x0slmsfwds-libmirrord_layer.dylib"
Couldn't execute Errno(Errno { code: 2, description: Some("No such file or directory") })

How are you running mirrord? This happens when the binary path is not valid

[aviramha]

Launching "go-e2e-dns/19" with arguments []
Extracted library file to "/var/folders/xx/5c4wz4613s780646p68l6g380000gn/T/test.ut6bK01O1C3G/x0slmsfwds-libmirrord_layer.dylib"
Couldn't execute Errno(Errno { code: 2, description: Some("No such file or directory") })

How are you running mirrord? This happens when the binary path is not valid

I assume they didn't compile the go e2e tests, we should add a note that this is required for go e2e to run.

[IsaacCloos]

Launching "go-e2e-dns/19" with arguments []
Extracted library file to "/var/folders/xx/5c4wz4613s780646p68l6g380000gn/T/test.ut6bK01O1C3G/x0slmsfwds-libmirrord_layer.dylib"
Couldn't execute Errno(Errno { code: 2, description: Some("No such file or directory") })

How are you running mirrord? This happens when the binary path is not valid

The online Testing & Development Guide recommends running mirrord with a local process:
MIRRORD_AGENT_IMAGE=test MIRRORD_AGENT_RUST_LOG=debug RUST_LOG=debug target/debug/mirrord exec -c --pod-name py-serv-deployment-ff89b5974-x9tjx node sample/node/app.mjs. This is what I've been doing (with the pod name changed).

---

I assume they didn't compile the go e2e tests, we should add a note that this is required for go e2e to run.

I have attempted the tests again after compiling all of the Go folders included in the project with no success. Could you elaborate on how exactly this should be done?

Thank you both for your feedback and advice 😄

[aviramha]

@IsaacCloos - Thank you and sorry for the troubles!
I think it's best to follow what we do in the CI yaml specifically the e2e part.
For each go directory, we both compile it for go 1.18 and go 1.19.

      - uses: actions/setup-go@v3
        with:
          go-version: "1.18.0"
      - run: |
          cd tests/go-e2e
          go build -o 18
      - uses: actions/setup-go@v3
        with:
          go-version: "1.19.0"
      - run: |
          cd tests/go-e2e
          go build -o 19

[IsaacCloos]

Thank you for the help!

Draft PR #480 is open for feedback and I'd love to iterate on that work until it meets the needs of this project.

Also,

The test suite of this project has been a learning curve. The different ways to run the project, container environments to use, and different architectures can lead to a lot of extra work to feel comfortable with tests (new and old) working correctly and reporting relevant failures/successes. With that said, after this issue is closed I'd love to provide feedback to perhaps improve the QOL for future contributors looking to work in the tests section of the project.

[IsaacCloos]

Absolutely no rush on this; can the title and description be updated to reflect the newly desired integration tests?

Do these tests live within the mirrord-layer>tests exclusively?

I want to make sure these tests are helpful and designed correctly. A little direction would make a huge difference 😄

[aviramha]

Absolutely no rush on this; can the title and description be updated to reflect the newly desired integration tests?

Do these tests live within the mirrord-layer>tests exclusively?

I want to make sure these tests are helpful and designed correctly. A little direction would make a huge difference 😄

Yes, that's the part in mirrord that is affected by frameworks/languages so it's actually testing it's flow and logic, hence being there.
You can see example of the other IT tests, but in general the idea is that we run an app with layer injected to it (using LD/DYLD method), connect it to a "mock" agent that validates the requests and sends the appropriate responses to see that it behaves correctly in a deterministic manner.