Repository navigation
Simplify (and futureproof) build infrastructure by optionally using Qemu #150
Copy link
Copy link
Open
Description
Activity
For x86 things are very healthy for this approach. You can create a script to check what is possible
#!/bin/bash # Extract only named CPU models (skipping headers/aliases) models=$(qemu-system-x86_64 -cpu help | awk '/x86/ {print $2}' | grep -v ":" | sed 's/\[//;s/\]//') echo "Testing CPU models for native KVM support..." echo "--------------------------------------------------------" for model in $models; do # Define the command to test # -monitor stdio + 'quit' ensures the process exits immediately cmd="qemu-system-x86_64 -enable-kvm -cpu $model,check -display none -monitor stdio" # Run command and capture warnings output=$(echo "quit" | $cmd 2>&1) if [[ $output == *"warning: host doesn't support"* ]]; then printf "[ ] %-20s | Cmd: %s\n" "$model" "FAIL (Needs Emulation)" else # Green tick for supported models printf "[\033[0;32m✓\033[0m] %-20s | %s\n" "$model" "qemu-system-x86_64 -enable-kvm -cpu $model" fi done
and this gives output like
ubuntu@ip-172-31-34-251:~$ sudo bash check_cpu.sh Testing CPU models for native KVM support... -------------------------------------------------------- [✓] 486 | qemu-system-x86_64 -enable-kvm -cpu 486 [✓] 486-v1 | qemu-system-x86_64 -enable-kvm -cpu 486-v1 [ ] Broadwell | Cmd: FAIL (Needs Emulation) [ ] Broadwell-IBRS | Cmd: FAIL (Needs Emulation) [✓] Broadwell-noTSX | qemu-system-x86_64 -enable-kvm -cpu Broadwell-noTSX [✓] Broadwell-noTSX-IBRS | qemu-system-x86_64 -enable-kvm -cpu Broadwell-noTSX-IBRS [ ] Broadwell-v1 | Cmd: FAIL (Needs Emulation) [✓] Broadwell-v2 | qemu-system-x86_64 -enable-kvm -cpu Broadwell-v2 [ ] Broadwell-v3 | Cmd: FAIL (Needs Emulation) [✓] Broadwell-v4 | qemu-system-x86_64 -enable-kvm -cpu Broadwell-v4 [ ] Cascadelake-Server | Cmd: FAIL (Needs Emulation) [✓] Cascadelake-Server-noTSX | qemu-system-x86_64 -enable-kvm -cpu Cascadelake-Server-noTSX [ ] Cascadelake-Server-v1 | Cmd: FAIL (Needs Emulation) [ ] Cascadelake-Server-v2 | Cmd: FAIL (Needs Emulation) [✓] Cascadelake-Server-v3 | qemu-system-x86_64 -enable-kvm -cpu Cascadelake-Server-v3 [✓] Cascadelake-Server-v4 | qemu-system-x86_64 -enable-kvm -cpu Cascadelake-Server-v4 [✓] Cascadelake-Server-v5 | qemu-system-x86_64 -enable-kvm -cpu Cascadelake-Server-v5 [✓] Conroe | qemu-system-x86_64 -enable-kvm -cpu Conroe [✓] Conroe-v1 | qemu-system-x86_64 -enable-kvm -cpu Conroe-v1 [ ] Cooperlake | Cmd: FAIL (Needs Emulation) [ ] Cooperlake-v1 | Cmd: FAIL (Needs Emulation) [ ] Cooperlake-v2 | Cmd: FAIL (Needs Emulation) [ ] Denverton | Cmd: FAIL (Needs Emulation) [ ] Denverton-v1 | Cmd: FAIL (Needs Emulation) [ ] Denverton-v2 | Cmd: FAIL (Needs Emulation) [ ] Denverton-v3 | Cmd: FAIL (Needs Emulation) [ ] Dhyana | Cmd: FAIL (Needs Emulation) [ ] Dhyana-v1 | Cmd: FAIL (Needs Emulation) [ ] Dhyana-v2 | Cmd: FAIL (Needs Emulation) [ ] EPYC | Cmd: FAIL (Needs Emulation) [ ] EPYC-Genoa | Cmd: FAIL (Needs Emulation) [ ] EPYC-Genoa-v1 | Cmd: FAIL (Needs Emulation) [ ] EPYC-IBPB | Cmd: FAIL (Needs Emulation) [ ] EPYC-Milan | Cmd: FAIL (Needs Emulation) [ ] EPYC-Milan-v1 | Cmd: FAIL (Needs Emulation) [ ] EPYC-Milan-v2 | Cmd: FAIL (Needs Emulation) [ ] EPYC-Rome | Cmd: FAIL (Needs Emulation) [ ] EPYC-Rome-v1 | Cmd: FAIL (Needs Emulation) [ ] EPYC-Rome-v2 | Cmd: FAIL (Needs Emulation) [ ] EPYC-Rome-v3 | Cmd: FAIL (Needs Emulation) [ ] EPYC-Rome-v4 | Cmd: FAIL (Needs Emulation) [ ] EPYC-v1 | Cmd: FAIL (Needs Emulation) [ ] EPYC-v2 | Cmd: FAIL (Needs Emulation) [ ] EPYC-v3 | Cmd: FAIL (Needs Emulation) [ ] EPYC-v4 | Cmd: FAIL (Needs Emulation) [ ] GraniteRapids | Cmd: FAIL (Needs Emulation) [ ] GraniteRapids-v1 | Cmd: FAIL (Needs Emulation) [ ] Haswell | Cmd: FAIL (Needs Emulation) [ ] Haswell-IBRS | Cmd: FAIL (Needs Emulation) [✓] Haswell-noTSX | qemu-system-x86_64 -enable-kvm -cpu Haswell-noTSX [✓] Haswell-noTSX-IBRS | qemu-system-x86_64 -enable-kvm -cpu Haswell-noTSX-IBRS [ ] Haswell-v1 | Cmd: FAIL (Needs Emulation) [✓] Haswell-v2 | qemu-system-x86_64 -enable-kvm -cpu Haswell-v2 [ ] Haswell-v3 | Cmd: FAIL (Needs Emulation) [✓] Haswell-v4 | qemu-system-x86_64 -enable-kvm -cpu Haswell-v4 [ ] Icelake-Server | Cmd: FAIL (Needs Emulation) [ ] Icelake-Server-noTSX | Cmd: FAIL (Needs Emulation) [ ] Icelake-Server-v1 | Cmd: FAIL (Needs Emulation) [ ] Icelake-Server-v2 | Cmd: FAIL (Needs Emulation) [ ] Icelake-Server-v3 | Cmd: FAIL (Needs Emulation) [ ] Icelake-Server-v4 | Cmd: FAIL (Needs Emulation) [ ] Icelake-Server-v5 | Cmd: FAIL (Needs Emulation) [ ] Icelake-Server-v6 | Cmd: FAIL (Needs Emulation) [✓] IvyBridge | qemu-system-x86_64 -enable-kvm -cpu IvyBridge [✓] IvyBridge-IBRS | qemu-system-x86_64 -enable-kvm -cpu IvyBridge-IBRS [✓] IvyBridge-v1 | qemu-system-x86_64 -enable-kvm -cpu IvyBridge-v1 [✓] IvyBridge-v2 | qemu-system-x86_64 -enable-kvm -cpu IvyBridge-v2 [ ] KnightsMill | Cmd: FAIL (Needs Emulation) [ ] KnightsMill-v1 | Cmd: FAIL (Needs Emulation) [✓] Nehalem | qemu-system-x86_64 -enable-kvm -cpu Nehalem [✓] Nehalem-IBRS | qemu-system-x86_64 -enable-kvm -cpu Nehalem-IBRS [✓] Nehalem-v1 | qemu-system-x86_64 -enable-kvm -cpu Nehalem-v1 [✓] Nehalem-v2 | qemu-system-x86_64 -enable-kvm -cpu Nehalem-v2 [✓] Opteron_G1 | qemu-system-x86_64 -enable-kvm -cpu Opteron_G1 [✓] Opteron_G1-v1 | qemu-system-x86_64 -enable-kvm -cpu Opteron_G1-v1 [ ] Opteron_G2 | Cmd: FAIL (Needs Emulation) [ ] Opteron_G2-v1 | Cmd: FAIL (Needs Emulation) [ ] Opteron_G3 | Cmd: FAIL (Needs Emulation) [ ] Opteron_G3-v1 | Cmd: FAIL (Needs Emulation) [ ] Opteron_G4 | Cmd: FAIL (Needs Emulation) [ ] Opteron_G4-v1 | Cmd: FAIL (Needs Emulation) [ ] Opteron_G5 | Cmd: FAIL (Needs Emulation) [ ] Opteron_G5-v1 | Cmd: FAIL (Needs Emulation) [✓] Penryn | qemu-system-x86_64 -enable-kvm -cpu Penryn [✓] Penryn-v1 | qemu-system-x86_64 -enable-kvm -cpu Penryn-v1 [✓] SandyBridge | qemu-system-x86_64 -enable-kvm -cpu SandyBridge [✓] SandyBridge-IBRS | qemu-system-x86_64 -enable-kvm -cpu SandyBridge-IBRS [✓] SandyBridge-v1 | qemu-system-x86_64 -enable-kvm -cpu SandyBridge-v1 [✓] SandyBridge-v2 | qemu-system-x86_64 -enable-kvm -cpu SandyBridge-v2 [ ] SapphireRapids | Cmd: FAIL (Needs Emulation) [ ] SapphireRapids-v1 | Cmd: FAIL (Needs Emulation) [ ] SapphireRapids-v2 | Cmd: FAIL (Needs Emulation) [ ] SierraForest | Cmd: FAIL (Needs Emulation) [ ] SierraForest-v1 | Cmd: FAIL (Needs Emulation) [ ] Skylake-Client | Cmd: FAIL (Needs Emulation) [ ] Skylake-Client-IBRS | Cmd: FAIL (Needs Emulation) [✓] Skylake-Client-noTSX-IBRS | qemu-system-x86_64 -enable-kvm -cpu Skylake-Client-noTSX-IBRS [ ] Skylake-Client-v1 | Cmd: FAIL (Needs Emulation) [ ] Skylake-Client-v2 | Cmd: FAIL (Needs Emulation) [✓] Skylake-Client-v3 | qemu-system-x86_64 -enable-kvm -cpu Skylake-Client-v3 [✓] Skylake-Client-v4 | qemu-system-x86_64 -enable-kvm -cpu Skylake-Client-v4 [ ] Skylake-Server | Cmd: FAIL (Needs Emulation) [ ] Skylake-Server-IBRS | Cmd: FAIL (Needs Emulation) [✓] Skylake-Server-noTSX-IBRS | qemu-system-x86_64 -enable-kvm -cpu Skylake-Server-noTSX-IBRS [ ] Skylake-Server-v1 | Cmd: FAIL (Needs Emulation) [ ] Skylake-Server-v2 | Cmd: FAIL (Needs Emulation) [✓] Skylake-Server-v3 | qemu-system-x86_64 -enable-kvm -cpu Skylake-Server-v3 [✓] Skylake-Server-v4 | qemu-system-x86_64 -enable-kvm -cpu Skylake-Server-v4 [✓] Skylake-Server-v5 | qemu-system-x86_64 -enable-kvm -cpu Skylake-Server-v5 [ ] Snowridge | Cmd: FAIL (Needs Emulation) [ ] Snowridge-v1 | Cmd: FAIL (Needs Emulation) [ ] Snowridge-v2 | Cmd: FAIL (Needs Emulation) [ ] Snowridge-v3 | Cmd: FAIL (Needs Emulation) [ ] Snowridge-v4 | Cmd: FAIL (Needs Emulation) [✓] Westmere | qemu-system-x86_64 -enable-kvm -cpu Westmere [✓] Westmere-IBRS | qemu-system-x86_64 -enable-kvm -cpu Westmere-IBRS [✓] Westmere-v1 | qemu-system-x86_64 -enable-kvm -cpu Westmere-v1 [✓] Westmere-v2 | qemu-system-x86_64 -enable-kvm -cpu Westmere-v2 [ ] athlon | Cmd: FAIL (Needs Emulation) [ ] athlon-v1 | Cmd: FAIL (Needs Emulation) [✓] core2duo | qemu-system-x86_64 -enable-kvm -cpu core2duo [✓] core2duo-v1 | qemu-system-x86_64 -enable-kvm -cpu core2duo-v1 [✓] coreduo | qemu-system-x86_64 -enable-kvm -cpu coreduo [✓] coreduo-v1 | qemu-system-x86_64 -enable-kvm -cpu coreduo-v1 [✓] kvm32 | qemu-system-x86_64 -enable-kvm -cpu kvm32 [✓] kvm32-v1 | qemu-system-x86_64 -enable-kvm -cpu kvm32-v1 [✓] kvm64 | qemu-system-x86_64 -enable-kvm -cpu kvm64 [✓] kvm64-v1 | qemu-system-x86_64 -enable-kvm -cpu kvm64-v1 [✓] n270 | qemu-system-x86_64 -enable-kvm -cpu n270 [✓] n270-v1 | qemu-system-x86_64 -enable-kvm -cpu n270-v1 [✓] pentium | qemu-system-x86_64 -enable-kvm -cpu pentium [✓] pentium-v1 | qemu-system-x86_64 -enable-kvm -cpu pentium-v1 [✓] pentium2 | qemu-system-x86_64 -enable-kvm -cpu pentium2 [✓] pentium2-v1 | qemu-system-x86_64 -enable-kvm -cpu pentium2-v1 [✓] pentium3 | qemu-system-x86_64 -enable-kvm -cpu pentium3 [✓] pentium3-v1 | qemu-system-x86_64 -enable-kvm -cpu pentium3-v1 [ ] phenom | Cmd: FAIL (Needs Emulation) [ ] phenom-v1 | Cmd: FAIL (Needs Emulation) [✓] qemu32 | qemu-system-x86_64 -enable-kvm -cpu qemu32 [✓] qemu32-v1 | qemu-system-x86_64 -enable-kvm -cpu qemu32-v1 [ ] qemu64 | Cmd: FAIL (Needs Emulation) [ ] qemu64-v1 | Cmd: FAIL (Needs Emulation) [✓] base | qemu-system-x86_64 -enable-kvm -cpu base [✓] host | qemu-system-x86_64 -enable-kvm -cpu host [✓] max | qemu-system-x86_64 -enable-kvm -cpu maxUnfortunately, you can't seem to get the same behaviour on Arm, KVM there is much more restricted and you basically only get the host without software emulation (which would be too slow for us probably)
I ran an upgraded version of the script on the new nested-virtualisation enabled C8i instances
#!/bin/bash echo "Checking QEMU/KVM permissions for user: $USER" echo "================================================" # Check if /dev/kvm exists if [[ ! -e /dev/kvm ]]; then echo "[FAIL] /dev/kvm not present. KVM not enabled in kernel or BIOS." exit 1 fi # Check permissions if [[ ! -r /dev/kvm || ! -w /dev/kvm ]]; then echo "[FAIL] Current user cannot access /dev/kvm" ls -l /dev/kvm exit 1 fi # Check group membership if groups "$USER" | grep -qw kvm; then echo "[OK] User belongs to 'kvm' group" else echo "[WARN] User is not in the 'kvm' group" fi # Test that QEMU can actually use KVM if ! echo quit | qemu-system-x86_64 \ -enable-kvm \ -cpu host \ -display none \ -monitor stdio \ >/dev/null 2>&1; then echo "[FAIL] QEMU cannot start with KVM acceleration" exit 1 fi echo "[OK] QEMU can start with KVM" echo echo "Testing CPU models for native KVM support..." echo "================================================" # Extract only named CPU models models=$(qemu-system-x86_64 -cpu help | awk '/x86/ {print $2}' | grep -v ":" | sed 's/\[//;s/\]//') for model in $models; do echo quit | qemu-system-x86_64 \ -enable-kvm \ -cpu "$model,enforce" \ -display none \ -monitor stdio \ -nodefaults \ >/dev/null 2>&1 rc=$? if [[ $rc -ne 0 ]]; then printf "[ ] %-20s | FAIL (Needs Emulation)\n" "$model" else printf "[\033[0;32m✓\033[0m] %-20s | Supported\n" "$model" fi done
which gives the output
Checking QEMU/KVM permissions for user: ubuntu ================================================ [OK] User belongs to 'kvm' group [OK] QEMU can start with KVM Testing CPU models for native KVM support... ================================================ [✓] 486 | Supported [✓] 486-v1 | Supported [ ] Broadwell | FAIL (Needs Emulation) [ ] Broadwell-IBRS | FAIL (Needs Emulation) [✓] Broadwell-noTSX | Supported [✓] Broadwell-noTSX-IBRS | Supported [ ] Broadwell-v1 | FAIL (Needs Emulation) [✓] Broadwell-v2 | Supported [ ] Broadwell-v3 | FAIL (Needs Emulation) [✓] Broadwell-v4 | Supported [ ] Cascadelake-Server | FAIL (Needs Emulation) [✓] Cascadelake-Server-noTSX | Supported [ ] Cascadelake-Server-v1 | FAIL (Needs Emulation) [ ] Cascadelake-Server-v2 | FAIL (Needs Emulation) [✓] Cascadelake-Server-v3 | Supported [✓] Cascadelake-Server-v4 | Supported [✓] Cascadelake-Server-v5 | Supported [✓] Conroe | Supported [✓] Conroe-v1 | Supported [ ] Cooperlake | FAIL (Needs Emulation) [ ] Cooperlake-v1 | FAIL (Needs Emulation) [ ] Cooperlake-v2 | FAIL (Needs Emulation) [ ] Denverton | FAIL (Needs Emulation) [ ] Denverton-v1 | FAIL (Needs Emulation) [✓] Denverton-v2 | Supported [✓] Denverton-v3 | Supported [ ] Dhyana | FAIL (Needs Emulation) [ ] Dhyana-v1 | FAIL (Needs Emulation) [ ] Dhyana-v2 | FAIL (Needs Emulation) [ ] EPYC | FAIL (Needs Emulation) [ ] EPYC-Genoa | FAIL (Needs Emulation) [ ] EPYC-Genoa-v1 | FAIL (Needs Emulation) [ ] EPYC-IBPB | FAIL (Needs Emulation) [ ] EPYC-Milan | FAIL (Needs Emulation) [ ] EPYC-Milan-v1 | FAIL (Needs Emulation) [ ] EPYC-Milan-v2 | FAIL (Needs Emulation) [ ] EPYC-Rome | FAIL (Needs Emulation) [ ] EPYC-Rome-v1 | FAIL (Needs Emulation) [ ] EPYC-Rome-v2 | FAIL (Needs Emulation) [ ] EPYC-Rome-v3 | FAIL (Needs Emulation) [ ] EPYC-Rome-v4 | FAIL (Needs Emulation) [ ] EPYC-v1 | FAIL (Needs Emulation) [ ] EPYC-v2 | FAIL (Needs Emulation) [ ] EPYC-v3 | FAIL (Needs Emulation) [ ] EPYC-v4 | FAIL (Needs Emulation) [ ] GraniteRapids | FAIL (Needs Emulation) [ ] GraniteRapids-v1 | FAIL (Needs Emulation) [ ] Haswell | FAIL (Needs Emulation) [ ] Haswell-IBRS | FAIL (Needs Emulation) [✓] Haswell-noTSX | Supported [✓] Haswell-noTSX-IBRS | Supported [ ] Haswell-v1 | FAIL (Needs Emulation) [✓] Haswell-v2 | Supported [ ] Haswell-v3 | FAIL (Needs Emulation) [✓] Haswell-v4 | Supported [ ] Icelake-Server | FAIL (Needs Emulation) [ ] Icelake-Server-noTSX | FAIL (Needs Emulation) [ ] Icelake-Server-v1 | FAIL (Needs Emulation) [ ] Icelake-Server-v2 | FAIL (Needs Emulation) [ ] Icelake-Server-v3 | FAIL (Needs Emulation) [ ] Icelake-Server-v4 | FAIL (Needs Emulation) [ ] Icelake-Server-v5 | FAIL (Needs Emulation) [ ] Icelake-Server-v6 | FAIL (Needs Emulation) [✓] IvyBridge | Supported [✓] IvyBridge-IBRS | Supported [✓] IvyBridge-v1 | Supported [✓] IvyBridge-v2 | Supported [ ] KnightsMill | FAIL (Needs Emulation) [ ] KnightsMill-v1 | FAIL (Needs Emulation) [✓] Nehalem | Supported [✓] Nehalem-IBRS | Supported [✓] Nehalem-v1 | Supported [✓] Nehalem-v2 | Supported [✓] Opteron_G1 | Supported [✓] Opteron_G1-v1 | Supported [ ] Opteron_G2 | FAIL (Needs Emulation) [ ] Opteron_G2-v1 | FAIL (Needs Emulation) [ ] Opteron_G3 | FAIL (Needs Emulation) [ ] Opteron_G3-v1 | FAIL (Needs Emulation) [ ] Opteron_G4 | FAIL (Needs Emulation) [ ] Opteron_G4-v1 | FAIL (Needs Emulation) [ ] Opteron_G5 | FAIL (Needs Emulation) [ ] Opteron_G5-v1 | FAIL (Needs Emulation) [✓] Penryn | Supported [✓] Penryn-v1 | Supported [✓] SandyBridge | Supported [✓] SandyBridge-IBRS | Supported [✓] SandyBridge-v1 | Supported [✓] SandyBridge-v2 | Supported [ ] SapphireRapids | FAIL (Needs Emulation) [ ] SapphireRapids-v1 | FAIL (Needs Emulation) [ ] SapphireRapids-v2 | FAIL (Needs Emulation) [ ] SierraForest | FAIL (Needs Emulation) [ ] SierraForest-v1 | FAIL (Needs Emulation) [ ] Skylake-Client | FAIL (Needs Emulation) [ ] Skylake-Client-IBRS | FAIL (Needs Emulation) [✓] Skylake-Client-noTSX-IBRS | Supported [ ] Skylake-Client-v1 | FAIL (Needs Emulation) [ ] Skylake-Client-v2 | FAIL (Needs Emulation) [✓] Skylake-Client-v3 | Supported [✓] Skylake-Client-v4 | Supported [ ] Skylake-Server | FAIL (Needs Emulation) [ ] Skylake-Server-IBRS | FAIL (Needs Emulation) [✓] Skylake-Server-noTSX-IBRS | Supported [ ] Skylake-Server-v1 | FAIL (Needs Emulation) [ ] Skylake-Server-v2 | FAIL (Needs Emulation) [✓] Skylake-Server-v3 | Supported [✓] Skylake-Server-v4 | Supported [✓] Skylake-Server-v5 | Supported [ ] Snowridge | FAIL (Needs Emulation) [ ] Snowridge-v1 | FAIL (Needs Emulation) [ ] Snowridge-v2 | FAIL (Needs Emulation) [ ] Snowridge-v3 | FAIL (Needs Emulation) [✓] Snowridge-v4 | Supported [✓] Westmere | Supported [✓] Westmere-IBRS | Supported [✓] Westmere-v1 | Supported [✓] Westmere-v2 | Supported [ ] athlon | FAIL (Needs Emulation) [ ] athlon-v1 | FAIL (Needs Emulation) [✓] core2duo | Supported [✓] core2duo-v1 | Supported [✓] coreduo | Supported [✓] coreduo-v1 | Supported [✓] kvm32 | Supported [✓] kvm32-v1 | Supported [✓] kvm64 | Supported [✓] kvm64-v1 | Supported [✓] n270 | Supported [✓] n270-v1 | Supported [✓] pentium | Supported [✓] pentium-v1 | Supported [✓] pentium2 | Supported [✓] pentium2-v1 | Supported [✓] pentium3 | Supported [✓] pentium3-v1 | Supported [ ] phenom | FAIL (Needs Emulation) [ ] phenom-v1 | FAIL (Needs Emulation) [✓] qemu32 | Supported [✓] qemu32-v1 | Supported [ ] qemu64 | FAIL (Needs Emulation) [ ] qemu64-v1 | FAIL (Needs Emulation) [✓] base | Supported [✓] host | Supported [✓] max | SupportedThis also gives us a path to test binaries on different CPUs
Metadata
Metadata
Assignees
Labels
No labels
Getting access to old architectures is going to get harder and harder. Already, there are only a very small number AWS regions that support all our target architectures (if we stick to EU region only we have to use a minimum of two). It seems like it could be a good approach to consider supporting a Qemu/KVM mode in our build scripts so that we support older CPUs on newer architectures.
This doesn't necessarily have to come with a performance penalty, if we group the CPUs into familes there doesn't need to be any translation between instructions (Intel Sapphaire Rapids can run all Intel CPUs down to generic x86, Zen5 can do all Zen families, Arm is a bit more complicated but the model still mostly works). For build clusters that we control this should be feasible, but of course where we run directly connected to an HPC system you won't have this option.
This approach should give us much better resilience, and make leveraging EOSC resources (for example) more attractive for us.