-march=native question

[lano1106]

the basics.md document mentions avoiding using the compiler switch -march=native but it does not say why...

is it because it does not add much since you detect and pick the best implementation at runtime or does it simply break the compilation of the various implementations?

[lemire]

It breaks runtime dispatching. The resulting binary will no longer run everywhere.

The basic documentation does not go into details, but elsewhere, we offer more detailed comments:

Instead of specifying a specific microarchitecture, you can let your compiler do the work. The -march=native flags says "target the current computer," which is a reasonable default for many applications which both compile and run on the same processor.

Passing -march=native to the compiler may make On Demand faster by allowing it to use optimizations specific to your machine. You cannot do this, however, if you are compiling code that might be run on less advanced machines. That is, be mindful that when compiling with the -march=native flag, the resulting binary will run on the current system but may not run on other systems (e.g., on an old processor).

If you are compiling on an ARM or POWER system, you do not need to be concerned with CPU selection during compilation. The -march=native flag is useful for best performance on x64 (e.g., Intel) systems but it is generally unsupported on some platforms such as ARM (aarch64) or POWER.

[lano1106]

I have just tried the small code snippet on my skylake-avx512 machine... runtime detection does indeed pick the AVX2 implementation...

simdjson v3.6.4
Detected the best implementation for your machine: haswell(Intel/AMD AVX2)

I am going to have fun looking into simdjson code with this challenge in my hands...

[lano1106]

I have looked into simdjson/icelake/intrinsics.h:

#include <avx512vbmiintrin.h>
#include <avx512vbmi2intrin.h>

of all the functions declared in those headers files (unless I did not search correctly),

no VBMI intrinsics are called. only VBMI2 and only at one location:

in simdjson/icelake/simd.h:

    // Copies to 'output" all bytes corresponding to a 0 in the mask (interpreted as a bitset).
    // Passing a 0 value for mask would be equivalent to writing out every byte to output.
    // Only the first 32 - count_ones(mask) bytes of the result are significant but 32 bytes
    // get written.
    // Design consideration: it seems like a function with the
    // signature simd8<L> compress(uint32_t mask) would be
    // sensible, but the AVX ISA makes this kind of approach difficult.
    template<typename L>
    simdjson_inline void compress(uint64_t mask, L * output) const {
      _mm512_mask_compressstoreu_epi8 (output,~mask,*this);
    }

I have found another VBMI instruction occurence in src/icelake.cpp:

simdjson_inline void bit_indexer::write(uint32_t idx, uint64_t bits) {
    // In some instances, the next branch is expensive because it is mispredicted.
    // Unfortunately, in other cases,
    // it helps tremendously.
    if (bits == 0) { return; }

    const __m512i indexes = _mm512_maskz_compress_epi8(bits, _mm512_set_epi32(
      0x3f3e3d3c, 0x3b3a3938, 0x37363534, 0x33323130,
      0x2f2e2d2c, 0x2b2a2928, 0x27262524, 0x23222120,
      0x1f1e1d1c, 0x1b1a1918, 0x17161514, 0x13121110,
      0x0f0e0d0c, 0x0b0a0908, 0x07060504, 0x03020100
    ));

this is the only spot that I detected needing VBMI... does that look correct to you?

[lano1106]

ok ok... I am going to stop looking into this... I have never worked with simd intrinsics functions... My initial idea would be to implement a skylake-avx512 implementation compress method in terms the haswell compress method...

I have no idea if it is possible or not but on the surface, this seems like a reasonable idea... My first hurdle is how to convert a __m512i value into 2 __m256i...

I have looked into gcc header files... __m512i is a relatively simple typedef:

typedef long long __m512i __attribute__ ((__vector_size__ (64), __may_alias__));

there must exist a macro to do just that... if not, that could be not too hard with relatively simple pointer arithmetic to do that... but then what is confusing me is the endianness... x86.. is little endian. hence the first 32 bytes would be the low __m256i..., right?

Bottomline, I would volunteer to look into that but I would certainly need some guidance from a more experienced simd user...

I have started giving your benchmark a try on my skylake-avx512 (AVX2 implementation):

$ benchmark/dom/parse ../jsonexamples/twitter.json
number of iterations 200 
                                                     
../jsonexamples/twitter.json
============================
     9867 blocks -     631515 bytes - 55262 structurals (  8.8 %)
special blocks with: utf8      2284 ( 23.1 %) - escape       598 (  6.1 %) - 0 structurals      1287 ( 13.0 %) - 1+ structurals      8581 ( 87.0 %) - 8+ structurals      3272 ( 33.2 %) - 16+ structurals         0 (  0.0 %)
special block flips: utf8      1104 ( 11.2 %) - escape       642 (  6.5 %) - 0 structurals       940 (  9.5 %) - 1+ structurals       940 (  9.5 %) - 8+ structurals      2593 ( 26.3 %) - 16+ structurals         0 (  0.0 %)

All Stages (excluding allocation)
|    Speed        :  27.7648 ns per block ( 94.09%) -   0.4339 ns per byte -   4.9579 ns per structural -   2.3049 GB/s
|- Stage 1
|    Speed        :  13.2574 ns per block ( 44.93%) -   0.2072 ns per byte -   2.3673 ns per structural -   4.8272 GB/s
|- Stage 2
|    Speed        :  14.4231 ns per block ( 48.88%) -   0.2254 ns per byte -   2.5755 ns per structural -   4.4371 GB/s

3649.9 documents parsed per second (best)

that would be my baseline to see if a new hypothetical skylake-avx512 is more performant

I have not checked the other tests in the project. Is there any implementation unittests around?
specifically to test simd8x64::compress() ?

[lemire]

I suppose that the regular skylake architecture would fit into the haswell category then

Yes. Skylake is served by the haswell kernel typically. We do not provide exhaustive lists of all CPUs and that's not even possible: it also depends on how the system is configured (i.e., you can have a skylake CPU with AVX disabled). We do give instructions on how to find out which kernel is used at runtime, however.

So if I get it right, because it is missing AVX-512_VBMI, AVX-512_VBMI2, it will not work. Correct?

Yes.

if it is doable but you did not have the time to work on that... creating a new implementation is maybe something that I can try to do...

We do invite contributions, but I am not going to enable AVX-512 on anything earlier than an Ice Lake system. If you need to have AVX-512 capabilities on another microarchitecture and you want to implement a private feature (disabled by default and undocumented), then I would accept that... but you'd likely be the sole user.

We are going to stand by this policy for users in general and I stand by it: « We will not be backporting AVX-512 to older CPUs. I recommend against using AVX-512 on Skylake-era processors. » Private features are ok though, we have a few of those to accommodate specific users.

Before you decide to run AVX-512 instructions on a Skylake-era processor, I suggest you read the following blog post I wrote back then:

• AVX-512: when and how to use these new instructions

[lemire]

Bottomline, I would volonteer to look into that but I would certainly need some guidance from a more experienced simd user...

My opinion is that working on an AVX-512 for old processors (Skylake) is a bad idea. I recommend against it, strongly. There are few reasons for my recommendation:

1. I'm skeptical that you could ever get good performance. Even if you got good performance on a microbenchmark, you have to account for the AVX-512 licensing which is a complex engineering issue. I did implement an AVX-512 back then and it did not beat the haswell kernel. It is possible that I did a poor job and that more work could prove me wrong, but I expect not.
2. Our haswell kernel on Skylake is well tuned and has been used in production for years. It works well and I recommend it.
3. Intel discontinued Skylake processors five years ago. It is several generations behind. I recommend you focus your efforts on the current generation of processors, or the previous one...

[lemire]

@lano1106 You may enjoy this blog post that I wrote today: JSON Parsing: Intel Sapphire Rapids versus AMD Zen 4

[lemire]

Yes... c6i, m6i... c7i, m7i... and c7a, m7a all supports AVX-512 quite well.

A c6i.large (Ice Lake) is 500$US/year on AWS currently.

If I could squeeze few cycles left and right by making AVX512 work, that would certainly be beneficial...

As you no doubt understand by now, I don't think that's possible. I mean, you could certainly find room for optimization in simdjson, that's always possible, but I don't think that getting the icelake kernel to run on an old CPU is the way to go.

[lano1106]

ok Daniel, you made me realize something... I have no idea why the pricing looks like that...

c5.large linux on demand pricing: 0.096/h
c6i.large linux on demand pricing:0.0912 USD per Hour
c7i.large linux on demand pricing: 0.09576/h

you are making my day!

I see no reason why someone would choose c5 over c7i unless he is not aware of the pricing like I was... I did not even bother to check. To me, it was obvious that c7 would be more expensive than c5... this is a bit weird... It feels almost like an error from Amazon...

btw, I just noticed that you are located in Montreal. We are neighbors! If you ever partipate in some public local event on a topic relevant to your work, let me know, I could certainly stop by and say hello!

[lemire]

@lano1106 That's correct. Good AVX-512 support in 2024 is not expensive or exotic.

Note that if you run a business, you should not pay the on demand price.

[lano1106]

AWS is a new thing to me... I run a small one man business...

I was aware that it was possible to get a discount in exchange of a commitment to run a machine for a long time... I relocate my client to be as close to the service that it is connected with...I think that in the last 18 months, they must have moved something like 4 times...

I have moved my client in San Jose, Seattle, Los Angeles and now it is in Dublin!

This is perfect for me to not commit for a year and being able to pull the plug at any time without any penalty... but if you know a way to save money on AWS machines, I am all ears...

FYI, I have just completed migration to a c7i.large instance... this simple objective became challenging for no apparent reason... I was not able to connect my vps when hosted on a c7i.large...

I could run it on a c5 and a c6 instance but I could not connect to it when on c7i... After analyzing the journalctl, I have discovered that the kernel was giving a different name to the ethernet device. systemd network configuration was expecting the device to be named ens5 but on c7i, it got renamed to

kernel: ena 0000:27:00.0 enp39s0: renamed from eth0

I guess that playing with software wouldn't be fun without random issues every day...

[lano1106]

btw, I have tried the benchmark on my new c7i. The difference is stunning!

All Stages (excluding allocation)
|    Speed        :  13.3320 ns per block ( 91.45%) -   0.2083 ns per byte -   2.3807 ns per structural -   4.8002 GB/s
|    Cycles       :  50.8191 per block    ( 96.30%) -   0.7941 per byte    -   9.0746 per structural    -    3.812 GHz est. frequency
|    Instructions : 211.0176 per block    (100.00%) -   3.2973 per byte    -  37.6809 per structural    -    4.152 per cycle
|    Misses       :     562 branch misses ( 89.26%) - 0 cache misses (  -nan%) - 0.00 cache references