Heads-up you may run into M1 specific issues

[lexidor]

I noticed your personal repo from facebook/hhvm#9397, hi 👋. Just a heads-up, you may run into some strange crashes/bugs when running hhvm on an M1 Mac. HHVM uses some AVX2 instructions that Apples Rosetta system does not support.

https://hhvm.com/blog/2022/06/17/deprecating-homebrew.html

We recommend users on Intel macs switch to using one of our docker images. We are unable to support M1 users due to the different architecture and limitations of Rosetta. Therefore, we must recommend use of an Intel linux host instead.

If you are using your docker image on a remote host in such a way that you can develop on your M1, but run on AMD64, that would work great.

I hope I have saved you a headache or two (or explained some crashes you've been having).

[justdan6]

Hey, thanks for the heads up!

I'm going to look into this more as it seems like there may be options...

1. Using colima (which uses a version of QEMU that supports AVX2 emulation) - Enabling AVX instructions on x86_64 arch VM with colima on M1 Mac  abiosoft/colima#315
2. Building HHVM with ENABLE_AVX2 turned off - https://github.com/search?q=repo%3Afacebook%2Fhhvm+avx2&type=code

[justdan6]

I'm able to build HHVM(at 6.26) and have most tests passing with colima and qemu 8.1.1 using this dockerfile on my m1 mbp - https://gist.github.com/justdan6/dde493f2c8ae36a4a00055228f818c5b . Without colima the build process always hangs.

From my very brief look I don't think any of the tests are failing because of AVX2 instructions. Any chance you have a code example that would normally crash?

root@9f90a3916849:/app/hhvm/build/hphp# cat /tmp/hphp-test-psn44z/test-failures 
test/slow/autoload/requireclass/requireclass.php
test/slow/coeffects/apc_pure.php
test/slow/compilation/open-rc-check.php
test/slow/debugger/modules.php
test/slow/decl-provider/embed_type_decl.php
test/slow/dv_array/ext_datetime.php
test/slow/dynamic-calls/watchman-callback-notice.php
test/slow/dynamic-calls/watchman-callback-throw.php
test/slow/dynamic-calls/watchman-callback.php
test/slow/ext_date/date_sunrise_test.php
test/slow/ext_date/date_sunset_test.php
test/slow/ext_datetime/date.php
test/slow/ext_datetime/date_sun_info.php
test/slow/ext_datetime/date_timezone.php
test/slow/ext_hsl/execve_execvpe.php
test/slow/ext_hsl/execve_execvpe_lightprocess.php
test/slow/ext_icu/date_pattern_gen.php
test/slow/ext_icu/uspoof.php
test/slow/ext_intl/numberformat_other.php
test/slow/ext_network/dns_get_record_caa.php
test/slow/ext_posix/ext_posix.php
test/slow/ext_process/no_stdin.php
test/slow/ext_socket/ext_socket.php
test/slow/ext_socket/persistent_socket.php
test/slow/ext_stream/ext_stream.php
test/slow/ext_string/htmlspecialchars.php
test/slow/ext_watchman/ext_watchman.php
test/slow/ext_zlib/zip_stream.php
test/slow/ext_zlib/ziparchive_setencryption.php
test/slow/facts/facts.php
test/slow/implicit-context/inaccessible-1.php
test/slow/implicit-context/memo-dynamic-1.php
test/slow/native-autoload-path-to-symbols/autoload_path_symbol_inverse.php
test/slow/native-autoload-path-to-symbols/autoload_path_to_symbols.php
test/slow/native-autoload-symlinked/native-autoload-symlinked.php
test/slow/native-autoload-xhp/mangle.php
test/slow/native-autoload-xhp/nomangle.php
test/slow/native-autoload/native-autoload.php
test/slow/native-autoload2/native-autoload.php
test/slow/program_functions/config_overrides.php
test/slow/program_functions/tier_overrides_default.php
test/slow/program_functions/tier_overrides_tier.php
test/slow/readonly_tdb/basic.php
test/slow/readonly_tdb/method_1.php
test/slow/readonly_tdb/scope_closure_1.php
test/slow/readonly_tdb/scope_for_3.php
test/slow/readonly_tdb/scope_if_1.php
test/slow/readonly_tdb/scope_if_2.php
test/slow/readonly_tdb/scope_if_3.php
test/slow/readonly_tdb/scope_switch_1.php
test/slow/readonly_tdb/scope_try_1.php
test/slow/readonly_tdb/scope_try_2.php
test/slow/readonly_tdb/self_method_1.php
test/slow/readonly_tdb/self_method_2.php
test/slow/readonly_tdb/static_method_1.php
test/slow/readonly_tdb/static_method_2.php
test/slow/readonly_tdb/static_prop_1.php
test/slow/types/autoload_alias/autoload_alias.php
test/slow/u_converter/2135.php
test/slow/xhp/xhpchild.php

[lexidor]

Hmmm, the AVX2 stuff mentioned applied to the JIT. I agree, these failures don't look like JIT specific failures to me.

What you could try is running hhvm/hhast. This library exercises the JIT very well. You'll have to make a small patch to vendor/facebook/hh-clilib in order to run hhast in from a web server context. The HH\global_get('argv') call needs a default value of vec['bin/hhast-lint.hack'].

You can then launch a server with hhvm -m server -p <port_number> -vServer.AllowRunAsRoot=1. You can cause hhast to lint itself by curling bin/hhast-lint.hack. Do so in a loop until the hhvm server prints that it is putting profiling data on a treadmill. The JIT will be in full force by then on a very heavy load.

Sorry for the large delay. I tried building an image from your Dockerfile, but this fails because of user error. (I wanted to see if the same failures would occur on x64.) I formatted my storage in a silly manner a long time ago. (A small root and a large /home partition.) Even after destroying all my containers, clearing log/tmp files, and doing a docker system purge -a I run out of disk space on the partition while building. I'll have to resize my partitions, which is doubly risky because I encrypted the drive.

I'll let you know once I have undergone this partition resize endeavour to validate your test results. This is something I'll do on the weekend. Partitions are something I only deal with once in a blue moon, and I don't want to push my luck and brick my system.

[lexidor]

Oh for hhast, build hhvm-4.168. That way you don't need to redo the hast codegen first.

[lexidor]

I took a little longer to redo my partitions. I hope you weren't in a hurry.

I built your docker image and I needed to do a couple things to make it work:

• run export HHVM_DISABLE_NUMA=1
• Edit /app/hhvm/hphp/test/run (add "/app/hhvm/build/hphp/hhvm/hhvm" to candidates)

I was then able to run the test suite.

• cd /app/hhvm
• ./hphp/test/run all

I get the following list of failures on x86:

...
Total time for all executed tests if run serially: 7016.08s
root@hexhexhexhex:/app/hhvm# cat /tmp/hphp-test-0y0Fwg/test-failures 
hphp/test/quick/chdir_posix.php
hphp/test/server/debugger/tests/runTest1.php
hphp/test/server/fastcgi/tests/authDigestTest.php
hphp/test/server/fastcgi/tests/disable_ini_zend_compat.php
hphp/test/server/fastcgi/tests/global_variables_server.php
hphp/test/server/fastcgi/tests/headerTest.php
hphp/test/server/fastcgi/tests/httpsTest.php
hphp/test/server/fastcgi/tests/invalid.php
hphp/test/server/fastcgi/tests/runTest1.php
hphp/test/server/http/tests/apache_proxygen_headers.php
hphp/test/server/http/tests/basic-multifile.php
hphp/test/server/http/tests/client_blocking_read_test.php
hphp/test/server/http/tests/codeCoverageTest.php
hphp/test/server/http/tests/get_headers_secure_dv_array.php
hphp/test/server/http/tests/globalDocumentTest.php
hphp/test/server/http/tests/method.php
hphp/test/server/http/tests/reify-multifile.php
hphp/test/server/http/tests/rename-multifile.php
hphp/test/server/http/tests/requesIdTest.php
hphp/test/server/http/tests/rqtrace.php
hphp/test/server/http/tests/runTest1.php
hphp/test/server/http/tests/staticContentHeaderTest.php
hphp/test/server/http/tests/staticContentTest.php
hphp/test/server/http/tests/takeoverTest.php
hphp/test/slow/autoload/requireclass/requireclass.php
hphp/test/slow/coeffects/apc_pure.php
hphp/test/slow/compilation/open-rc-check.php
hphp/test/slow/decl-provider/embed_type_decl.php
hphp/test/slow/dv_array/ext_datetime.php
hphp/test/slow/dynamic-calls/watchman-callback-notice.php
hphp/test/slow/dynamic-calls/watchman-callback-throw.php
hphp/test/slow/dynamic-calls/watchman-callback.php
hphp/test/slow/ext_date/date_sunrise_test.php
hphp/test/slow/ext_date/date_sunset_test.php
hphp/test/slow/ext_datetime/date.php
hphp/test/slow/ext_datetime/date_sun_info.php
hphp/test/slow/ext_datetime/date_timezone.php
hphp/test/slow/ext_icu/date_pattern_gen.php
hphp/test/slow/ext_icu/uspoof.php
hphp/test/slow/ext_intl/numberformat_other.php
hphp/test/slow/ext_posix/ext_posix.php
hphp/test/slow/ext_process/no_stdin.php
hphp/test/slow/ext_socket/ext_socket.php
hphp/test/slow/ext_socket/persistent_socket.php
hphp/test/slow/ext_stream/ext_stream.php
hphp/test/slow/ext_string/htmlspecialchars.php
hphp/test/slow/ext_watchman/ext_watchman.php
hphp/test/slow/ext_zlib/zip_stream.php
hphp/test/slow/ext_zlib/ziparchive_setencryption.php
hphp/test/slow/facts/facts.php
hphp/test/slow/implicit-context/inaccessible-1.php
hphp/test/slow/implicit-context/memo-dynamic-1.php
hphp/test/slow/included-enum-modify.php
hphp/test/slow/native-autoload-path-to-symbols/autoload_path_symbol_inverse.php
hphp/test/slow/native-autoload-path-to-symbols/autoload_path_to_symbols.php
hphp/test/slow/native-autoload-symlinked/native-autoload-symlinked.php
hphp/test/slow/native-autoload-xhp/mangle.php
hphp/test/slow/native-autoload-xhp/nomangle.php
hphp/test/slow/native-autoload/native-autoload.php
hphp/test/slow/native-autoload2/native-autoload.php
hphp/test/slow/program_functions/config_overrides.php
hphp/test/slow/program_functions/tier_overrides_default.php
hphp/test/slow/program_functions/tier_overrides_tier.php
hphp/test/slow/readonly_tdb/basic.php
hphp/test/slow/readonly_tdb/method_1.php
hphp/test/slow/readonly_tdb/scope_closure_1.php
hphp/test/slow/readonly_tdb/scope_for_3.php
hphp/test/slow/readonly_tdb/scope_if_1.php
hphp/test/slow/readonly_tdb/scope_if_2.php
hphp/test/slow/readonly_tdb/scope_if_3.php
hphp/test/slow/readonly_tdb/scope_switch_1.php
hphp/test/slow/readonly_tdb/scope_try_1.php
hphp/test/slow/readonly_tdb/scope_try_2.php
hphp/test/slow/readonly_tdb/self_method_1.php
hphp/test/slow/readonly_tdb/self_method_2.php
hphp/test/slow/readonly_tdb/static_method_1.php
hphp/test/slow/readonly_tdb/static_method_2.php
hphp/test/slow/readonly_tdb/static_prop_1.php
hphp/test/slow/types/autoload_alias/autoload_alias.php
hphp/test/slow/u_converter/2135.php
hphp/test/slow/xhp/xhpchild.php
hphp/test/zend/good/ext/date/tests/DateTimeZone_getTransitions_basic1.php
hphp/test/zend/good/ext/date/tests/bug33415-2.php
hphp/test/zend/good/ext/date/tests/bug35499.php
hphp/test/zend/good/ext/date/tests/bug41964.php
hphp/test/zend/good/ext/date/tests/bug45866.php
hphp/test/zend/good/ext/date/tests/bug51819.php
hphp/test/zend/good/ext/date/tests/bug67253.php
hphp/test/zend/good/ext/date/tests/date-time-modify-times.php
hphp/test/zend/good/ext/date/tests/date_parse_001.php
hphp/test/zend/good/ext/date/tests/date_sun_info_001.php
hphp/test/zend/good/ext/date/tests/date_sun_info_002.php
hphp/test/zend/good/ext/date/tests/date_sun_info_variation1.php
hphp/test/zend/good/ext/date/tests/date_sun_info_variation2.php
hphp/test/zend/good/ext/date/tests/date_sun_info_variation3.php
hphp/test/zend/good/ext/date/tests/date_sunrise_variation3.php
hphp/test/zend/good/ext/date/tests/date_sunrise_variation4.php
hphp/test/zend/good/ext/date/tests/date_sunrise_variation5.php
hphp/test/zend/good/ext/date/tests/date_sunrise_variation6.php
hphp/test/zend/good/ext/date/tests/date_sunrise_variation7.php
hphp/test/zend/good/ext/date/tests/date_sunrise_variation9.php
hphp/test/zend/good/ext/date/tests/date_sunset_variation3.php
hphp/test/zend/good/ext/date/tests/date_sunset_variation4.php
hphp/test/zend/good/ext/date/tests/date_sunset_variation5.php
hphp/test/zend/good/ext/date/tests/date_sunset_variation6.php
hphp/test/zend/good/ext/date/tests/date_sunset_variation7.php
hphp/test/zend/good/ext/date/tests/date_sunset_variation9.php
hphp/test/zend/good/ext/date/tests/rfc-datetime_and_daylight_saving_time-type3-bd1.php
hphp/test/zend/good/ext/date/tests/sunfuncts.php
hphp/test/zend/good/ext/date/tests/timezone_name_from_abbr_basic1.php
hphp/test/zend/good/ext/date/tests/timezone_transitions_get_basic1.php
hphp/test/zend/good/ext/intl/tests/breakiter_preceding_basic.php
hphp/test/zend/good/ext/intl/tests/formatter_format_currency2.php
hphp/test/zend/good/ext/intl/tests/formatter_get_locale_variant2.php
hphp/test/zend/good/ext/intl/tests/formatter_get_set_pattern.php
hphp/test/zend/good/ext/intl/tests/formatter_get_set_symbol2.php
hphp/test/zend/good/ext/intl/tests/formatter_get_set_text_attribute.php
hphp/test/zend/good/ext/intl/tests/grapheme.php
hphp/test/zend/good/ext/intl/tests/locale_get_display_language.php
hphp/test/zend/good/ext/intl/tests/locale_get_primary_language.php
hphp/test/zend/good/ext/intl/tests/msgfmt_millisecond_dates.php
hphp/test/zend/good/ext/intl/tests/spoofchecker_001.php
hphp/test/zend/good/ext/intl/tests/spoofchecker_004.php
hphp/test/zend/good/ext/intl/tests/uconverter_oop_callback.php
hphp/test/zend/good/ext/intl/uchar/tests/basic-functionality.php
hphp/test/zend/good/ext/mbstring/bug28220.php
hphp/test/zend/good/ext/mbstring/tests/mb_ereg_replace_variation1.php
hphp/test/zend/good/ext/sockets/tests/ipv6loop.php
hphp/test/zend/good/ext/standard/tests/file/unlink_variation1.php
hphp/test/zend/good/ext/standard/tests/strings/bug50052.php
hphp/test/zend/good/ext/zip/tests/bug53885.php

Any errors you see in your list that are also part of my list are not M1 processor specific.

[lexidor]

These are the tests that only fail on your machine:

• test/slow/debugger/modules.php
• test/slow/ext_hsl/execve_execvpe.php ¹
• test/slow/ext_hsl/execve_execvpe_lightprocess.php ¹
• test/slow/ext_network/dns_get_record_caa.php

So there is no jit AVX2 error in your tests.

Footnotes

1. There is an #if defined(__APPLE__) that might explain these failures (because I run on linux). ↩ ↩²

[justdan6]

No problem, not in a hurry at all. Thanks for running those tests :)

I've been having some trouble building 4.168 so I was planning on just pulling from whatever is in https://dl.hhvm.com/ubuntu and running shell_exec("/app/vendor/bin/hhast-lint.hack /app/vendor/bin/hhast-lint.hack") at an endpoint I'll spam.

[lexidor]

Oh, that will be more trouble. It will crash with an exception complaining the Hack AST (the version of the Hack syntax) does not match the hhvm version you are running on. I'll create a hhvm 6.26 compatible version of hhast for you instead.

[lexidor]

I have made a patched version of hhast available for this purpose. You can find it at https://github.com/lexidor/hhast-bundled. The README should hopefully be helpful enough. Don't be hesitant to ask for clarification if needed.

[justdan6]

This is awesome, thank you so much! I'll try to run it this weekend.

[justdan6]

Looks like it works!

1  | Failed to hugify the .text section
1  | BootStats: pagein_self done, took 93ms wall, 93ms cpu, 29 MB RSS
1  | BootStats: init_repo_file done, took 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: pthread_init done, took 5ms wall, 5ms cpu, 0 MB RSS
1  | BootStats: Process::InitProcessStatics done, took 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: timezone_init done, took 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: xenon done, took 83ms wall, 82ms cpu, 18 MB RSS
1  | BootStats: strobelight done, took 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: pcre_reinit done, took 4ms wall, 4ms cpu, 0 MB RSS
1  | BootStats: onig_init done, took 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: g_vmProcessInit done, took 3654ms wall, 3650ms cpu, 37 MB RSS
1  | BootStats: PageletServer::Restart done, took 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: XboxServer::Restart done, took 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: Stream::RegisterCoreWrappers done, took 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: ExtensionRegistry::moduleInit done, took 6702ms wall, 6700ms cpu, 13 MB RSS
1  | BootStats: extra_process_init done, took 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: rds::requestExit done, took 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: ExecutionContext done, took 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: extra_process_init_concurrent_wait done, took 1ms wall, 1ms cpu, 0 MB RSS
1  | WARNING: Logging before InitGoogleLogging() is written to STDERR
1  | I1020 00:01:08.362480     1 WorkerThread.cpp:81] Created WorkerThread 0x7f6e17175940, evb =  ProxygenWorker
1  | I1020 00:01:08.363715     1 WorkerThread.cpp:81] Created WorkerThread 0x7f6e17175780, evb =  ProxygenWorker
1  | BootStats: loading static content...
1  | BootStats: loading static content block done, took 0ms wall, 0ms cpu, 0 MB RSS
1  | Warming up
1  | BootStats: warmup done, took 10ms wall, 11ms cpu, 1 MB RSS
1  | page server started
1  | BootStats: servers started done, took 12ms wall, 16ms cpu, 2 MB RSS
1  | all servers started
1  | BootStats: all done, took 10584ms wall, 10581ms cpu, 100 MB RSS total
1  | BootStats: ExecutionContext = 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: ExtensionRegistry::moduleInit = 6702ms wall, 6700ms cpu, 13 MB RSS
1  | BootStats: PageletServer::Restart = 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: Process::InitProcessStatics = 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: Stream::RegisterCoreWrappers = 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: TOTAL = 10584ms wall, 10581ms cpu, 100 MB RSS
1  | BootStats: XboxServer::Restart = 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: extra_process_init = 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: extra_process_init_concurrent_wait = 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: g_vmProcessInit = 3654ms wall, 3650ms cpu, 37 MB RSS
1  | BootStats: init_repo_file = 1ms wall, 1ms cpu, 0 MB RSS
1  | BootStats: loading static content = 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: onig_init = 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: pagein_self = 93ms wall, 93ms cpu, 29 MB RSS
1  | BootStats: pcre_reinit = 4ms wall, 4ms cpu, 0 MB RSS
1  | BootStats: pthread_init = 5ms wall, 5ms cpu, 0 MB RSS
1  | BootStats: rds::requestExit = 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: servers started = 12ms wall, 16ms cpu, 2 MB RSS
1  | BootStats: strobelight = 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: timezone_init = 0ms wall, 0ms cpu, 0 MB RSS
1  | BootStats: warmup = 10ms wall, 11ms cpu, 1 MB RSS
1  | BootStats: xenon = 83ms wall, 82ms cpu, 18 MB RSS
1  | HardwareCounter: perf_event_open failed with: Operation not permitted
1  | HardwareCounter: perf_event_open failed with: Operation not permitted
1  | HardwareCounter: perf_event_open failed with: Operation not permitted
1  | \nFatal error: entire web request took longer than 200 seconds and timed out in /app/hhvm/build/hhast-bundled/src/nodes/Node.hack on line 27
1  | retranslateAll: scheduled after 214 seconds
1  | BootStats: RTA_create_callgraph...
1  | BootStats: RTA_create_callgraph block done, took 21ms wall, 24ms cpu, -1 MB RSS
1  | retranslateAll: finished clusterizing the functions
1  | BootStats: RTA_translate_and_relocate...
1  | retranslateAll: starting to relocate functions
1  | retranslateAll: finished optimizing and relocating functions
1  | BootStats: RTA_smash_opt_funcs...
1  | BootStats: RTA_smash_opt_funcs block done, took 6ms wall, 6ms cpu, 0 MB RSS
1  | retranslateAll: starting to publish functions
1  | BootStats: RTA_invalidate_prof_srckeys...
1  | BootStats: RTA_invalidate_prof_srckeys block done, took 295ms wall, 295ms cpu, 0 MB RSS
1  | BootStats: RTA_publish_meta...
1  | BootStats: RTA_publish_meta block done, took 156ms wall, 155ms cpu, -4 MB RSS
1  | BootStats: RTA_publish_code...
1  | BootStats: RTA_publish_code block done, took 67ms wall, 67ms cpu, 0 MB RSS
1  | BootStats: RTA_translate_and_relocate block done, took 23991ms wall, 70203ms cpu, -707 MB RSS
1  | retranslateAll: finished retranslating all optimized translations!
1  | Putting JIT ProfData on Treadmill
1  | Deleting JIT ProfData

[lexidor]

Amazing to hear! 🎉 🍰 🥇 🥳 🍾 Apple could have improved Rosseta2 since the post was made (it's a moving target after all). Nothing is ever 100% certain, but you just exercised the living bats out of the JIT. I am just going to say "It works, unless a counter example is found in the wild somewhere." This comes as a great and welcome surprise.

If you wouldn't mind, I'll link your findings back to the user group. I remember there being some folks who are on older hhvm versions and they are using Macs. If the post from the hhvm team was holding them back from upgrading, maybe this could give them renewed hope and they'll give upgrading another shot.

Looking forward to chatting with you again. Don't hesitate to ping me for anything docker or hhvm related.

[justdan6]

Definitely glad we were able to get this to work, thank you for all of your help!

I'll update the repo to have a step by step guide on getting this to work.

Looking forward to chatting with you again too :)