[CRASH] Unable to start redis-server on Apple Silicon M1 as non-root #8062
Comments
|
@qwertywelch thank you for submitting this report. i'm guessing si_code of 2 means SEGV_ACCERR. can you please confirm (look for it in your the stack trace looks a bit odd, can you try to build redis with debug symbols and no optimizations ( |
|
@oranagra This perhaps what you're looking for? I'll try doing a debug build. |
|
ok, this stack trace looks much better |
|
It does succeed with sudo |
|
again, i suggest running it with i really don't see how a bug in redis can lead to segfault in fopen unless the filename is garbage, and if that was the case it would have happened with sudo too. |
|
If I understand correctly, this is a x86_64 binary running on M1 arm64 after being translated so theoretically could also be an issue with the translation process. Not very optimistic about strace helping in this case, it may require someone taking a step further in debugging this on the said platform. |
|
@Developer-Ecosystem-Engineering maybe you know anyone that can look into a possible problem with the translator. |
|
I also noticed this when trying to use Redis with homebrew on an M1 Mac. Homebrew reports that it started the server correctly, but it does not. Running For folks who might be finding this thread, a "workaround" in the meantime is to start the Redis server like this manually: |
Taking a look, might be a while given the time of year. |
|
Not reproducing this, working fine either way |
|
For me, it was happening on an 13" M1 MacBook pro. |
|
Same here - 13" M1 MBP. I'm seeing it fail as both root and user through |
|
I tried yesterday via arm version of homebrew and restarting the service works. However, I have the same problem two days ago. |
|
It has been reproduced, don't think it is related to our original PR at the moment but still working on it thanks. |
|
This appears to be a long standing bug in Redis. It can reproduce on Intel or Apple silicon based Macs and is unrelated to our prior contribution. When redis-server starts up, it takes note of where argv[0] is located and how far it, the other args, and env vars extend [1]. Later it overwrites that to customize what's shown in tools like ps [2]. There's an assumption in the implementation that argv[0], the other args, and env vars are all stored contiguously, one string after the other. So, it walks over each arg and env var keeping track of last address for all the strings. The trouble comes when one adds a new env var before the initialization start / end of args and env vars takes place. The new env var can be allocated in some other region, which breaks the assumption about all of them being together. If the new env var is allocated in a lower memory address, no problem its ignored. On the other hand, if the new env var is allocated in a higher memory address, it updates the notion of the end of the buffer. Later, when Redis goes to memset / bzero the buffer, the length calculation is way too large and written off the end of thread 0's stack. As for why sudo works around the issue, it changes the env variables! Why is it showing up now more often on Apple silicon? The stack for thread 0 on Apple silicon has a lot of open address space above. There was a similar issue diagnosed #8088 a few days ago [1] Line 182 in 2f41a38 [2] Line 251 in 2f41a38 |
|
@Developer-Ecosystem-Engineering thank you for this detailed analysis. |
|
Pushed a possible fix in #8150 but can someone who is able to reproduce this problem confirm that it is indeed solved? Thanks! |
|
@yossigo found this thread after I ran into this an hour ago w the new m1 macbook. Pulled your commit down, built all locally, and it's looking good over here. Nice work! |
|
@yossigo I was also able to build it using your fix on an m1 macbook using rosetta and run it as a non root. I noticed some failing tests though (but that is probably not related) and i was not able to build it without rosetta. Are there any plans on supporting m1 builds? |
|
Thanks @erf, Can you post more info about the build failures and failed tests please? |
|
@yossigo i was able to build Redis now both with and without Rosetta (by opening iTerm2 with the See attached make_test_log.txt file. |
|
@erf these are false test failures introduced in a new test of a recent commit, a fix was merged to |
Aha! That helped me find this simple workaround: (Yep, I just got a new M1 mac.) |
|
Thank you @anamba, super helpful. I hit this issue from the context of my brew service managed launchagent failing to start redis on boot (ie $ launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.redis@4.0.plist(I'm using the @4.0 formula here, but the default redis version should work similarly) Update ~/Library/LaunchAgents/homebrew.mxcl.redis@4.0.plist <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key>
<dict>
<key>SuccessfulExit</key>
<false/>
</dict>
<key>Label</key>
<string>homebrew.mxcl.redis@4.0</string>
<key>ProgramArguments</key>
<array>
+ <string>env -i</string>
<string>/usr/local/opt/redis@4.0/bin/redis-server</string>
<string>/usr/local/etc/redis.conf</string>
<string>--daemonize no</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>WorkingDirectory</key>
<string>/usr/local/var</string>
<key>StandardErrorPath</key>
<string>/usr/local/var/log/redis.log</string>
<key>StandardOutPath</key>
<string>/usr/local/var/log/redis.log</string>
</dict>
</plist>$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis@4.0.plistI contemplated opening a PR in brew for this, but hopefully this is a temporary bandaid until redis fix their bug. |
|
FYI, this bug is fixed in Redis 5.0.11, 6.0.10 and 6.2 RC2 |
|
@rahim: Your plist fix is super helpful. I think you should contribute it to brew :) |
|
same solution works fine. before: after: <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key>
<true/>
<key>Label</key>
<string>homebrew.mxcl.redis@4.0</string>
<key>LimitLoadToSessionType</key>
<array>
<string>Aqua</string>
<string>Background</string>
<string>LoginWindow</string>
<string>StandardIO</string>
<string>System</string>
</array>
<key>ProgramArguments</key>
<array>
<string>arch</string>
<string>-x86_64</string>
<string>/usr/local/opt/redis@4.0/bin/redis-server</string>
<string>/usr/local/etc/redis.conf</string>
<string>--daemonize</string>
<string>no</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>StandardErrorPath</key>
<string>/usr/local/var/log/redis.log</string>
<key>StandardOutPath</key>
<string>/usr/local/var/log/redis.log</string>
<key>WorkingDirectory</key>
<string>/usr/local/var</string>
</dict>
</plist>$ brew services list
Name Status User File
redis@4.0 started xxx ~/Library/LaunchAgents/homebrew.mxcl.redis@4.0.plist
|
Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
Aditional information
OS distribution and version: macOS Big Sur 11.0.1, MacBook Air M1 (8-core, 7-core GPU)
Steps to reproduce (if any)
arch -x86_64 brew install redis.redis-serverorarch -x86_64 redis-serveras a normal user.Runs without issue with
sudo.The text was updated successfully, but these errors were encountered: