Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

can't run on openjdk #174

Closed
1 task done
aaronwong1989 opened this issue Sep 28, 2018 · 12 comments
Closed
1 task done

can't run on openjdk #174

aaronwong1989 opened this issue Sep 28, 2018 · 12 comments
Labels
question-answered

Comments

@aaronwong1989
Copy link

aaronwong1989 commented Sep 28, 2018 •
edited

  • I have searched the issues of this repository and believe that this is not a duplicate.

Environment

  • Arthas version: 3.0.3
  • Operating System version: CentOS Linux release 7.5.1804 (Core)
  • Java version of target JVM: java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
  • Java version of JVM used to attach: java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64

Steps to reproduce this issue

[root@javawa ~]# ./as.sh 24110
Arthas script version: 3.0.4
Calculating attach execution time...
Attaching to 24110 using version 3.0.4...
Start arthas failed, exception stack trace:
java.lang.InternalError: instrument library is missing in target VM
	at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:105)
	at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:84)
	at com.taobao.arthas.core.Arthas.<init>(Arthas.java:25)
	at com.taobao.arthas.core.Arthas.main(Arthas.java:96)
Caused by: com.sun.tools.attach.AgentLoadException: Failed to load agent library
	at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:224)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:58)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:79)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
	... 3 more
attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.
[root@javawa ~]# cat /root/logs/arthas/arthas.log
cat: /root/logs/arthas/arthas.log: No such file or directory

Expected Result

Actual Result

What actually happens?

If there is an exception, please attach the exception trace:

Just put your stack trace here!
@hengyunabc
Copy link
Collaborator

Try to run bash -x as.sh, or try https://alibaba.github.io/arthas/en/install-detail.html#advanced-manual-boot

@aaronwong1989
Copy link
Author

i have change my jvm to oraclejdk1.8 and the problem still exists.
when run sh -x as.sh
i got

++ ps -p 24110 -o user=
+ local target_user=root
+ '[' root '!=' root ']'
+ echo 'Calculating attach execution time...'
Calculating attach execution time...
+ attach_jvm 3.0.4
+ local arthas_version=3.0.4
+ local arthas_lib_dir=/root/.arthas/lib/3.0.4/arthas
+ echo 'Attaching to 24110 using version 3.0.4...'
Attaching to 24110 using version 3.0.4...
+ '[' 127.0.0.1 = 127.0.0.1 ']'
+ [[ 3.0.4 > 3.0 ]]
+ /usr/lib/jvm/oracle-jdk-1.8.0_181/bin/java -Djava.awt.headless=true -Xbootclasspath/a:/usr/lib/jvm/oracle-jdk-1.8.0_181/lib/tools.jar -jar /root/.arthas/lib/3.0.4/arthas/arthas-core.jar -pid 24110 -target-ip 127.0.0.1 -telnet-port 3658 -http-port 8563 -core /root/.arthas/lib/3.0.4/arthas/arthas-core.jar -agent /root/.arthas/lib/3.0.4/arthas/arthas-agent.jar
Start arthas failed, exception stack trace:
java.lang.InternalError: instrument library is missing in target VM
	at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:105)
	at com.taobao.arthas.core.Arthas.attachAgent(Arthas.java:84)
	at com.taobao.arthas.core.Arthas.<init>(Arthas.java:25)
	at com.taobao.arthas.core.Arthas.main(Arthas.java:96)
Caused by: com.sun.tools.attach.AgentLoadException: Failed to load agent library
	at sun.tools.attach.LinuxVirtualMachine.execute(LinuxVirtualMachine.java:224)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:58)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:79)
	at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
	... 3 more
+ exit 1
+ '[' 1 -ne 0 ']'
+ exit_on_err 1 'attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.'
+ [[ ! -z attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions. ]]
+ echo 'attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.'
attach to target jvm (24110) failed, check /root/logs/arthas/arthas.log or stderr of target jvm for any exceptions.
+ exit 1

@aaronwong1989
Copy link
Author

@hengyunabc

[root@javawa arthas]# ll
total 20072
-rw-r--r-- 1 root root  9777184 Sep 28 16:26 arthas-3.0.4-bin.zip
-rw-r--r-- 1 root root     8015 Sep 14 18:52 arthas-agent.jar
-rw-r--r-- 1 root root   103660 Sep 14 18:52 arthas-client.jar
-rw-r--r-- 1 root root 10590270 Sep 14 18:52 arthas-core.jar
-rw-r--r-- 1 root root     2949 Sep 14 18:51 arthas-spy.jar
-rw-r--r-- 1 root root     2430 Sep 14 18:49 as.bat
-rwxr-xr-x 1 root root    14456 Sep 14 18:49 as.sh
-rwxr-xr-x 1 root root      633 Sep 14 18:49 install-local.sh
[root@javawa arthas]# pwd
/root/.arthas/lib/3.0.4/arthas

@hengyunabc
Copy link
Collaborator

@aaronwong1989 Try to run:

find /usr/lib/jvm/oracle-jdk-1.8.0_181 | grep 'instrument'

In my ubuntu, result is:

# find /xxx/jdk/jdk1.8.0_171/ | grep 'instrument'
/xxx/jdk/jdk1.8.0_171/jre/lib/amd64/libinstrument.so

@aaronwong1989
Copy link
Author

@hengyunabc
yes，libinstrument.so appears

@hengyunabc
Copy link
Collaborator

It seems that the target JVM can not load the instrument library.

  1. Check if the target JVM version >= 6 (not the JVM that starts arthas)
  2. Check if there is libinstrument.so under the target JVM (not the JVM that starts arthas)
  3. Try on other machines

@aaronwong1989
Copy link
Author

@hengyunabc
It works, thanks very much.

Problem appears when The target JVM is java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64， and The JVM that starts arthas is oracle-jdk-1.8.0_181 .

Initially, there was only one JVM( java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64) on my server, and arthas does not works. then I installed oracle-jdk-1.8.0_181 and set $JAVA_HOME to it, but I did not resatrt the target java process.

There are 2 java process on my server:

* [1]: 11901 /usr/lib/jenkins/jenkins.war
  [2]: 24110 ajax-srv-0.0.1.jar

after restart jenkins (do not restart ajax-srv-0.0.1.jar)，arthas can monitor Jenkins but can't works on process [2]. then I restart ajax-srv-0.0.1.jar, and it works !

the conclusion seems likes : arthas can't run on openjdk ，at least on my machine.

more information

find /usr/lib/jvm/oracle-jdk-1.8.0_181 |grep 'instrument'
/usr/lib/jvm/oracle-jdk-1.8.0_181/jre/lib/amd64/libinstrument.so

find /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64 |grep 'instrument'
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/lib/amd64/libinstrument.so

It seems that it has nothing to do with "libinstrument.so".

@aaronwong1989
Copy link
Author

Corrections ：Arthas can't monitor java processes that run on The openjdk

@hengyunabc
Copy link
Collaborator

@aaronwong1989 有没有比较确定的办法来重现这个问题，我本地开发用的就是 opentjdk，测试mac/linux opent jdk都没有问题。

@oakax
Copy link

oakax commented Oct 15, 2019

@hengyunabc 生产环境也是openjdk，第一次运行arthas也是这个问题，重启程序之后再启动arthas就能attach上了...

@hengyunabc hengyunabc mentioned this issue May 21, 2020
Closed
1 task
@jacksparrow414
Copy link

和 @oakax 一样，生成环境也是openjdk，第一次运行就会出现这个问题，重启应用程序在启动arthas就可以了

@xyh353875870
Copy link

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question-answered
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@hengyunabc @aaronwong1989 @oakax @xyh353875870 @jacksparrow414