Skip to content

OpenMPI security paranoia about ptrace #3397

Closed
@claudevms

Description

@claudevms

Please fill out the below information:

  • Your Windows build number: (Type ver at a Windows Command Prompt)
    Windows 10 Pro
    Version: 1803
    OS Build 17134.18
    Ubuntu
    16.04.4 LTS
    xenial

  • What you're doing and what's happening: (Copy&paste specific commands and their output, or include screen shots)

Running LAMMPS software inside WSL at the bash command prompt
mike@Tak: mpirun -np 32 --hostfile hostfile /home/mike/bin/sxlmp_umpi -in pdms50_script.in

  • What's wrong / what should be happening instead:
    The following message is generated at the start of execution. This did not happen prior to the 1803 build

WARNING: Linux kernel CMA support was requested via the btl_vader_single_copy_mechanism MCA variable, but CMA support is not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy mechanism if one is available. This may result in lower performance.
Local host: Tak

What should happen:

  1. The above message shouldn't be displayed. Does this require building OpenMPI a different way or
    is there some security "feature" that needs to be disabled in WSL?
  2. There should be some easy to follow instructions to turn off the restrictive ptrace settings.
  3. I shouldn't have to settle for bad performance because some security guy is paranoid. Security should
    be opt-in not opt-out.

Possible rant:

Just a short note: Many of us with compute servers don't care about security. That's what "air gaps" are for. I have yet to find how Microsoft plans to distribute Sperctre and Meltdown to everyone whether they want it or not. Many of us need a tool to see if that crap was installed and hurting performance
and instructions to remove it if found.

By the way:

THANK YOU FOR WSL!!! This is the best way for me to build LAMMPS and run it on my servers and
analysis nodes. I get access to Linux and Windows software!!!

  • Strace of the failing command, if applicable: (If some_command is failing, then run strace -o some_command.strace -f some_command some_args, and link the contents of some_command.strace in a gist here).

  • For WSL launch issues, please collect detailed logs.

See our contributing instructions for assistance.

Activity

NablaCFD

NablaCFD commented on Aug 30, 2018

@NablaCFD

Microsoft Windows 10 Pro [Version 10.0.17134.228]
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
++++++++++++++++++++++++++++++++++++++++
I encounter the same problem.

type the following COMMAND in bash:
mpirun -np 3 test_openmpi

WARNING: Linux kernel CMA support was requested via the
btl_vader_single_copy_mechanism MCA variable, but CMA support is
not available due to restrictive ptrace settings.

The vader shared memory BTL will fall back on another single-copy
mechanism if one is available. This may result in lower performance.

Local host: workstation

[workstation:04636] [[3228,0],0] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file util/show_help.c at line 507
Hello: rank 1, world: 3
Hello: rank 0, world: 3
Hello: rank 2, world: 3
[workstation:04636] 1 more process has sent help message help-btl-vader.txt / cma-permission-denied
[workstation:04636] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

here is the file: test_openmpi.cpp
cat test_openmpi.cpp
/* Put this text inside hello.c file */
#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
int rank;
int world;

MPI_Init(NULL, NULL);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &world);
printf("Hello: rank %d, world: %d\n",rank, world);
MPI_Finalize();
}

I have tested it using the OpenMPI-3.1.1 and OpenMPI-3.1.2 compiled by myself, the above problem always happens.

Of course I can use the following option to turn off this warning, but I do not sure whether it will affect the performance. Thus, I think it should be solved by a developer from either WSL or OpenMPI.

mpirun --mca btl_vader_single_copy_mechanism none -np 3 test_openmpi

+++++++++++++++++++++++++++++++++++++++++
In addition:
I agree the above author's suggestion, in fact we care more about the performance than the safety. So maybe if Win10 can give the more switches/options to tune the performance and turn off the safety options if they will influent the performance.

This link may be useful to someone.
open-mpi/ompi#4948

Thanks in advance.
QL

Shweppie

Shweppie commented on Sep 1, 2018

@Shweppie

Same issue... Try the following:
https://groups.io/g/OpenHPC-users/topic/openmpi_and_shared_memory/16489081?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,16489081

$ echo 0 > /proc/sys/kernel/yama/ptrace_scope
or
$ sudo echo 0 > /proc/sys/kernel/yama/ptrace_scope
or
$ echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

claudevms

claudevms commented on Sep 3, 2018

@claudevms
Author
therealkenc

therealkenc commented on Sep 3, 2018

@therealkenc

Yeah 1 appears to be the default on unadulterated Ubuntu. Maybe change the issue title to "change default ptrace_scope to 1 if that's really the concrete ask we're talking about here. [Someone will suggest making it a wsl.conf thing, and then I'll feel sad inside.]

claudevms

claudevms commented on Sep 3, 2018

@claudevms
Author
KristianHolsheimer

KristianHolsheimer commented on Jan 23, 2020

@KristianHolsheimer

If you'd like this setting to be persistent, then you need to edit the config file /etc/sysctl.d/10-ptrace.conf instead, e.g.

$ sudo vim /etc/sysctl.d/10-ptrace.conf

and edit the line that says

kernel.yama.ptrace_scope = 1

to

kernel.yama.ptrace_scope = 0

N.B. This was taken from this answer on Stack Overflow.

therealkenc

therealkenc commented on Jan 23, 2020

@therealkenc

/etc/sysctl.d/10-ptrace.conf

Nothing reads that file on WSL (nothing starts systemd-sysctl.service), de-facto landing zone #994.

KristianHolsheimer

KristianHolsheimer commented on Jan 23, 2020

@KristianHolsheimer

Ah I see. Right never mind then

lokapal

lokapal commented on Feb 21, 2021

@lokapal

Well, how to set kernel.yama.ptrace_scope to 0 permanently? There is no possibility to set kernel variable in wsl.conf (at least this is not documented) and WSL just ignores rc.local and /etc/sysctl.d files....

therealkenc

therealkenc commented on Feb 22, 2021

@therealkenc

Test and set it in .bashrc or .profile is the de-facto standard operating procedure right now. Use sudoers to avoid the password prompt if you like.

yiqiangjizhang

yiqiangjizhang commented on Mar 1, 2021

@yiqiangjizhang

Yeah 1 appears to be the default on unadulterated Ubuntu. Maybe change the issue title to "change default ptrace_scope to 1 if that's really the concrete ask we're talking about here. [Someone will suggest making it a wsl.conf thing, and then I'll feel sad inside.]

How did you change it? I have the same issue but I can't manage to figure out how to set it to 0

yiqiangjizhang

yiqiangjizhang commented on Mar 1, 2021

@yiqiangjizhang

I will change it in wsl.conf to 0! That is what I need!!! Thank you!!! A value of 1 hurts OpenMPI performance

On Mon, Sep 3, 2018, 3:12 PM therealkenc @.***> wrote: Yeah 1 appears to be the default on unadulterated Ubuntu. Maybe change the issue title to "change default ptrace_scope to 1if that's really the concrete ask we're talking about here. [Someone will suggest making it awsl.conf` thing, and then I'll feel sad inside.] — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3397 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AdNyb267W1hMNBCcxFMpuuKbjtt_BFNJks5uXalYgaJpZM4VZwDE .

How did you change it? I have the same issue but I can't manage to figure out how to set it to 0

5 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @lokapal@therealkenc@KristianHolsheimer@NablaCFD@claudevms

        Issue actions

          OpenMPI security paranoia about ptrace · Issue #3397 · microsoft/WSL