Skip to content

May update broke publickey authentification #1617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Meivyn opened this issue Jun 2, 2020 · 23 comments
Closed

May update broke publickey authentification #1617

Meivyn opened this issue Jun 2, 2020 · 23 comments

Comments

@Meivyn
Copy link

Meivyn commented Jun 2, 2020

"OpenSSH for Windows" version
7.7.2.3

Server OperatingSystem
Windows 10 Pro

Client OperatingSystem
Arch Linux

What is failing

As in the title, I can't SSH into my Windows 10 machine using publickey authentification.

This is definitely an issue related to the new May 2004 update (19041.264). My authentification setup was working perfectly fine prior to upgrading my system.

Password authentification is not affected by this issue. I'm using ed25519 key, I didn't tested another format. I am not even prompted to enter my passphrase.

Looking at logs using verbose, I have this:

sshd: Connection from <client_ip> port <random_port> on <server_ip> port 22
sshd: Failed publickey for <client> from <client_ip> port <random_port> ssh2: ED25519 SHA256: <sha256>
sshd: Connection closed by authenticating user <client> <client_ip> port <random_port> [preauth]

I can't find anything useful using DEBUG3, I have only these lines after the authorized_keys part:

10852 2020-06-02 16:59:36.340 debug2: input_userauth_request: setting up authctxt for <client> [preauth]
10852 2020-06-02 16:59:36.340 debug3: mm_inform_authserv entering [preauth]
10852 2020-06-02 16:59:36.340 debug3: mm_request_send entering: type 4 [preauth]
10852 2020-06-02 16:59:36.340 debug2: input_userauth_request: try method none [preauth]
10852 2020-06-02 16:59:36.340 debug3: userauth_finish: failure partial=0 next methods="publickey" [preauth]
10852 2020-06-02 16:59:36.340 debug3: send packet: type 51 [preauth]
@bagajjal
Copy link
Collaborator

bagajjal commented Jun 3, 2020

@Meivyn - May update doesn't have new openssh binaries. I am guessing this is something to do with ssh-agent. Please check if ssh-agent is started on your machine.

There is a known bug that after windows update, ssh-agent is stopped so there is a manual step to start the ssh-agent service. We fixed this bug in the next windows release which is available during fall time.

@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

As far as I know, ssh-agent is used only on the client-side to retain keys? I never used it on Windows, nor on Linux. In fact, I choose to not use a passphrase to not bother me with it.

But to answer, "OpenSSH Authentification Agent" is disabled in services. I don't know if it was before the update. I don't think this is required either, at least on Linux servers. Windows have another behavior?

I managed to "fix" this by using administrators_authorized_keys with correct permissions, instead of ~/.ssh/authorized_keys. But this is still an issue in a way or another since this was working perfectly fine before that.

In fact, installing OpenSSH Server from Windows features directly was the only thing I had to do to set up the server. Then I deployed my key using scp from Linux.

@ssaavedra
Copy link

I am in this exact same situation.

What are the correct permissions for that file?
I cannot get it working either with ed25519 nor rsa keys, neither on ~/.ssh/authorized_keys nor on ProgramData/ssh/administrators_authorized_keys.

On an open ssh.exe -d it says:

debug1: KEX done [preauth]
debug1: userauth-request for user user service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: user  matched group list administrators at line 84
debug1: userauth-request for user user service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test pkalg ssh-ed25519 pkblob ED25519 SHA256:PE6XXXXXXXXXXXXXXXXXX [preauth]
debug1: trying public key file __PROGRAMDATA__/ssh/administrators_authorized_keys
Authentication refused.
Failed publickey for user from 192.168.103.2 port 57394 ssh2: ED25519 SHA256:PE6XXXXXXXXXXXXXXXXXX

@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

Using administrators_authorized_keys with loglevel=DEBUG3, SSH will tell you permissions aren't secure enough (before that Authentication refused.). I fixed this following the wiki: Security-protection-of-various-files-in-Win32-OpenSSH#administrators_authorized_keys

Sorry, something went wrong.

@ssaavedra
Copy link

I used ssh.exe -ddd to get that debug level and I saw the permissions problem.

Then I used:

$objSID = New-Object System.Security.Principal.SecurityIdentifier `
("ENTER-SID-HERE")
$objUser = $objSID.Translate( [System.Security.Principal.NTAccount])
$objUser.Value 

to understand which was the SID that was being complained about.

Thanks @Meivyn !

Sorry, something went wrong.

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 3, 2020

Glad you figure out the issue.

Default sshd_config expects the administrator keys be present in $env:programdata\ssh\administrators_authorized_keys file. This is added to fix a security issue and is expected.

I'm closing this issue.

@bagajjal bagajjal closed this as completed Jun 3, 2020
@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

In that case, why it was working in the first place? It stopped working only after the may update.

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 3, 2020

@Meivyn - I wonder that too. There is no version update in the May release.
Did you recently add the administrator keys to $env:programdata\ssh\administrators_authorized_keys file during the upgrade time? Is this file exist before the upgrade? If yes, what are the ACLs that you removed? Trying to understand if the ACLs are modified during the upgrade which is very very unlikely.

@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

I had to create this file and add my key manually only after this update, this file wasn't in ssh folder before that. I didn't modify anything, as I said, everything was already working fine by installing OpenSSH server from Windows optional features.

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 3, 2020

@Meivyn - Did you modify your $env:programdata\ssh\sshd_config file? The default config expects the admin keys to be present in $env:programdata\ssh\administrators_authorized_keys.

@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

I modified it in a way that allows only public key authentification. Nothing more than that.

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 3, 2020

so the below config exist in the $env:programdata\ssh\sshd_config file before the upgrade? If yes, then I doubt the issue should be present before the upgrade as well as there is no change to the sshd.exe binary.

Match Group administrators
AuthorizedKeysFile PROGRAMDATA/ssh/administrators_authorized_keys

@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

I didn't memorize everything in that file, so I can't answer that. I just know that it was working literally before the update as I used SSH right before, then it stopped working after the reboot. Based on the fact that these lines are in the file now, I'd say it was also there before the update since the file shouldn't have been modified by the update.

Something is going with this update, in a way or another in my opinion.
The exact same issue happened to someone else here and based on the date of the report, it coincides with the May update as well.

But if you tell me it wasn't supposed to work before the update, then the issue was before the update and it actually fixed it. I still have an ISO of Windows 1909, I could install it on a virtual machine and see if it's working with this version.

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 3, 2020

@Meivyn - May update shouldn't update anything related to binaries or the configuration.
On the VM without May update

  1. Check if the sshd_config has the config mentioned earlier which looks for admin keys in $env:programdata/ssh/administrators_authorized_keys
  2. Now try the key based authentication for an administrator account.
  3. Verify in the sshd.log, whether it reads the public key from $env:programdata/ssh/administrators_authorized_keys file or not.

@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

So after installing Windows on the VM:

  1. Yes, these lines are in the config
  2. I can log in using ~/.ssh/authorized_keys and administrators_authorized_keys doesn't exist
  3. Looks like I'm not recognized as an administrator on Windows 1909

Here is the full log:

7492 2020-06-03 19:35:35.499 debug2: fd 3 setting O_NONBLOCK
7492 2020-06-03 19:35:35.499 debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
7492 2020-06-03 19:35:35.499 debug1: Bind to port 22 on ::.
7492 2020-06-03 19:35:35.499 Server listening on :: port 22.
7492 2020-06-03 19:35:35.499 debug2: fd 4 setting O_NONBLOCK
7492 2020-06-03 19:35:35.499 debug1: Bind to port 22 on 0.0.0.0.
7492 2020-06-03 19:35:35.499 Server listening on 0.0.0.0 port 22.
7492 2020-06-03 19:35:40.733 debug3: fd 5 is not O_NONBLOCK
7492 2020-06-03 19:35:40.749 debug3: spawning "C:\\Windows\\System32\\OpenSSH\\sshd.exe" "-R"
7492 2020-06-03 19:35:40.749 debug3: send_rexec_state: entering fd = 8 config len 336
7492 2020-06-03 19:35:40.749 debug3: ssh_msg_send: type 0
7492 2020-06-03 19:35:40.749 debug3: send_rexec_state: done
4832 2020-06-03 19:35:40.811 debug1: inetd sockets after dupping: 3, 3
4832 2020-06-03 19:35:40.811 Connection from 192.168.122.1 port 51016 on 192.168.122.57 port 22
4832 2020-06-03 19:35:40.811 debug1: Client protocol version 2.0; client software version OpenSSH_8.3
4832 2020-06-03 19:35:40.811 debug1: match: OpenSSH_8.3 pat OpenSSH* compat 0x04000000
4832 2020-06-03 19:35:40.811 debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
4832 2020-06-03 19:35:40.811 debug2: fd 3 setting O_NONBLOCK
4832 2020-06-03 19:35:40.827 debug3: spawning "C:\\Windows\\System32\\OpenSSH\\sshd.exe" "-y"
4832 2020-06-03 19:35:40.827 debug2: Network child is on pid 1032
4832 2020-06-03 19:35:40.827 debug3: send_rexec_state: entering fd = 6 config len 336
4832 2020-06-03 19:35:40.827 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.827 debug3: send_rexec_state: done
4832 2020-06-03 19:35:40.827 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.827 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.827 debug3: preauth child monitor started
4832 2020-06-03 19:35:40.858 debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
4832 2020-06-03 19:35:40.858 debug3: send packet: type 20 [preauth]
4832 2020-06-03 19:35:40.858 debug1: SSH2_MSG_KEXINIT sent [preauth]
4832 2020-06-03 19:35:40.858 debug3: receive packet: type 20 [preauth]
4832 2020-06-03 19:35:40.858 debug1: SSH2_MSG_KEXINIT received [preauth]
4832 2020-06-03 19:35:40.858 debug2: local server KEXINIT proposal [preauth]
4832 2020-06-03 19:35:40.858 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth]
4832 2020-06-03 19:35:40.858 debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
4832 2020-06-03 19:35:40.858 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
4832 2020-06-03 19:35:40.858 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
4832 2020-06-03 19:35:40.858 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
4832 2020-06-03 19:35:40.858 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
4832 2020-06-03 19:35:40.858 debug2: compression ctos: none [preauth]
4832 2020-06-03 19:35:40.858 debug2: compression stoc: none [preauth]
4832 2020-06-03 19:35:40.858 debug2: languages ctos:  [preauth]
4832 2020-06-03 19:35:40.858 debug2: languages stoc:  [preauth]
4832 2020-06-03 19:35:40.858 debug2: first_kex_follows 0  [preauth]
4832 2020-06-03 19:35:40.858 debug2: reserved 0  [preauth]
4832 2020-06-03 19:35:40.858 debug2: peer client KEXINIT proposal [preauth]
4832 2020-06-03 19:35:40.858 debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c [preauth]
4832 2020-06-03 19:35:40.858 debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]
4832 2020-06-03 19:35:40.858 debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
4832 2020-06-03 19:35:40.858 debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
4832 2020-06-03 19:35:40.858 debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
4832 2020-06-03 19:35:40.858 debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
4832 2020-06-03 19:35:40.858 debug2: compression ctos: none,zlib@openssh.com,zlib [preauth]
4832 2020-06-03 19:35:40.858 debug2: compression stoc: none,zlib@openssh.com,zlib [preauth]
4832 2020-06-03 19:35:40.858 debug2: languages ctos:  [preauth]
4832 2020-06-03 19:35:40.858 debug2: languages stoc:  [preauth]
4832 2020-06-03 19:35:40.858 debug2: first_kex_follows 0  [preauth]
4832 2020-06-03 19:35:40.858 debug2: reserved 0  [preauth]
4832 2020-06-03 19:35:40.858 debug1: kex: algorithm: curve25519-sha256 [preauth]
4832 2020-06-03 19:35:40.858 debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
4832 2020-06-03 19:35:40.858 debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
4832 2020-06-03 19:35:40.858 debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
4832 2020-06-03 19:35:40.858 debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
4832 2020-06-03 19:35:40.858 debug3: receive packet: type 30 [preauth]
4832 2020-06-03 19:35:40.858 debug3: mm_key_sign entering [preauth]
4832 2020-06-03 19:35:40.858 debug3: mm_request_send entering: type 6 [preauth]
4832 2020-06-03 19:35:40.858 debug3: mm_key_sign: waiting for MONITOR_ANS_SIGN [preauth]
4832 2020-06-03 19:35:40.858 debug3: mm_request_receive_expect entering: type 7 [preauth]
4832 2020-06-03 19:35:40.858 debug3: mm_request_receive entering [preauth]
4832 2020-06-03 19:35:40.858 debug3: mm_request_receive entering
4832 2020-06-03 19:35:40.858 debug3: monitor_read: checking request 6
4832 2020-06-03 19:35:40.858 debug3: mm_answer_sign
4832 2020-06-03 19:35:40.874 debug3: mm_answer_sign: hostkey proof signature 00000174C4D7B060(100)
4832 2020-06-03 19:35:40.874 debug3: mm_request_send entering: type 7
4832 2020-06-03 19:35:40.874 debug2: monitor_read: 6 used once, disabling now
4832 2020-06-03 19:35:40.874 debug3: send packet: type 31 [preauth]
4832 2020-06-03 19:35:40.874 debug3: send packet: type 21 [preauth]
4832 2020-06-03 19:35:40.874 debug2: set_newkeys: mode 1 [preauth]
4832 2020-06-03 19:35:40.874 debug1: rekey after 134217728 blocks [preauth]
4832 2020-06-03 19:35:40.874 debug1: SSH2_MSG_NEWKEYS sent [preauth]
4832 2020-06-03 19:35:40.874 debug1: expecting SSH2_MSG_NEWKEYS [preauth]
4832 2020-06-03 19:35:40.874 debug3: send packet: type 7 [preauth]
4832 2020-06-03 19:35:40.874 debug3: receive packet: type 21 [preauth]
4832 2020-06-03 19:35:40.874 debug1: SSH2_MSG_NEWKEYS received [preauth]
4832 2020-06-03 19:35:40.874 debug2: set_newkeys: mode 0 [preauth]
4832 2020-06-03 19:35:40.874 debug1: rekey after 134217728 blocks [preauth]
4832 2020-06-03 19:35:40.874 debug1: KEX done [preauth]
4832 2020-06-03 19:35:40.921 debug3: receive packet: type 5 [preauth]
4832 2020-06-03 19:35:40.921 debug3: send packet: type 6 [preauth]
4832 2020-06-03 19:35:40.921 debug3: receive packet: type 50 [preauth]
4832 2020-06-03 19:35:40.921 debug1: userauth-request for user test service ssh-connection method none [preauth]
4832 2020-06-03 19:35:40.921 debug1: attempt 0 failures 0 [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_getpwnamallow entering [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_send entering: type 8 [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_receive_expect entering: type 9 [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_receive entering [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_receive entering
4832 2020-06-03 19:35:40.921 debug3: monitor_read: checking request 8
4832 2020-06-03 19:35:40.921 debug3: mm_answer_pwnamallow
4832 2020-06-03 19:35:40.921 debug2: parse_server_config: config reprocess config len 336
4832 2020-06-03 19:35:40.921 debug3: checking match for 'Group administrators' user test host 192.168.122.1 addr 192.168.122.1 laddr 192.168.122.57 lport 22
4832 2020-06-03 19:35:40.921 debug3: LsaLogonUser Succeeded (Impersonation: 0)
4832 2020-06-03 19:35:40.921 error: unable to resolve group administrators
4832 2020-06-03 19:35:40.921 debug1: user test does not match group list administrators at line 84
4832 2020-06-03 19:35:40.921 debug3: match not found
4832 2020-06-03 19:35:40.921 debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
4832 2020-06-03 19:35:40.921 debug3: mm_request_send entering: type 9
4832 2020-06-03 19:35:40.921 debug2: monitor_read: 8 used once, disabling now
4832 2020-06-03 19:35:40.921 debug2: input_userauth_request: setting up authctxt for test [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_inform_authserv entering [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_send entering: type 4 [preauth]
4832 2020-06-03 19:35:40.921 debug2: input_userauth_request: try method none [preauth]
4832 2020-06-03 19:35:40.921 debug3: userauth_finish: failure partial=0 next methods="publickey,keyboard-interactive" [preauth]
4832 2020-06-03 19:35:40.921 debug3: send packet: type 51 [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_receive entering
4832 2020-06-03 19:35:40.921 debug3: monitor_read: checking request 4
4832 2020-06-03 19:35:40.921 debug3: mm_answer_authserv: service=ssh-connection, style=
4832 2020-06-03 19:35:40.921 debug2: monitor_read: 4 used once, disabling now
4832 2020-06-03 19:35:40.921 debug3: receive packet: type 50 [preauth]
4832 2020-06-03 19:35:40.921 debug1: userauth-request for user test service ssh-connection method publickey [preauth]
4832 2020-06-03 19:35:40.921 debug1: attempt 1 failures 0 [preauth]
4832 2020-06-03 19:35:40.921 debug2: input_userauth_request: try method publickey [preauth]
4832 2020-06-03 19:35:40.921 debug1: userauth_pubkey: test pkalg ssh-ed25519 pkblob ED25519 SHA256:Bn2to7RgF5TTsUFB7Ims1qnJgRH4IFGALngXWCRpGnE [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_key_allowed entering [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_send entering: type 22 [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_receive_expect entering: type 23 [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_receive entering [preauth]
4832 2020-06-03 19:35:40.921 debug3: mm_request_receive entering
4832 2020-06-03 19:35:40.921 debug3: monitor_read: checking request 22
4832 2020-06-03 19:35:40.921 debug3: mm_answer_keyallowed entering
4832 2020-06-03 19:35:40.921 debug3: mm_answer_keyallowed: key_from_blob: 00000174C4D7B370
4832 2020-06-03 19:35:40.921 debug1: trying public key file C:\\Users\\Test\\.ssh/authorized_keys
4832 2020-06-03 19:35:40.921 debug1: C:\\Users\\Test\\.ssh/authorized_keys:1: matching key found: ED25519 SHA256:Bn2to7RgF5TTsUFB7Ims1qnJgRH4IFGALngXWCRpGnE
4832 2020-06-03 19:35:40.921 debug1: C:\\Users\\Test\\.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
4832 2020-06-03 19:35:40.921 Accepted key ED25519 SHA256:Bn2to7RgF5TTsUFB7Ims1qnJgRH4IFGALngXWCRpGnE found at C:\\Users\\Test\\.ssh/authorized_keys:1
4832 2020-06-03 19:35:40.921 debug3: mm_answer_keyallowed: publickey authentication test: ED25519 key is allowed
4832 2020-06-03 19:35:40.921 debug3: mm_request_send entering: type 23
4832 2020-06-03 19:35:40.921 debug3: send packet: type 60 [preauth]
4832 2020-06-03 19:35:40.921 debug2: userauth_pubkey: authenticated 0 pkalg ssh-ed25519 [preauth]
4832 2020-06-03 19:35:40.921 Postponed publickey for test from 192.168.122.1 port 51016 ssh2 [preauth]
4832 2020-06-03 19:35:40.936 debug3: receive packet: type 50 [preauth]
4832 2020-06-03 19:35:40.936 debug1: userauth-request for user test service ssh-connection method publickey [preauth]
4832 2020-06-03 19:35:40.936 debug1: attempt 2 failures 0 [preauth]
4832 2020-06-03 19:35:40.936 debug2: input_userauth_request: try method publickey [preauth]
4832 2020-06-03 19:35:40.936 debug3: userauth_pubkey: have ssh-ed25519 signature for ED25519 SHA256:Bn2to7RgF5TTsUFB7Ims1qnJgRH4IFGALngXWCRpGnE [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_key_allowed entering [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_send entering: type 22 [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive_expect entering: type 23 [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive entering [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive entering
4832 2020-06-03 19:35:40.936 debug3: monitor_read: checking request 22
4832 2020-06-03 19:35:40.936 debug3: mm_answer_keyallowed entering
4832 2020-06-03 19:35:40.936 debug3: mm_answer_keyallowed: key_from_blob: 00000174C4D7B060
4832 2020-06-03 19:35:40.936 debug1: trying public key file C:\\Users\\Test\\.ssh/authorized_keys
4832 2020-06-03 19:35:40.936 debug1: C:\\Users\\Test\\.ssh/authorized_keys:1: matching key found: ED25519 SHA256:Bn2to7RgF5TTsUFB7Ims1qnJgRH4IFGALngXWCRpGnE
4832 2020-06-03 19:35:40.936 debug1: C:\\Users\\Test\\.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
4832 2020-06-03 19:35:40.936 Accepted key ED25519 SHA256:Bn2to7RgF5TTsUFB7Ims1qnJgRH4IFGALngXWCRpGnE found at C:\\Users\\Test\\.ssh/authorized_keys:1
4832 2020-06-03 19:35:40.936 debug3: mm_answer_keyallowed: publickey authentication: ED25519 key is allowed
4832 2020-06-03 19:35:40.936 debug3: mm_request_send entering: type 23
4832 2020-06-03 19:35:40.936 debug3: mm_sshkey_verify entering [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_send entering: type 24 [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_sshkey_verify: waiting for MONITOR_ANS_KEYVERIFY [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive_expect entering: type 25 [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive entering [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive entering
4832 2020-06-03 19:35:40.936 debug3: monitor_read: checking request 24
4832 2020-06-03 19:35:40.936 debug3: mm_answer_keyverify: publickey 00000174C4D7B0D0 signature verified
4832 2020-06-03 19:35:40.936 debug1: auth_activate_options: setting new authentication options
4832 2020-06-03 19:35:40.936 debug3: mm_request_send entering: type 25
4832 2020-06-03 19:35:40.936 Accepted publickey for test from 192.168.122.1 port 51016 ssh2: ED25519 SHA256:Bn2to7RgF5TTsUFB7Ims1qnJgRH4IFGALngXWCRpGnE
4832 2020-06-03 19:35:40.936 debug1: monitor_child_preauth: test has been authenticated by privileged process
4832 2020-06-03 19:35:40.936 debug3: mm_get_keystate: Waiting for new keys
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive_expect entering: type 26
4832 2020-06-03 19:35:40.936 debug3: mm_request_receive entering
4832 2020-06-03 19:35:40.936 debug3: mm_get_keystate: GOT new keys
4832 2020-06-03 19:35:40.936 debug1: auth_activate_options: setting new authentication options [preauth]
4832 2020-06-03 19:35:40.936 debug2: userauth_pubkey: authenticated 1 pkalg ssh-ed25519 [preauth]
4832 2020-06-03 19:35:40.936 debug3: send packet: type 52 [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_request_send entering: type 26 [preauth]
4832 2020-06-03 19:35:40.936 debug3: mm_send_keystate: Finished sending state [preauth]
4832 2020-06-03 19:35:40.952 debug1: monitor_read_log: child log fd closed
4832 2020-06-03 19:35:40.952 debug3: LsaLogonUser Succeeded (Impersonation: 1)
4832 2020-06-03 19:35:40.952 debug3: spawning "C:\\Windows\\System32\\OpenSSH\\sshd.exe" "-z"
4832 2020-06-03 19:35:40.952 User child is on pid 6748
4832 2020-06-03 19:35:40.952 debug3: send_rexec_state: entering fd = 5 config len 336
4832 2020-06-03 19:35:40.952 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.952 debug3: send_rexec_state: done
4832 2020-06-03 19:35:40.952 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.952 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.952 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.952 debug3: ssh_msg_send: type 0
4832 2020-06-03 19:35:40.952 debug3: ssh_msg_send: type 0
6748 2020-06-03 19:35:40.968 debug3: ssh_msg_recv entering
6748 2020-06-03 19:35:40.968 debug2: fd 4 setting O_NONBLOCK
6748 2020-06-03 19:35:40.968 debug3: ssh_msg_recv entering
6748 2020-06-03 19:35:40.968 debug2: parse_server_config: config reprocess config len 336
6748 2020-06-03 19:35:40.968 debug3: checking match for 'Group administrators' user test host 192.168.122.1 addr 192.168.122.1 laddr 192.168.122.57 lport 22
6748 2020-06-03 19:35:40.983 debug3: get_user_token - i am running as test, returning process token
6748 2020-06-03 19:35:40.983 error: unable to resolve group administrators
6748 2020-06-03 19:35:40.983 debug1: user test does not match group list administrators at line 84
6748 2020-06-03 19:35:40.983 debug3: match not found
6748 2020-06-03 19:35:40.983 debug3: ssh_msg_recv entering
6748 2020-06-03 19:35:40.983 debug3: monitor_apply_keystate: packet_set_state
6748 2020-06-03 19:35:40.983 debug2: set_newkeys: mode 0
6748 2020-06-03 19:35:40.983 debug1: rekey after 134217728 blocks
6748 2020-06-03 19:35:40.983 debug2: set_newkeys: mode 1
6748 2020-06-03 19:35:40.983 debug1: rekey after 134217728 blocks
6748 2020-06-03 19:35:40.983 debug1: ssh_packet_set_postauth: called
6748 2020-06-03 19:35:40.983 debug3: ssh_packet_set_state: done
6748 2020-06-03 19:35:40.983 debug3: ssh_msg_recv entering
6748 2020-06-03 19:35:40.983 debug3: notify_hostkeys: key 0: ssh-rsa SHA256:TbwwRZss0Eq9M9urdcbTszGJLP3bPCQ/0E7HFcz/4Wk
6748 2020-06-03 19:35:40.983 debug3: notify_hostkeys: key 1: ecdsa-sha2-nistp256 SHA256:DY8T3ArzB2J2BKMKtZA/h8/iHHKsmNIyQIpkNY04trc
6748 2020-06-03 19:35:40.983 debug3: notify_hostkeys: key 2: ssh-ed25519 SHA256:zCf5QlCVXxLiJwO7VvD6YGKdqleGq280GZhI9RPk1IQ
6748 2020-06-03 19:35:40.983 debug3: notify_hostkeys: sent 3 hostkeys
6748 2020-06-03 19:35:40.983 debug3: send packet: type 80
6748 2020-06-03 19:35:40.983 debug1: active: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
6748 2020-06-03 19:35:40.983 debug1: Entering interactive session for SSH2.
6748 2020-06-03 19:35:40.983 debug2: fd 7 setting O_NONBLOCK
6748 2020-06-03 19:35:40.983 debug2: fd 8 setting O_NONBLOCK
6748 2020-06-03 19:35:40.983 debug1: server_init_dispatch
6748 2020-06-03 19:35:40.983 debug3: receive packet: type 90
6748 2020-06-03 19:35:40.983 debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
6748 2020-06-03 19:35:40.983 debug1: input_session_request
6748 2020-06-03 19:35:40.983 debug1: channel 0: new [server-session]
6748 2020-06-03 19:35:40.983 debug2: session_new: allocate (allocated 0 max 10)
6748 2020-06-03 19:35:40.983 debug3: session_unused: session id 0 unused
6748 2020-06-03 19:35:40.983 debug1: session_new: session 0
6748 2020-06-03 19:35:40.983 debug1: session_open: channel 0
6748 2020-06-03 19:35:40.983 debug1: session_open: session 0: link with channel 0
6748 2020-06-03 19:35:40.983 debug1: server_input_channel_open: confirm session
6748 2020-06-03 19:35:40.983 debug3: send packet: type 91
6748 2020-06-03 19:35:40.983 debug3: receive packet: type 80
6748 2020-06-03 19:35:40.983 debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
6748 2020-06-03 19:35:41.014 debug3: receive packet: type 98
6748 2020-06-03 19:35:41.014 debug1: server_input_channel_req: channel 0 request pty-req reply 1
6748 2020-06-03 19:35:41.014 debug1: session_by_channel: session 0 channel 0
6748 2020-06-03 19:35:41.014 debug1: session_input_channel_req: session 0 req pty-req
6748 2020-06-03 19:35:41.014 debug1: Allocating pty.
6748 2020-06-03 19:35:41.014 debug3: fd 9 is not O_NONBLOCK
6748 2020-06-03 19:35:41.014 debug3: fd 10 is not O_NONBLOCK
6748 2020-06-03 19:35:41.014 debug1: session_pty_req: session 0 alloc windows-pty
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 1 (0x1)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 2 (0x2)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 3 (0x3)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 4 (0x4)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 5 (0x5)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 6 (0x6)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 7 (0x7)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 8 (0x8)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 9 (0x9)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 10 (0xa)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 12 (0xc)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 13 (0xd)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 14 (0xe)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 18 (0x12)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 30 (0x1e)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 31 (0x1f)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 32 (0x20)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 33 (0x21)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 34 (0x22)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 35 (0x23)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 36 (0x24)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 37 (0x25)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 38 (0x26)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 39 (0x27)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 40 (0x28)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 41 (0x29)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 42 (0x2a)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 50 (0x32)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 51 (0x33)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 52 (0x34)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 53 (0x35)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 54 (0x36)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 55 (0x37)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 56 (0x38)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 57 (0x39)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 58 (0x3a)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 59 (0x3b)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 60 (0x3c)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 61 (0x3d)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 62 (0x3e)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 70 (0x46)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 71 (0x47)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 72 (0x48)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 73 (0x49)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 74 (0x4a)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 75 (0x4b)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 90 (0x5a)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 91 (0x5b)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 92 (0x5c)
6748 2020-06-03 19:35:41.014 debug1: Ignoring unsupported tty mode opcode 93 (0x5d)
6748 2020-06-03 19:35:41.014 debug3: send packet: type 99
6748 2020-06-03 19:35:41.014 debug3: receive packet: type 98
6748 2020-06-03 19:35:41.014 debug1: server_input_channel_req: channel 0 request shell reply 1
6748 2020-06-03 19:35:41.014 debug1: session_by_channel: session 0 channel 0
6748 2020-06-03 19:35:41.014 debug1: session_input_channel_req: session 0 req shell
6748 2020-06-03 19:35:41.014 Starting session: shell on windows-pty for test from 192.168.122.1 port 51016 id 0
6748 2020-06-03 19:35:41.014 debug2: fd 11 setting O_NONBLOCK
6748 2020-06-03 19:35:41.014 debug2: fd 12 setting O_NONBLOCK
6748 2020-06-03 19:35:41.030 debug2: fd 13 setting O_NONBLOCK
6748 2020-06-03 19:35:41.030 debug2: fd 14 setting O_NONBLOCK
6748 2020-06-03 19:35:41.030 debug2: fd 15 setting O_NONBLOCK
6748 2020-06-03 19:35:41.030 debug2: fd 16 setting O_NONBLOCK
6748 2020-06-03 19:35:41.030 debug1: Executing command: "c:\\windows\\system32\\cmd.exe" with pty
6748 2020-06-03 19:35:41.030 debug3: pty commandline: C:\\Windows\\system32\\conhost.exe --headless --width 210 --height 56 --signal 0x1dc -- "c:\\windows\\system32\\cmd.exe"
6748 2020-06-03 19:35:41.030 debug2: fd 4 setting TCP_NODELAY
6748 2020-06-03 19:35:41.030 debug2: channel 0: rfd 13 isatty
6748 2020-06-03 19:35:41.030 debug3: fd 13 is O_NONBLOCK
6748 2020-06-03 19:35:41.030 debug3: fd 12 is O_NONBLOCK
6748 2020-06-03 19:35:41.030 debug3: send packet: type 99

And full config:

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey __PROGRAMDATA__/ssh/ssh_host_rsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_dsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ecdsa_key
#HostKey __PROGRAMDATA__/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
SyslogFacility LOCAL0
LogLevel DEBUG3

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

PubkeyAuthentication yes

# The default is to check both .ssh/authorized_keys and .ssh/authorized_keys2
# but this is overridden so installations will only check .ssh/authorized_keys
AuthorizedKeysFile	.ssh/authorized_keys

#AuthorizedPrincipalsFile none

# For this to work you will also need host keys in %programData%/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#PermitTTY yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# override default of no subsystems
Subsystem	sftp	sftp-server.exe

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	AllowTcpForwarding no
#	PermitTTY no
#	ForceCommand cvs server

Match Group administrators
       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 3, 2020

What is the OpenSSH version before upgrade?

@Meivyn
Copy link
Author

Meivyn commented Jun 3, 2020

Looks like it's 7.7.2.2

This seems to be the issue:

4832 2020-06-03 19:35:40.921 error: unable to resolve group administrators
4832 2020-06-03 19:35:40.921 debug1: user test does not match group list administrators at line 84
4832 2020-06-03 19:35:40.921 debug3: match not found

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 4, 2020

@Meivyn, Couple more questions,

  1. v7.7.2.3 is for non-english versions of windows. Are you using non-english windows OS version?
  2. Did you run security updates before you migrate to the May 2004 update (19041.264)?
    I guess v7.7.2.3 is released as a security patch for non-english windows version. This doesn't require major update like May 2004 update (19041.264).
    Could you please confirm this.

I am not around when v7.7.2.3 is released so I don't have the full context how v7.7.2.3 is patched.

@Meivyn
Copy link
Author

Meivyn commented Jun 4, 2020

  1. I'm using French Canadian Windows version, same on the VM
  2. Nope, at least not manually

I could try migrating the VM to Windows 2004 so we can see the server version after the upgrade...
Also, I didn't update the VM yet, maybe with the updates I will be on 7.7.2.3 as well

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 4, 2020

Can you try to run windows update to get security patches installed. Note that this shouldn't be the update to May 2004 update (19041.264)?

I am expecting V7.7.2.3 is part of security patch. I just need confirmation.

@Meivyn
Copy link
Author

Meivyn commented Jun 4, 2020

After the updates, I am still on v7.7.2.2, and I'm not offered to migrate on Windows 2004 when I'm manually searching for updates.

@bagajjal
Copy link
Collaborator

bagajjal commented Jun 4, 2020

Thanks @Meivyn for helping to debug the issue.

All non-english windows OS version will have this issue when they update to May 2004 update (19041.264).

For future reference, this is expected behavior.

@Meivyn
Copy link
Author

Meivyn commented Jun 4, 2020

Just in case, I confirm the May update installs the version v.7.7.2.3 of the server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants