/cc @zichangg

Status update:
I build a local sdk, and find that the crash is because:

in secure_socket.dart, we got a null socket object.

  static Future<RawSecureSocket> secure(RawSocket socket,
      {StreamSubscription<RawSocketEvent> subscription,
      host,
      SecurityContext context,
      bool onBadCertificate(X509Certificate certificate),
      List<String> supportedProtocols}) {
     print('RawSecureSocket.secure socket = $socket');   
    **//crashed at the following line. socket == null**
    socket.readEventsEnabled = false;
    print(' socket.readEventsEnabled = false;');   
    socket.writeEventsEnabled = false;
    return _RawSecureSocket.connect(
        host != null ? host : socket.address.host, socket.port,
        is_server: false,
        socket: socket,
        subscription: subscription,
        context: context,
        onBadCertificate: onBadCertificate,
        supportedProtocols: supportedProtocols);
  }

I added some log to track why socket == null, it seems that after createProxyTunnel send CONNECT message, the connection is closed and socket is set to null.

_HttpClient
_HttpClient._openUrl, method = get, uri = https://www.google.com/
_HttpClient._getConnection, uriHost = www.google.com, uriPort = 443
_ConnectionTarget.connect www.google.com 443 127.0.0.1 45653 Instance of '_HttpClient'
_ConnectionTarget.connect isSecure && proxy.isDirect = false
_ConnectionTarget.connect connectionTask, return from startConnect
_ConnectionTarget socketFuture 1 times trying, socket = Instance of '_Socket'
before connection.createProxyTunnel
_HttpClientConnection.createProxyTunnel host = www.google.com port = 443 proxy.host = 127.0.0.1 proxy.port = 45653 false
_HttpClientConnection.send //www.google.com:443 443 CONNECT Instance of '_Proxy'
send()
_HttpClientRequest.close
_HttpClientRequest.get done _response = null
_HttpClientRequest.get return _response = Instance of 'Future'
_HttpClientRequest.get done _response = Instance of 'Future'
_HttpClientRequest.get return _response = Instance of 'Future'
send() _streamFuture send socket = Instance of '_Socket'
send() got response
send() incoming.dataDone incoming.upgraded = false 200
send() incoming.dataDone closing = false, _dispose = true, incoming.headers.persistentConnection = true true
send() incoming.dataDone - > destroy()
destroy()
_Socket.destroy
_Socket._closeRawSocket
createProxyTunnel, get response from tunnel request. connectionInfo = null 200 OK
createProxyTunnel, SecureSocket.secure. Instance of '_Socket'
SecureSocket.secure Socket socket = Instance of '_Socket'
_Socket.detachRaw
_Socket.detachRaw._detachReady.future. null
SecureSocket.secure detachedRaw = null
RawSecureSocket.secure socket = null
Unhandled exception:
NoSuchMethodError: The setter 'readEventsEnabled=' was called on null.
Receiver: null
Tried calling: readEventsEnabled=false
#0 _rootHandleUncaughtError. (dart:async/zone.dart:1112:29)
#1 _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#2 _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#3 _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:116:13)
#4 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:173:5)

socket should be closed at here:

http_impl.dart -> 
_HttpClientRequest send(Uri uri, int port, String method, _Proxy proxy) {
......
          if (!closing &&
              !_dispose &&
              incoming.headers.persistentConnection &&
              request.persistentConnection) {
            // Return connection, now we are done.
            _httpClient._returnConnection(this);
            _subscription.resume();
          } else {
            print('send() incoming.dataDone - > destroy()');
            ///////////////////////////
            destroy();
          }

Is this closing action corrent? should it keep a connection fot http proxy? please advise, thanks.
@zichangg

Thanks for posting! Let me assign myself first and take a look later.

Hi @zichangg,
I think I found the cause: Lantern's response has a Keep-Alive field but dart sdk did not parse it and closed the socket.

I set up a tinyproxy as a test proxy server, the HttpClient could use it as the http proxy tunnel.
I used tcpdump to compare the Http Response of tinyproxy vs. lantern:

tinyproxy:

HTTP/1.0 200 Connection established
Proxy-agent: tinyproxy/1.8.4

lantern:

HTTP/1.1 200 OK
Date: Wednesday, 14-Aug-19 16:13:22 CST
Keep-Alive: timeout=58
Content-Length: 0

I tracked the code that in http_parser.dart, _HttpParser._onData will process the response. And when received lanterns response, because of "Content-Length: 0", _HttpParser will close the socket, which will cause the above crash.

bool _headersEnd() {
......
if (_transferLength == 0 ||
(_messageType == _MessageType.RESPONSE && _noMessageBody)) {
_reset();
var tmp = _incoming;
// socket will closed here as "Content-Length: 0"
_closeIncoming();
_controller.add(tmp);
return false;
} else if (_chunked) {
_state = _State.CHUNK_SIZE;
_remainingContent = 0;
} else if (_transferLength > 0) {
_remainingContent = _transferLength;
_state = _State.BODY;
} else {
// tinyproxy will go to this branch. not closing socket
// Neither chunked nor content length. End of body
// indicated by close.
_state = _State.BODY;
}

currenttly, I add a _keepAlive flag to indicate that even content length == 0, it should not close the socket. and it works for me.
void _doParse() {
...
if (headerField == "keep-alive") {
_keepAlive = true;
}
...

if ((_transferLength == 0 && !_keepAlive)
...

Next, I am going to check the definition of "Keep-Alive: timeout=58", check if it need to translate into more details.
Thanks.

Thanks for the effort! But I'm not an expert on this area. The problem is parser does not handle this field correctly, considering both httpserver and httpclient both have a field called [idleTimeout]. Maybe RFC2616 can give more info on how to parse [keep-alive].

@mkustermann I saw you were the reviewer of original cl. Any thought?

@zichangg The c6e632b you refer to seems only related to be more lenient when parsing the headers. I don't think parsing is the problem here.

It seems that dart:io's proxy code is issuing a CONNECT and if it gets a response with content-length: 0 / keep-alive: ... it closes the connection where it shouldn't.

@sortie Could you take a look at this?

Status update:
Hi Dart team,
I re-checked the issue and found that when _HttpParser received a lantern's HttpRespond with empty body, it closed the HttpIncoming buffer, which I think it is reasonable behavior. But after that, the call back funtion located in _HttpClientConnection.send closed the socket, which is unconsiderable.

I think at this point, I should check if it is a "CONNECT"'s response and return directly if success.
So I modified the code, only change 1 line in http_impl.dart. And in this way, HttpClient will behave the same using tinyproxy or lantern.

Please check the patch atached.
Thanks.

0001-fix-crash-issue-when-HttpClient-using-lantern-proxy.patch.txt

Is there a solution?

@sortie It does not seem to be resolved, I still have this problem in version 2.8.1. The tag of 236dd991e540e26db22473998cc5354051464bf1 contains version 2.8.1

@zichangg What do you think?

I can confirm the problem does have. I'm able to have a repro locally. Work on the fix now.

Not only exists with Ubuntu, this is a bug not caused by any platform, but the code bug of Http Client

I've just started getting this issue in a Flutter project, due to an update to the Proxyman app on OSX.

My Dart/Flutter details:

 Flutter (Channel beta, 1.18.0-11.1.pre, on Mac OS X 10.15.4 19E287, locale en-AU)
    • Flutter version 1.18.0-11.1.pre at /Users/athor/dev/flutter
    • Framework revision 2738a1148b (5 weeks ago), 2020-05-13 15:24:36 -0700
    • Engine revision ef9215ceb2
    • Dart version 2.9.0 (build 2.9.0-8.2.beta)

Due to a bunch of issues on null safety, the review process is quite slow recently. It probably needs another 1 or 2 days to land. Also thanks to @lixiangthinker for diagnosing the problem.

still have this problem in Flutter 1.20.0-1.0.pre
fluttert --version:
Flutter 1.20.0-1.0.pre • channel dev • https://github.com/flutter/flutter.git
Framework • revision f73f498da1 (4 days ago) • 2020-06-18 08:23:22 -0700
Engine • revision c0d52b98d1
Tools • Dart 2.9.0 (build 2.9.0-16.0.dev 2b917f5)

@lyf571321556 "dart" you were using was still build 2.9.0-16.0.dev. The fix went in 2.9.0-19.0.dev.