Skip to content

Proxy Support over HTTPS requests #363

Closed
@frxncisjoseph

Description

@frxncisjoseph

Hello,

I'm currently a minor roadblock whilst using the fasthttp package.

I have done research into proxy support, however, I have only found one outstanding solution as of yet
This would be using the HostClient and setting the address (as shown in the code example below):

client = &fasthttp.HostClient{ Addr: "localhost:24000", }

However, when I attempt to make outgoing HTTPS requests, they do not work. Instead a non-HTTP request is sent/made. There appears to be no support for http.Transport either to set the proxy through this instance.

Is there any other way to set a proxy and make outgoing HTTPS (GET/POST/etc) requests using FastHTTP?

Activity

kirillDanshin

kirillDanshin commented on Aug 15, 2018

@kirillDanshin
Collaborator

hi @frxncisjoseph, welcome to the family :)

We can prepare an example for full-featured reverse proxy. I did one a couple of months ago for myself but it may take a while 'cause I need to prepare the sources for example purposes :)

frxncisjoseph

frxncisjoseph commented on Aug 15, 2018

@frxncisjoseph
Author

Hi @kirillDanshin, thank you for the greetings! I would absolutely love that a lot as I'm in the need of guidance, haha. I'd love to be able to make outgoing HTTPs requests under a proxy through .Get or other means.

erikdubbelboer

erikdubbelboer commented on Sep 1, 2018

@erikdubbelboer
Collaborator

@frxncisjoseph how would you implement this if you did have something like net.Transport? I see https://godoc.org/golang.org/x/net/proxy doesn't support HTTP proxies. For the net/http package it seems buildin.

I guess something like this might work? I don't have any HTTP proxy so I can't test it at the moment.

func FasthttpHTTPDialer(proxyAddr string) fasthttp.DialFunc {
  return func(addr string) (net.Conn, error) {
    conn, err := fasthttp.Dial(proxyAddr)
    if err != nil {
      return nil, err
    }

    req := "CONNECT " + addr + " HTTP/1.1\r\n"
    // req += "Proxy-Authorization: xxx\r\n"
    req += "\r\n" 

    if _, err := conn.Write([]byte(req)); err != nil {
      return nil, err
    }

    res := fasthttp.AcquireResponse()
    defer fasthttp.ReleaseResponse(res)

    res.SkipBody = true

    if err := res.Read(bufio.NewReader(conn)); err != nil {
      conn.Close()
      return nil, err
    }
    if res.Header.StatusCode() != 200 {
      conn.Close()
      return nil, fmt.Errorf("could not connect to proxy")
    }
    return conn, nil
  }
}

Which you can then use with:

	c := &fasthttp.Client{
		Dial: FasthttpHTTPDialer("localhost:9050"),
	}
frxncisjoseph

frxncisjoseph commented on Sep 6, 2018

@frxncisjoseph
Author

@erikdubbelboer That didn't work, sadly. :(

erikdubbelboer

erikdubbelboer commented on Sep 7, 2018

@erikdubbelboer
Collaborator

@frxncisjoseph I have modified my comment to a version that does work for me with https://tinyproxy.github.io/. Does it work for you now?

frxncisjoseph

frxncisjoseph commented on Oct 4, 2018

@frxncisjoseph
Author

Excellent work, bravo!

tsenart

tsenart commented on Mar 26, 2020

@tsenart
Contributor

@erikdubbelboer: As part of my effort to integrate this library with Vegeta, I found that the Client is missing any built-in support for proxying HTTP(S) requests. For the time being, I'll make it so vegeta uses the standard library client if users specify proxy options, but it'd be great to have this natively supported in fasthttp.

erikdubbelboer

erikdubbelboer commented on Mar 27, 2020

@erikdubbelboer
Collaborator

@tsenart the FasthttpHTTPDialer function I commented above doesn't work for you? If it does I guess we could add it to https://godoc.org/github.com/valyala/fasthttp/fasthttpproxy so everyone can use it more easy.

tsenart

tsenart commented on Mar 27, 2020

@tsenart
Contributor

@erikdubbelboer: I haven't tried it. I'm wondering if it's spec compliant? Does it support HTTPS proxying? Would like to see some tests in place for that code, rather than having it be somewhat fringe utility package.

erikdubbelboer

erikdubbelboer commented on Mar 30, 2020

@erikdubbelboer
Collaborator

To be honest I have no idea if it's spec compliant. As I said above I tested the code with https://tinyproxy.github.io/ and it worked. That's all I did. I'm afraid I currently don't have time to test this more and read the spec. You have tried using the code for your use case?

Fenny

Fenny commented on Apr 14, 2020

@Fenny
Contributor

@erikdubbelboer, I tested your example with multiple http/https proxies from multiple providers. All worked fine without errors 👍

It might be good to add an optional user:pass parser if exist in addr user:pass@proxy:port

func FasthttpHTTPDialer(proxy string) fasthttp.DialFunc {
	return func(addr string) (net.Conn, error) {
		var auth string

		if strings.Contains(proxy, "@") {
			split := strings.Split(proxy, "@")
			auth = base64.StdEncoding.EncodeToString([]byte(split[0]))
			proxy = split[1]

		}

		conn, err := fasthttp.Dial(proxy)
		if err != nil {
			return nil, err
		}

		req := "CONNECT " + addr + " HTTP/1.1\r\n"
		if auth != "" {
			req += "Proxy-Authorization: Basic " + auth + "\r\n"
		}
		req += "\r\n"

		if _, err := conn.Write([]byte(req)); err != nil {
			return nil, err
		}

		res := fasthttp.AcquireResponse()
		defer fasthttp.ReleaseResponse(res)

		res.SkipBody = true

		if err := res.Read(bufio.NewReader(conn)); err != nil {
			conn.Close()
			return nil, err
		}
		if res.Header.StatusCode() != 200 {
			conn.Close()
			return nil, fmt.Errorf("could not connect to proxy")
		}
		return conn, nil
	}
}
// usage
client := &fasthttp.Client{
	Dial: FasthttpHTTPDialer("john:doe@proxy.com:1337"),
}

req := fasthttp.AcquireRequest()
resp := fasthttp.AcquireResponse()

defer fasthttp.ReleaseRequest(req)
defer fasthttp.ReleaseResponse(resp)

req.SetRequestURI(url)

client.Do(req, resp)

body := string(resp.Body())
erikdubbelboer

erikdubbelboer commented on Apr 21, 2020

@erikdubbelboer
Collaborator

I have added the code to this repo. See: 54df169 Thanks!

SaishNaik

SaishNaik commented on Jul 18, 2020

@SaishNaik

@erikdubbelboer , #290 (comment) .
Following your suggestion here,

I used this to connect to proxy URL

c := &fasthttp.Client{
		Dial: fasthttpproxy.FasthttpHTTPDialer(proxyUrl)
}

My proxy is of the following form :
http://username-country-us:password@72.229.28.185:4055

Now when I use FasthttpHTTPDialer, It tries to connect with 72.229.28.185:4055 as proxy , not the whole proxy url I mentioned above.
If I edit FasthttpHTTPDialer to use the exact proxy, get this error again "error: too many colons in address".

Note: this same proxy url works well with standard http client :

proxyUrl,_ := url.Parse(proxy)
myClient := &http.Client{
                     Transport: &http.Transport{
                             Proxy: http.ProxyURL(proxyUrl)
                     }
             }

any suggestions?

erikdubbelboer

erikdubbelboer commented on Jul 18, 2020

@erikdubbelboer
Collaborator

That's not how authentication works for HTTP proxies. The username and password get turned into a Proxy-Authorization header that gets send with the CONNECT request. See:

req := "CONNECT " + addr + " HTTP/1.1\r\n"
if auth != "" {
req += "Proxy-Authorization: Basic " + auth + "\r\n"

What you are trying to do doesn't make any sense.

karthik-uj

karthik-uj commented on Oct 17, 2024

@karthik-uj

We can pass a CA certificate (if available for the proxy server) like so:

func FastHTTPDialerWithCert(proxy, caCertFile string) fasthttp.DialFunc {
	return func(addr string) (net.Conn, error) {
		var auth string

		// Load CA certificate
		caCert, err := os.ReadFile(caCertFile)
		if err != nil {
			return nil, fmt.Errorf("failed to read CA cert file: %v", err)
		}

		// Create a certificate pool from the CA
		caCertPool := x509.NewCertPool()
		if ok := caCertPool.AppendCertsFromPEM(caCert); !ok {
			return nil, fmt.Errorf("failed to append CA cert")
		}

		// Dial the proxy server
		conn, err := fasthttp.Dial(proxy)
		if err != nil {
			return nil, err
		}

		// Send CONNECT request to proxy
		req := "CONNECT " + addr + " HTTP/1.1\r\n"
		if auth != "" {
			req += "Proxy-Authorization: Basic " + auth + "\r\n"
		}
		req += "\r\n"

		if _, err := conn.Write([]byte(req)); err != nil {
			conn.Close()
			return nil, err
		}

		// Read response from the proxy
		res := fasthttp.AcquireResponse()
		defer fasthttp.ReleaseResponse(res)
		res.SkipBody = true

		if err := res.Read(bufio.NewReader(conn)); err != nil {
			conn.Close()
			return nil, err
		}

		if res.Header.StatusCode() != 200 {
			conn.Close()
			return nil, fmt.Errorf("could not connect to proxy: %s", res.String())
		}

		// Create a TLS connection with the CA cert pool
		tlsConn := tls.Client(conn, &tls.Config{
			RootCAs:            caCertPool,
			InsecureSkipVerify: false,
			ServerName:         strings.Split(addr, ":")[0],
			MinVersion:         tls.VersionTLS12, // Adjust based on the server
		})

		// Perform the TLS handshake
		if err := tlsConn.Handshake(); err != nil {
			tlsConn.Close()
			return nil, fmt.Errorf("TLS handshake failed: %v", err)
		}

		return tlsConn, nil
	}
}

You can also turn off SSL certificate verification using the InsecureSkipVerify option if a CA certificate is not available. Do not use that on production.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @tsenart@erikdubbelboer@kirillDanshin@SaishNaik@frxncisjoseph

      Issue actions

        Proxy Support over HTTPS requests · Issue #363 · valyala/fasthttp