Python HTTPS requests do not work while Fiddler is running

[IamMusavaRibica]

I tried running some requests in Python to see if they will appear in Fiddler

For a HTTP request, it worked with no problem:

However, trying to send a request over HTTPS results in long blocking and eventually fails. Nothing is logged to Fiddler, because no actual request was made. After 20-30 seconds, a long error is raised in Python:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: /?id=1234567890 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)'))))

What is causing Python to stop working (only while Fiddler is running)?  My current Fiddler version is v5.0.20204.45441 which is, I know, an old version. I trusted the root certificate, even installed it manually, and the HTTPS traffic from my browser is displayed normally as it should.

[Lini]

Hi,

I noticed that a similar was reported recently on the urllib3 issue tracker and that a fix was merged for 1.26.x about 10 days ago. The next update of the library should work in your case or display a more informative error message.

In any case, please check your environment variables and make sure that HTTP_PROXY and HTTPS_PROXY (or ALL_PROXY) are set to the same value - e.g. "http://127.0.0.1:8888" if you are using the default Fiddler settings. If the environment variables look OK or your proxy is configured in another way, you need to modify the settings there.

Alternatively, you can try setting your proxy manually with code before the tests request:

import requests
...
proxies = {
   'http': 'http://127.0.0.1:8888',
   'https': 'http://127.0.0.1:8888'
}

response = requests.post(url, proxies=proxies)

Regards,

Lini
Progress Telerik

[IamMusavaRibica]

Thanks for your response! However, I find this problem too generic, and even if I manage to send a request and it being captured, there would probably be countless other apps and software where requests would just be blocked

[Lini]

If an app supports using a proxy for network traffic, then Fiddler can capture it. Almost all UI applications will just work out of the box with the system proxy settings, so the only thing you need to do is turn on capturing in Fiddler. For console applications, most will work with the "HTTP_PROXY/HTTPS_PROXY/ALL_PROXY" environment variables. You just need to set these before you execute the shell command. Finally, only if you write your own network requests using some API, then you will need to take into account adding the Fiddler proxy settings manually.

[Hiroshi]

I got a same problem and arrived at this discussion.

What Lini said sounds like true and I found a workaround to install older version of urllib3.

pip install --upgrade urllib3==1.25.11

Lini,

I tried the latest urllib3 1.26.10 which was released on July 7, 2022, but the problem still persists. Can you tell me an exact post of urllib3 issue tracker? I was not able to find it.

[Nick Iliev]

Lini probably referenced this issue:

https://github.com/urllib3/urllib3/issues/2564 and the linked fix https://github.com/urllib3/urllib3/pull/2613 

[Hiroshi]

Thank you, Nick. I see that change is already included in 1.26.10, but I still cannot make HTTPS call via Fiddler HTTPS proxy. This might be a different issue (of urllib3).

[Hiroshi]

I tested the scenario today and it works! I have no idea why it is recovered now.

My test environment today:

• Windows 10 (21H2)
• Fiddler Classic v5.0.20211.51073 for .NET 4.6.1
• Python 3.7.5
  
• requests module 2.28.1
• urllib3 module 1.26.12 (latest) and also 1.26.10 (force downgraded)

In case anyone interested in, below is the exact code I tested. I used virtualenv to install only minimum modules (requests and its dependencies). I added verify=False in order to separate the discussion of SSL cert, which is unrelated to this issue.

import requests
url = 'https://cnn.com/'
proxies = {
   'http': 'http://127.0.0.1:8888',
   'https': 'http://127.0.0.1:8888'
}
response = requests.post(url, proxies=proxies, verify=False)
print(response)

[Noob Saibot]

It took me like whole day to figure everything out to be able to sniff Python apps with Fiddler. It was hard mostly because i don't have any background on programming.

1. Use following package to allow python to use windows certificates store
https://pypi.org/project/pip-system-certs/

2. Launch Fiddler and uncheck Capture - Very important because i think this is where the problem lies. Python just doesn't work while WinINET proxy is active. I was curious and updated my urllib3 to 1.26.12 because of the comments above mentioning fix . Still didn't work.

Though i still think problem lies in Python because Charles proxy/Python also doesn't work while proxy is active.

3. Open command prompt and set proxy environment variables. Don't close the window. We're going to use this window to launch python app.

set http_proxy=127.0.0.1:8888
set https_proxy=127.0.0.1:8888

4. Launch python app via the command prompt window above

C:\Users\Admin\TEMP\Python2ExeApp.exe

py C:\temp\pythonvirus.pyc

python C:\Users\Admin\TEMP\HelloWorld.py

etc