➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Modulefinder does not consider source file encoding
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Nicholas Feix, lucianamarques
Priority: normal Keywords: patch

Created on 2020-01-03 21:33 by Nicholas Feix, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 17817 open python-dev, 2020-01-04 01:49
Messages (3)
msg359259 - (view) Author: Nicholas Feix (Nicholas Feix) Date: 2020-01-03 21:33 
The modulefinder._find_module(...) function returns file objects text mode for source modules using the system encoding.
ModuleFinder.load_module(...) can run into decoding issues when the source file encoding does not match the system default.

The prior implementation imp.find_module(...) detected the encoding correctly using the tokenize.detect_encoding(...) function.
With the following code segment the detection would work again with UTF-8 BOM and PEP 263 type cookies.

    encoding = None
    if 'b' not in mode:
        with open(file_path, 'rb') as file:
            encoding = tokenize.detect_encoding(file.readline)[0]

    file = open(file_path, mode, encoding=encoding)
msg359267 - (view) Author: Luciana (lucianamarques) * Date: 2020-01-04 00:56 
I just wanted to comment here that I am looking into this.
msg359990 - (view) Author: Luciana (lucianamarques) * Date: 2020-01-14 20:32 
Hi there, can somebody who is a core dev please review my PR? Thanks :)
History
Date User Action Args
2022-04-11 14:59:24adminsetgithub: 83387
2020-01-14 20:32:51lucianamarquessetmessages: + msg359990
2020-01-04 01:49:59python-devsetkeywords: + patch
stage: patch review
pull_requests: + pull_request17243
2020-01-04 00:56:55lucianamarquessetnosy: + lucianamarques
messages: + msg359267
2020-01-03 21:33:55Nicholas Feixcreate