Indeed. If I replace the file with 
    f = io.BytesIO(open("tst", "rb").read())
timings are divided by 20...

After quick profiling, it seems that PyLong_New would benefit from a
free list. len(bytearray) is called very often.
To stay simple, it would be enough to only store longs of length 1 (<2**15).