On my FreeBSD 12.0-RELEASE-p10 VM, test_threading.test_recursion_limit() does crash with SIGSEGV and create a coredump.

vstinner@freebsd$ ./python -m test -v test_threading -m test_recursion_limit
== CPython 3.9.0a0 (heads/master:e0b6117e27, Aug 21 2019, 12:23:28) [Clang 6.0.1 (tags/RELEASE_601/final 335540)]
== FreeBSD-12.0-RELEASE-p10-amd64-64bit-ELF little-endian
== cwd: /usr/home/vstinner/python/master/build/test_python_3547
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
Run tests sequentially
0:00:01 load avg: 4.85 [1/1] test_threading
test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... FAIL

======================================================================
FAIL: test_recursion_limit (test.test_threading.ThreadingExceptionTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/vstinner/python/master/Lib/test/test_threading.py", line 1086, in test_recursion_limit
    self.assertEqual(p.returncode, 0, "Unexpected error: " + stderr.decode())
AssertionError: -11 != 0 : Unexpected error: 

----------------------------------------------------------------------

Ran 1 test in 6.017s

FAILED (failures=1)
Warning -- files was modified by test_threading
  Before: []
  After:  ['python.core'] 
test test_threading failed
test_threading failed

== Tests result: FAILURE ==

1 test failed:
    test_threading

Total duration: 7 sec 284 ms
Tests result: FAILURE

I used git bisect in the 3.8 branch and I found this change:

commit 8399641c34d8136c3151fda6461cc4727a20b28e
Author: Miss Islington (bot) <[email protected]>
Date:   Thu Aug 1 07:38:57 2019 -0700

    bpo-18049: Sync thread stack size to main thread size on macOS (GH-14748)
    
    
    This changeset increases the default size of the stack
    for threads on macOS to the size of the stack
    of the main thread and reenables the relevant
    recursion test.
    (cherry picked from commit 1a057bab0f18d6ad843ce321d1d77a4819497ae4)
    
    Co-authored-by: Ronald Oussoren <[email protected]>

Before this change, the test was skipped on FreeBSD:

...
test_recursion_limit (test.test_threading.ThreadingExceptionTests) ... skipped 'test macosx problem'
...

The crash start to occur with a Python callstack depth larger than 750. It doesn't crash with setrecursionlimit(750). See attached stack.py. Example:

vstinner@freebsd$ ./python stack.py 750 10240
setrecursionlimit(750)
stack_size: 10240.0 kiB = 10.0 MiB
end of main thread

It seems like calling _thread.stack_size(s) doesn't help:

vstinner@freebsd$ ./python stack.py 1000 10240
setrecursionlimit(1000)
stack_size: 10240.0 kiB = 10.0 MiB
Segmentation fault (core dumped)

--

I see different options:

* Reduce the Python recursion limit in the test
* Find a way to increase the default thread stack size on FreeBSD
* Skip the test on FreeBSD

On macOS, configure.ac uses a stack of 8 MiB:

                # Issue #18075: the default maximum stack size (8MBytes) is too
                # small for the default recursion limit. Increase the stack size
                # to ensure that tests don't crash
                # Note: This matches the value of THREAD_STACK_SIZE in
                # thread_pthread.h
                LINKFORSHARED="-Wl,-stack_size,1000000 $LINKFORSHARED"

Oh, my script called _thread.stack_size() to read the stack size, but that doesn't work: calling _thread.stack_size() sets the stack size to 0 again.

stack2.py is the fixed script. It seems like the test works with a stack of 8 MiB and the default recursion limit of 1000 Python frames:

vstinner@freebsd$ ./python stack.py 1000 4096
setrecursionlimit(1000)
stack_size: 4096.0 kiB = 4.0 MiB
Segmentation fault (core dumped)

vstinner@freebsd$ ./python stack.py 1000 8192
setrecursionlimit(1000)
stack_size: 8192.0 kiB = 8.0 MiB
end of main thread

So the problem is that the FreeBSD default thread stack size is too small.

I'd increase the default stack size for FreeBSD as well. AFAIK FreeBSD uses clang as the default compiler like macOS, and it is therefore likely that the two platforms have similar stack usage for similar code.

BTW. I can provide a PR for this but cannot easily test on FreeBSD.

On my FreeBSD 12.0 VM, /usr/bin/ld is a symbolic link to ld.ldd: it's the LLVM linker called "LLD":
https://lld.llvm.org/

I modified manually Makefile to give 16 MiB stack to the main thread using:

LINKFORSHARED=  -Wl,--export-dynamic "-Wl,-z" "-Wl,stack-size=0x1000000"

I added manually the following to Python/thread_pthread.h to give 16 MiB stack to thread:

#undef  THREAD_STACK_SIZE
#define THREAD_STACK_SIZE       0x1000000

Using that, the RecurisionError is raised at a depth of 1000 Python function calls: before we exhaust the C stack.

--

The syntax used in configure.ac doesn't work with LLD. I get such error:

cc -pthread     -Wl,--export-dynamic -Wl,-stack_size,1000000 -o Programs/_testembed Programs/_testembed.o libpython3.9d.a -lcrypt -ldl  -lutil -lm   -lm

/usr/bin/ld: error: cannot open 1000000: No such file or directory

See the android related issue #38852.