Skip to content

bpo-32941: Add madvise() for mmap objects#6172

Merged
pitrou merged 7 commits intopython:masterfrom
ZackerySpytz:bpo-32941-mmap-madvise
May 27, 2019
Merged

bpo-32941: Add madvise() for mmap objects#6172
pitrou merged 7 commits intopython:masterfrom
ZackerySpytz:bpo-32941-mmap-madvise

Conversation

@ZackerySpytz
Copy link
Copy Markdown
Contributor

@ZackerySpytz ZackerySpytz commented Mar 21, 2018

Allow mmap objects to access the madvise() system call.

https://bugs.python.org/issue32941

Allow mmap objects to access the madvise() system call.
Comment thread Modules/mmapmodule.c Outdated
Comment thread Modules/mmapmodule.c
Comment thread Modules/mmapmodule.c Outdated
Comment thread Doc/library/mmap.rst
Copy link
Copy Markdown
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this! I left a couple comments that need to be addressed.

@bedevere-bot
Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be put in the comfy chair!

@mhvk
Copy link
Copy Markdown

mhvk commented Mar 22, 2019

Over at numpy/numpy#13172, it became clear that having this would be very useful for ~numpy.memmap (which wraps a mmap as an array). @ZackerySpytz - will you be able to implement requested changes? If not, I'll note in the numpy issue that it would be good for someone to take it over; it seems fairly straightforward.

@pitrou
Copy link
Copy Markdown
Member

pitrou commented Mar 22, 2019

Might also be useful for multiprocessing. @applio

Comment thread Modules/mmapmodule.c Outdated
Comment thread Doc/library/mmap.rst Outdated
@ZackerySpytz
Copy link
Copy Markdown
Contributor Author

I will make the requested changes.

@pitrou, thank you for the review, and I'm sorry for the enormous delay.

Allow lengths greater than self->size.
Add docs for MADV_* Constants.
@ZackerySpytz
Copy link
Copy Markdown
Contributor Author

I have made the requested changes; please review again.

I have also added a sentence to the docs explaining that start must be a multiple of the PAGESIZE.

@bedevere-bot
Copy link
Copy Markdown

Thanks for making the requested changes!

@pitrou: please review the changes made to this pull request.

Copy link
Copy Markdown
Member

@pitrou pitrou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! A few nits below, otherwise the PR looks good to me.

Comment thread Modules/mmapmodule.c Outdated
Comment thread Modules/mmapmodule.c
return NULL;
}
if (length <= 0) {
PyErr_SetString(PyExc_ValueError, "madvise length invalid");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here as well: perhaps "madvise length must be > 0"?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, Linux madvise() allows length to be 0 (according to its manpage).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I'm not sure why one would pass a length of zero. I think it's okay to change the check to length < 0, though.

Comment thread Modules/mmapmodule.c Outdated
@bedevere-bot
Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@pitrou
Copy link
Copy Markdown
Member

pitrou commented May 22, 2019

(also, please ensure you rebase or merge from master and fix conflicts)

@ZackerySpytz
Copy link
Copy Markdown
Contributor Author

Thanks, @pitrou.

I have made the requested changes; please review again.

@bedevere-bot
Copy link
Copy Markdown

Thanks for making the requested changes!

@pitrou: please review the changes made to this pull request.

@ZackerySpytz
Copy link
Copy Markdown
Contributor Author

@pitrou Ping. I would like to get this merged.

@pitrou
Copy link
Copy Markdown
Member

pitrou commented May 27, 2019

Yes, I'm taking a final look this evening.

@pitrou
Copy link
Copy Markdown
Member

pitrou commented May 27, 2019

Thank you @ZackerySpytz :-)

@bedevere-bot
Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64 Fedora 3.x has failed when building commit 02db696.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/53/builds/3027) and take a look at the build logs.
  4. Check if the failure is related to this commit (02db696) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/53/builds/3027

Click to see traceback logs
From https://github.com/python/cpython
 * branch            master     -> FETCH_HEAD
Reset branch 'master'

Objects/obmalloc.c:1376:1: warning: ‘no_sanitize_thread’ attribute directive ignored [-Wattributes]
 {
 ^

test_devpoll skipped -- test works only on Solaris OS family
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_kqueue skipped -- test works only on BSD
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_winreg skipped -- No module named 'winreg'
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_874f9b9b': [Errno 2] No such file or directory: '//psm_874f9b9b'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
test_winsound skipped -- No module named 'winsound'
test_msilib skipped -- No module named '_msi'
test_startfile skipped -- object <module 'os' from '/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/os.py'> has no attribute 'startfile'
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_around_2GB (test.test_mmap.LargeMmapTests) ... ok
test_around_4GB (test.test_mmap.LargeMmapTests) ... ok
test_large_filesize (test.test_mmap.LargeMmapTests) ... ok
test_large_offset (test.test_mmap.LargeMmapTests) ... ok
test_access_parameter (test.test_mmap.MmapTests) ... ok
test_anonymous (test.test_mmap.MmapTests) ... ok
test_bad_file_desc (test.test_mmap.MmapTests) ... ok
test_basic (test.test_mmap.MmapTests) ... ok
test_concat_repeat_exception (test.test_mmap.MmapTests) ... ok
test_context_manager (test.test_mmap.MmapTests) ... ok
test_context_manager_exception (test.test_mmap.MmapTests) ... ok
test_crasher_on_windows (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_double_close (test.test_mmap.MmapTests) ... ok
test_empty_file (test.test_mmap.MmapTests) ... ok
test_entire_file (test.test_mmap.MmapTests) ... ok
test_error (test.test_mmap.MmapTests) ... ok
test_extended_getslice (test.test_mmap.MmapTests) ... ok
test_extended_set_del_slice (test.test_mmap.MmapTests) ... ok
test_find_end (test.test_mmap.MmapTests) ... ok
test_flush_return_value (test.test_mmap.MmapTests) ... ok
test_invalid_descriptor (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_io_methods (test.test_mmap.MmapTests) ... ok
test_length_0_large_offset (test.test_mmap.MmapTests) ... ok
test_length_0_offset (test.test_mmap.MmapTests) ... ok
test_madvise (test.test_mmap.MmapTests) ... ERROR
test_move (test.test_mmap.MmapTests) ... ok
test_non_ascii_byte (test.test_mmap.MmapTests) ... ok
test_offset (test.test_mmap.MmapTests) ... ok
test_prot_readonly (test.test_mmap.MmapTests) ... ok
test_read_all (test.test_mmap.MmapTests) ... ok
test_read_invalid_arg (test.test_mmap.MmapTests) ... ok
test_resize_past_pos (test.test_mmap.MmapTests) ... ok
test_rfind (test.test_mmap.MmapTests) ... ok
test_sizeof (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_subclass (test.test_mmap.MmapTests) ... ok
test_tagname (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_tougher_find (test.test_mmap.MmapTests) ... ok
test_weakref (test.test_mmap.MmapTests) ... ok
test_write_returning_the_number_of_bytes_written (test.test_mmap.MmapTests) ... ok

======================================================================
ERROR: test_madvise (test.test_mmap.MmapTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/test/test_mmap.py", line 754, in test_madvise
    m.madvise(mmap.MADV_NORMAL, PAGESIZE, sys.maxsize)
ValueError: madvise start out of bounds

----------------------------------------------------------------------

Ran 39 tests in 0.802s

FAILED (errors=1, skipped=4)
test test_mmap failed
stty: standard input: Inappropriate ioctl for device
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_winconsoleio skipped -- test only relevant on win32
test_ioctl skipped -- Unable to open /dev/tty
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... ok
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

----------------------------------------------------------------------
Ran 24 tests in 6.757s

OK (skipped=2)
test test_mmap failed
make: *** [buildbottest] Error 2

@bedevere-bot
Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE Fedora 3.x has failed when building commit 02db696.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/85/builds/2848) and take a look at the build logs.
  4. Check if the failure is related to this commit (02db696) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/85/builds/2848

Click to see traceback logs
From https://github.com/python/cpython
 * branch            master     -> FETCH_HEAD
Reset branch 'master'

Objects/obmalloc.c:1376:1: warning: ‘no_sanitize_thread’ attribute directive ignored [-Wattributes]
 {
 ^

test_winreg skipped -- No module named 'winreg'
test_devpoll skipped -- test works only on Solaris OS family
test_msilib skipped -- No module named '_msi'
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_ioctl skipped -- Unable to open /dev/tty
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_101838b4': [Errno 2] No such file or directory: '//psm_101838b4'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
stty: standard input: Inappropriate ioctl for device
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_cb70aca7': [Errno 2] No such file or directory: '//psm_cb70aca7'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
test_kqueue skipped -- test works only on BSD
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: '//psm_ea9bd3e2': [Errno 2] No such file or directory: '//psm_ea9bd3e2'
  warnings.warn('resource_tracker: %r: %s' % (name, e))
test_around_2GB (test.test_mmap.LargeMmapTests) ... ok
test_around_4GB (test.test_mmap.LargeMmapTests) ... ok
test_large_filesize (test.test_mmap.LargeMmapTests) ... ok
test_large_offset (test.test_mmap.LargeMmapTests) ... ok
test_access_parameter (test.test_mmap.MmapTests) ... ok
test_anonymous (test.test_mmap.MmapTests) ... ok
test_bad_file_desc (test.test_mmap.MmapTests) ... ok
test_basic (test.test_mmap.MmapTests) ... ok
test_concat_repeat_exception (test.test_mmap.MmapTests) ... ok
test_context_manager (test.test_mmap.MmapTests) ... ok
test_context_manager_exception (test.test_mmap.MmapTests) ... ok
test_crasher_on_windows (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_double_close (test.test_mmap.MmapTests) ... ok
test_empty_file (test.test_mmap.MmapTests) ... ok
test_entire_file (test.test_mmap.MmapTests) ... ok
test_error (test.test_mmap.MmapTests) ... ok
test_extended_getslice (test.test_mmap.MmapTests) ... ok
test_extended_set_del_slice (test.test_mmap.MmapTests) ... ok
test_find_end (test.test_mmap.MmapTests) ... ok
test_flush_return_value (test.test_mmap.MmapTests) ... ok
test_invalid_descriptor (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_io_methods (test.test_mmap.MmapTests) ... ok
test_length_0_large_offset (test.test_mmap.MmapTests) ... ok
test_length_0_offset (test.test_mmap.MmapTests) ... ok
test_madvise (test.test_mmap.MmapTests) ... ERROR
test_move (test.test_mmap.MmapTests) ... ok
test_non_ascii_byte (test.test_mmap.MmapTests) ... ok
test_offset (test.test_mmap.MmapTests) ... ok
test_prot_readonly (test.test_mmap.MmapTests) ... ok
test_read_all (test.test_mmap.MmapTests) ... ok
test_read_invalid_arg (test.test_mmap.MmapTests) ... ok
test_resize_past_pos (test.test_mmap.MmapTests) ... ok
test_rfind (test.test_mmap.MmapTests) ... ok
test_sizeof (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_subclass (test.test_mmap.MmapTests) ... ok
test_tagname (test.test_mmap.MmapTests) ... skipped 'requires Windows'
test_tougher_find (test.test_mmap.MmapTests) ... ok
test_weakref (test.test_mmap.MmapTests) ... ok
test_write_returning_the_number_of_bytes_written (test.test_mmap.MmapTests) ... ok

======================================================================
ERROR: test_madvise (test.test_mmap.MmapTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/test/test_mmap.py", line 754, in test_madvise
    m.madvise(mmap.MADV_NORMAL, PAGESIZE, sys.maxsize)
ValueError: madvise start out of bounds

----------------------------------------------------------------------

Ran 39 tests in 1.052s

FAILED (errors=1, skipped=4)
test test_mmap failed
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_startfile skipped -- object <module 'os' from '/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64le/build/Lib/os.py'> has no attribute 'startfile'
test_winsound skipped -- No module named 'winsound'
test_flock (__main__.FNTLEINTRTest) ... ok
test_lockf (__main__.FNTLEINTRTest) ... ok
test_read (__main__.OSEINTRTest) ... ok
test_wait (__main__.OSEINTRTest) ... ok
test_wait3 (__main__.OSEINTRTest) ... ok
test_wait4 (__main__.OSEINTRTest) ... ok
test_waitpid (__main__.OSEINTRTest) ... ok
test_write (__main__.OSEINTRTest) ... ok
test_devpoll (__main__.SelectEINTRTest) ... skipped 'need select.devpoll'
test_epoll (__main__.SelectEINTRTest) ... ok
test_kqueue (__main__.SelectEINTRTest) ... skipped 'need select.kqueue'
test_poll (__main__.SelectEINTRTest) ... ok
test_select (__main__.SelectEINTRTest) ... ok
test_sigtimedwait (__main__.SignalEINTRTest) ... ok
test_sigwaitinfo (__main__.SignalEINTRTest) ... ok
test_accept (__main__.SocketEINTRTest) ... ok
test_open (__main__.SocketEINTRTest) ... ok
test_os_open (__main__.SocketEINTRTest) ... ok
test_recv (__main__.SocketEINTRTest) ... ok
test_recvmsg (__main__.SocketEINTRTest) ... ok
test_send (__main__.SocketEINTRTest) ... ok
test_sendall (__main__.SocketEINTRTest) ... ok
test_sendmsg (__main__.SocketEINTRTest) ... ok
test_sleep (__main__.TimeEINTRTest) ... ok

----------------------------------------------------------------------
Ran 24 tests in 7.082s

OK (skipped=2)
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_winconsoleio skipped -- test only relevant on win32
test test_mmap failed
make: *** [buildbottest] Error 2

@pitrou
Copy link
Copy Markdown
Member

pitrou commented May 27, 2019

Interesting. It seems PPC64 has a 64kB page size.

@vstinner What is the procedure to test a quick fix on a buildbot without going through Github? Edit: I found the answer.

@pitrou
Copy link
Copy Markdown
Member

pitrou commented May 27, 2019

Fix at #13596

DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
Allow mmap objects to access the madvise() system call.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants