Skip to content

python3Packages.s3transfer: enable tests#154280

Merged
fabaff merged 3 commits intoNixOS:masterfrom
fabaff:tests-s3transfer
Jan 11, 2022
Merged

python3Packages.s3transfer: enable tests#154280
fabaff merged 3 commits intoNixOS:masterfrom
fabaff:tests-s3transfer

Conversation

@fabaff
Copy link
Copy Markdown
Member

@fabaff fabaff commented Jan 10, 2022

Motivation for this change

Enable tests

Related #154264

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.05 Release Notes (or backporting 21.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

@github-actions github-actions Bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Jan 10, 2022
@ofborg ofborg Bot requested review from anhdle14, bhipple and davegallant January 10, 2022 15:19
@ofborg ofborg Bot added 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux. labels Jan 10, 2022
@fabaff fabaff marked this pull request as draft January 10, 2022 15:36
@fabaff
Copy link
Copy Markdown
Member Author

fabaff commented Jan 10, 2022

Result of nixpkgs-review pr 154280 run on x86_64-linux 1

1 package marked as broken and skipped:
  • mlflow-server
51 packages failed to build:
  • aws-sam-cli
  • awslimitchecker
  • home-assistant
  • lexicon
  • mapproxy
  • python310Packages.buildbot
  • python310Packages.buildbot-full
  • python310Packages.buildbot-ui
  • python310Packages.cartopy
  • python310Packages.celery
  • python310Packages.certbot-dns-route53
  • python310Packages.cloudsplaining
  • python310Packages.dependency-injector
  • python310Packages.django-raster
  • python310Packages.django-rq
  • python310Packages.djmail
  • python310Packages.fiona
  • python310Packages.flower
  • python310Packages.flowlogs_reader
  • python310Packages.gensim
  • python310Packages.geopandas
  • python310Packages.gradient
  • python310Packages.hass-nabucasa
  • python310Packages.mapbox
  • python310Packages.moto
  • python310Packages.osmnx
  • python310Packages.parsedmarc
  • python310Packages.pathy
  • python310Packages.pyhiveapi
  • python310Packages.sentry-sdk
  • python310Packages.serverlessrepo
  • python310Packages.slack-sdk
  • python310Packages.slicedimage
  • python310Packages.smart-open
  • python310Packages.snowflake-connector-python
  • python310Packages.snowflake-sqlalchemy
  • python310Packages.spacy
  • python310Packages.spacy-transformers
  • python310Packages.tensorboardx
  • python310Packages.textacy
  • python39Packages.cartopy
  • python39Packages.geopandas
  • python39Packages.osmnx
  • python39Packages.serverlessrepo
  • python39Packages.snowflake-connector-python
  • python39Packages.snowflake-sqlalchemy
  • python39Packages.spacy
  • python39Packages.spacy-transformers
  • python39Packages.textacy
  • sourcehut.metasrht
  • tts
102 packages built:
  • alerta-server
  • aws-google-auth
  • aws-mfa
  • awscli
  • awscli2
  • awslogs
  • barman
  • bash-my-aws
  • buildbot (python39Packages.buildbot)
  • buildbot-full (python39Packages.buildbot-full)
  • buildbot-ui (python39Packages.buildbot-ui)
  • certbot-full
  • checkov
  • cloud-custodian
  • credstash (python39Packages.credstash)
  • cura
  • deja-dup
  • duplicity
  • duply
  • dvc-with-remotes
  • elasticsearch-curator
  • git-remote-codecommit
  • gns3-gui
  • gns3-server
  • iceshelf
  • luigi
  • moodle-dl
  • moz-phab
  • nimbo
  • ntfy
  • octoprint
  • parquet-tools
  • python310Packages.aws-sam-translator
  • python310Packages.boto3
  • python310Packages.cfn-lint
  • python310Packages.credstash
  • python310Packages.django-anymail
  • python310Packages.ec2instanceconnectcli
  • python310Packages.fs-s3fs
  • python310Packages.mypy-boto3-builder
  • python310Packages.mypy-boto3-s3
  • python310Packages.pyathena
  • python310Packages.pycognito
  • python310Packages.rasterio
  • python310Packages.s3transfer
  • python310Packages.sagemaker
  • python310Packages.sapi-python-client
  • python310Packages.warrant
  • python39Packages.aws-sam-translator
  • python39Packages.boto3
  • python39Packages.celery
  • python39Packages.certbot-dns-route53
  • python39Packages.cfn-lint
  • python39Packages.cloudsplaining
  • python39Packages.dependency-injector
  • python39Packages.django-anymail
  • python39Packages.django-raster
  • python39Packages.django-rq
  • python39Packages.djmail
  • python39Packages.ec2instanceconnectcli
  • python39Packages.fiona
  • python39Packages.flower
  • python39Packages.flowlogs_reader
  • python39Packages.fs-s3fs
  • python39Packages.gensim
  • python39Packages.gradient
  • python39Packages.hass-nabucasa
  • python39Packages.mapbox
  • python39Packages.moto
  • python39Packages.mypy-boto3-builder
  • python39Packages.mypy-boto3-s3
  • python39Packages.parsedmarc
  • python39Packages.pathy
  • python39Packages.pyathena
  • python39Packages.pycognito
  • python39Packages.pyhiveapi
  • python39Packages.rasterio
  • python39Packages.s3transfer
  • python39Packages.sagemaker
  • python39Packages.sapi-python-client
  • python39Packages.sentry-sdk
  • python39Packages.slack-sdk
  • python39Packages.slicedimage
  • python39Packages.smart-open
  • python39Packages.tensorboardx
  • python39Packages.warrant
  • s3bro
  • s4cmd
  • snakemake
  • sourcehut.buildsrht
  • sourcehut.coresrht
  • sourcehut.dispatchsrht
  • sourcehut.gitsrht
  • sourcehut.hgsrht
  • sourcehut.hubsrht
  • sourcehut.listssrht
  • sourcehut.mansrht
  • sourcehut.pastesrht
  • sourcehut.todosrht
  • ssm-session-manager-plugin
  • streamlit
  • tribler

@fabaff fabaff marked this pull request as ready for review January 10, 2022 20:29
@fabaff fabaff merged commit 23afed2 into NixOS:master Jan 11, 2022
@fabaff fabaff deleted the tests-s3transfer branch January 11, 2022 08:30
@lamdor
Copy link
Copy Markdown
Contributor

lamdor commented Jan 12, 2022

@fabaff This seems to be failing for me on aarch64-darwin on MacOS. The test failure I'm getting is:

error: builder for '/nix/store/w5nwvfn71y90b2nm6cfgqv5q1kvwhpq9-python3.9-s3transfer-0.5.0.drv' failed with exit code 1;
       last 10 log lines:
       >     with self.assertRaisesRegexp(
       >
       > tests/unit/test_subscribers.py::TestSubscribers::test_not_callable_in_subclass_subscriber_method
       >   /private/tmp/nix-build-python3.9-s3transfer-0.5.0.drv-0/source/tests/unit/test_subscribers.py:81: DeprecationWarning: Please use assertRaisesRegex instead.
       >     with self.assertRaisesRegexp(
       >
       > -- Docs: https://docs.pytest.org/en/stable/warnings.html
       > =========================== short test summary info ============================
       > FAILED tests/unit/test_compat.py::TestBaseManager::test_can_provide_signal_handler_initializers_to_start
       > =========== 1 failed, 601 passed, 28 skipped, 44 warnings in 23.11s ============
       For full logs, run 'nix log /nix/store/w5nwvfn71y90b2nm6cfgqv5q1kvwhpq9-python3.9-s3transfer-0.5.0.drv'.

It fails for me consistently.

Here's the full build log s3transfer-build-log.txt.

Please let me know if I can help run a specific branch/test to help or if maybe tests/unit/test_compat.py needs added to disabledTestPaths? I'm sort of at a loss on where to start to fix it.

@duckpuppy
Copy link
Copy Markdown

duckpuppy commented Jan 13, 2022 •
edited
Loading

@fabaff I'm getting the error on x86_64-darwin as well. The actual test failure is below:

____ TestBaseManager.test_can_provide_signal_handler_initializers_to_start _____

self = <tests.unit.test_compat.TestBaseManager testMethod=test_can_provide_signal_handler_initializers_to_start>

    @skip_if_windows('os.kill() with SIGINT not supported on Windows')
    def test_can_provide_signal_handler_initializers_to_start(self):
        manager = self.create_pid_manager()
>       manager.start(signal.signal, (signal.SIGINT, signal.SIG_IGN))

tests/unit/test_compat.py:100:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/managers.py:554: in start
    self._process.start()
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/context.py:284: in _Popen
    return Popen(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <SpawnProcess name='PIDManager-43' parent=65299 initial>
file = <_io.BytesIO object at 0x1131d6630>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'TestBaseManager.create_pid_manager.<locals>.PIDManager'

/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/multiprocessing/reduction.py:60: AttributeError

@YorikSar
Copy link
Copy Markdown
Contributor

YorikSar commented Jan 18, 2022

I'm also affected by this.

@lamdor @duckpuppy Did you happen to open an issue about this?

YorikSar added a commit to YorikSar/dotfiles that referenced this pull request Jan 18, 2022
@YorikSar
Update flake inputs
ec535cb 
Pin nixpkgs to version before NixOS/nixpkgs#154280 was merged that
breaks awscli dependency.
@lamdor
Copy link
Copy Markdown
Contributor

lamdor commented Jan 18, 2022

@YorikSar I haven't yet had time to look more into it and create an issue. If someone beats me to it, that would be great.

@fabaff
Copy link
Copy Markdown
Member Author

fabaff commented Jan 18, 2022

It still fails (https://hydra.nixos.org/eval/1738910?filter=s3transfer&compare=1738843&full=#tabs-still-fail).

Looks like that a condition for tests is needed for Darwin.

@fabaff
Copy link
Copy Markdown
Member Author

fabaff commented Jan 18, 2022

Perhaps somebody from @NixOS/darwin-maintainers could help.

@thefloweringash
Copy link
Copy Markdown
Member

thefloweringash commented Jan 20, 2022

This passes for me on python 3.7, but fails on 3.8 and 3.9 (and probably any later version). The stack trace implicates multiprocessing. There was a change in python 3.8 that defaults multiprocessing to spawn instead of fork on macOS (bpo-33725, python/cpython#13603). I suspect the underlying issue here is that upstream tests aren't compatible with spawn multiprocessing, and pass on linux where the default is still fork.

@turion turion mentioned this pull request Jan 20, 2022
Merged
@lamdor lamdor mentioned this pull request Jan 20, 2022
Merged
13 tasks
@lamdor
Copy link
Copy Markdown
Contributor

lamdor commented Jan 20, 2022

I submitted a PR which ignores that test_compat on darwin. I've verified that then it is able to build s3transfer without any issue: #155896

@lamdor lamdor mentioned this pull request Jan 20, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FRidh FRidh Awaiting requested review from FRidh

@jonringer jonringer Awaiting requested review from jonringer

@davegallant davegallant Awaiting requested review from davegallant

@bhipple bhipple Awaiting requested review from bhipple

@anhdle14 anhdle14 Awaiting requested review from anhdle14

Assignees

No one assigned

Labels

6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 101-500 This PR causes between 101 and 500 packages to rebuild on Linux.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@fabaff @lamdor @duckpuppy @YorikSar @thefloweringash