Conda support #108

mathieu1 · 2016-05-24T08:19:32Z

This is a reboot of #94 targetting the dev branch. Still WIP, discussion can start after some documentation is provided.

coveralls · 2016-05-24T08:21:52Z

Coverage decreased (-2.2%) to 74.816% when pulling ea78346 on mathieu1:conda_support into 1202044 on Miserlou:dev.

coveralls · 2016-05-26T10:11:17Z

Coverage decreased (-2.2%) to 74.847% when pulling 618aa8e on mathieu1:conda_support into 1202044 on Miserlou:dev.

coveralls · 2016-05-27T09:01:04Z

Coverage decreased (-2.3%) to 74.724% when pulling 2916e9e on mathieu1:conda_support into 1202044 on Miserlou:dev.

mathieu1 · 2016-05-27T11:05:58Z

Ok, the discussion can start. Here's the current status:

Implement condasupport, including full content of conda environment (not only python packages).
Make a conda package for zappa and for each of its dependencies (uploaded to my channel on anaconda.org). The current version of this branch is packaged under version 0.17.7.dev.
Create a repository (see mathieu1/zappa_conda) which serves both to test the current branch and can serve as an initial documentation for now, to help decide on the next steps.
Document how to use zappa with conda on other platforms than linux-64 (and discuss limitations, e.g. Windows doesn't have symlinks)
Work further on minifying conda packages. On this topic, available conda packages of numpy depend on large numerical optimization libraries (MKL, >100MB! or openblas, >10MB). This contributes to making it hard to fit other packages that depend on numpy such as pandas, scipy, scikit-learn, (although each of them is large on its own). A "noblas" version of numpy is underway, see Attempt at providing a noblas. conda-forge/numpy-feedstock#2

To start the discussion, here are a few questions/issues:

Given the advantages listed in Support conda environments #94 and the fact that it does not change any behavior for non-conda users, does this PR look viable for inclusion in Zappa? Should I explain/motivate further?
I am still unsure how to proceed with testing. The best I have in that direction is the above-mentioned repository, but it's hard to automate this testing from within zappa.

coveralls · 2016-05-27T16:46:01Z

Coverage decreased (-2.6%) to 74.45% when pulling 75b98f6 on mathieu1:conda_support into 1202044 on Miserlou:dev.

Miserlou · 2016-05-31T20:19:14Z

This is really cool progress! Sorry I haven't been very responsive on this thread.

mathom · 2016-07-06T17:02:57Z

zappa/cli.py

-                exclude=self.zappa_settings[self.api_stage].get('exclude', [])
+                exclude=self.zappa_settings[self.api_stage].get('exclude', []),
+                exclude_conda_packages = self.zappa_settings[self.api_stage].get('exclude_conda_packages',
+                    ['pip','python','readline','sqlite','wheel', 'boto3', 'botocore'])


I believe openssl and tk (among other things) are included with miniconda as well. You probably won't want them in your lambda function either.

mathom · 2016-07-06T17:09:24Z

As far as testing goes I'd recommend mocking and building out units vs trying to figure out how to make some AWS deployment for it.

Edit: testing packaging/unpackaging locally would probably be a good idea too.

mathom · 2016-07-06T17:15:32Z

It might be useful to warn the end user if there are large packages in their conda environment, too. The limits can be found here: http://docs.aws.amazon.com/lambda/latest/dg/limits.html . Note that there's a compressed as well as an uncompressed limitation.

smrtslckr · 2016-07-11T18:35:30Z

Tried out the dev branch and did some tinkering. Could not get conda support to work for me. I'm going to stick with elastic beanstalk for now. Definitely, be back in the future when I have more time.

sdementen · 2016-09-11T17:21:21Z

hello @mathieu1,
At the target, would the approach you are exploring/developing allow to develop within a conda environement on whatever Conda supported system (linux/windows/MacOS, 32/64 bit, ...) and then to package in a zip file the equivalent environment for the AWS image running behind AWS lambda ?
With the ability to choose "light" version to fit within AWS limits ?

brendancol · 2016-11-26T17:50:12Z

@mathieu1 @Miserlou this is super exciting.

coveralls · 2016-11-27T16:56:18Z

Coverage decreased (-1.1%) to 77.094% when pulling 35618b0 on mathieu1:conda_support into bee2601 on Miserlou:dev.

kalefranz · 2016-11-28T17:57:28Z

Hey everyone. I'm the lead developer on the conda project right now. Just wanted to chime in and say I think this is awesome. I'll try to spend more time with this in the coming days. If there are any specific questions or things you are stuck on, please ask! Tagging @msarahan here too, who's the tech lead of the conda-build project.

Miserlou · 2016-11-28T18:02:34Z

Cool! Hey @kalefranz and @msarahan ! Thanks for stopping by, I would love to have your support on this awesome but neglected PR.

Please also feel free to stop by the Slack channel for discussion: https://slack.zappa.io/

viksit · 2016-12-05T05:09:09Z

+1 here. I think the conda support for Zappa is going to be critical for those of us who want to deploy existing applications which rely on scientific computing libraries (numpy, tensorflow et al) which are already running in these environments. It's impossible to move them to a virtuaenv format given conda ships with platform compliant versions of the lower level C libraries that speed up their python wrappers.

Miserlou · 2016-12-05T05:17:43Z

Zappa does support hundreds of those libraries already, however conda support would certainly improve that number. See more here: https://blog.zappa.io/posts/zappa-adds-support-for-manylinux-wheels

yoavram · 2016-12-19T19:16:56Z

+1

hello-di · 2016-12-24T15:16:16Z

+1 for conda support!

postelrich · 2016-12-28T14:58:27Z

zappa/zappa.py

                quit()

+            if conda_mode:


Except for the precompiled_packages warning, why does this matter if its conda_mode or not?

It changes the way packaging works. In the standard virtualenv mode, zappa takes the site-packages folder of the virtualenv and possibly replaces some binaries using lambda-packages. In conda_mode zappa does a few things:

Create a temporary copy of the conda environment for the following manipulations

Remove excluded conda packages (like python itself)

Move the site-packages folder to the root of the conda environment

Zip up this moved site-packages along with all other remaining binaries in the conda environment.

postelrich · 2016-12-28T15:09:23Z

zappa/zappa.py

            if 'VIRTUAL_ENV' in os.environ:
                venv = os.environ['VIRTUAL_ENV']
+            elif 'CONDA_ENV_PATH' in os.environ:


envvar is now CONDA_PREFIX

may want to check both CONDA_ENV_PATH and CONDA_PREFIX while people have various versions of conda installed.

Thanks, this is fixed.

bearnshaw · 2017-01-04T23:47:42Z

+1 from me too. @Miserlou What needs to be done here to merge? Get the coverage to 100%?

coveralls · 2017-01-07T12:52:14Z

Coverage increased (+1.6%) to 79.786% when pulling fd85cca on mathieu1:conda_support into bee2601 on Miserlou:dev.

coveralls · 2017-01-07T12:57:38Z

Coverage increased (+1.6%) to 79.786% when pulling 03e8d95 on mathieu1:conda_support into bee2601 on Miserlou:dev.

ghost · 2019-07-02T14:18:22Z

+1 for Conda support

guilhermeprokisch · 2019-09-10T01:12:49Z

+1 Conda Support

mwos-sl · 2019-09-19T19:24:36Z

+1

nagmo · 2019-11-07T09:17:33Z

+1

isthisthat · 2019-12-17T19:32:10Z

+1

jhulten · 2020-02-21T19:02:50Z

Would love to see these merge conflicts resolved...

xzenggit · 2020-04-01T18:29:18Z

+1 for conda support

anibalsolon · 2020-04-03T18:08:02Z

+1 for conda support

willyraedy · 2020-04-30T17:08:02Z

+1 conda support

sc305495 · 2020-05-21T16:24:47Z

+1 for conda support

digitizdat · 2020-08-06T01:58:52Z

+1 conda support please

hannonq · 2020-09-28T16:56:01Z

+1 conda support

santiag0m · 2020-10-10T18:01:12Z

+1 conda support

andresinostroza13 · 2021-05-10T21:09:25Z

+1 conda support

alfredvijender · 2021-06-30T07:04:59Z

+1 conda support

MichelML · 2021-09-13T15:24:26Z

+1 conda support

gbossy · 2021-09-23T08:55:36Z

+1 conda support

salomonMuriel · 2021-11-12T21:49:56Z

Please conda support

TheFenrisLycaon

Seems okay to me. +1 for conda support !

gustavorps · 2022-10-03T17:03:31Z

+1

codeAshu · 2022-10-04T12:33:04Z

+1 for conda support

rudralabs · 2022-10-08T16:40:53Z

Using alias z="export VIRTUAL_ENV='${CONDA_PREFIX}'; zappa" as a workaround for now

LuizFDuarte · 2023-03-17T18:51:47Z

+1 for conda support

drius-22 · 2023-06-18T04:55:51Z

+100 conda support

KNehe · 2023-07-10T18:13:08Z

+1 conda support

monkut · 2023-07-12T01:18:56Z

I noticed there are recent comments on this PR.
For those interested, zappa moved to zappa/Zappa years ago, and this, Miserlou/Zappa is no longer maintained.

I'm not sure what "conda support" would require from zappa, but I've reopend the related issue here:
zappa/Zappa#43

If anyone is interested enough to rebase the required changes in this PR against the zappa/Zappa master branch, we can review and consider it for inclusion.

mathieu1 mentioned this pull request May 24, 2016

Support conda environments #94

Closed

7 tasks

mathieu1 force-pushed the conda_support branch from ea78346 to 618aa8e Compare May 26, 2016 10:09

Miserlou mentioned this pull request Jul 6, 2016

Conda Virtual Environments? #167

Open

mathom reviewed Jul 6, 2016
View reviewed changes

mathieu1 force-pushed the conda_support branch 2 times, most recently from 4047089 to 2707025 Compare November 27, 2016 16:29

postelrich reviewed Dec 28, 2016

View reviewed changes

mathieu1 force-pushed the conda_support branch from 35618b0 to fd85cca Compare January 7, 2017 12:45

shamir92 approved these changes Feb 1, 2020

View reviewed changes

JordanTreDaniel mentioned this pull request Jul 27, 2020

502 While Deploying (ModuleNotFound Errors) #2143

Open

This was referenced Feb 20, 2021

[Migrated] Conda support zappa/Zappa#43

Closed

[Migrated] 502 While Deploying (ModuleNotFound Errors) zappa/Zappa#887

Closed

TheFenrisLycaon reviewed Jan 26, 2022

View reviewed changes

Conda support #108

Are you sure you want to change the base?

Conda support #108

Conversation

mathieu1 commented May 24, 2016

coveralls commented May 24, 2016

coveralls commented May 26, 2016

coveralls commented May 27, 2016

mathieu1 commented May 27, 2016 • edited Loading

coveralls commented May 27, 2016

Miserlou commented May 31, 2016

mathom Jul 6, 2016

Choose a reason for hiding this comment

mathom commented Jul 6, 2016 • edited Loading

mathom commented Jul 6, 2016

smrtslckr commented Jul 11, 2016

sdementen commented Sep 11, 2016

brendancol commented Nov 26, 2016

coveralls commented Nov 27, 2016

kalefranz commented Nov 28, 2016

Miserlou commented Nov 28, 2016

viksit commented Dec 5, 2016

Miserlou commented Dec 5, 2016

yoavram commented Dec 19, 2016

hello-di commented Dec 24, 2016

postelrich Dec 28, 2016

Choose a reason for hiding this comment

mathieu1 Jan 8, 2017

Choose a reason for hiding this comment

postelrich Dec 28, 2016

Choose a reason for hiding this comment

bearnshaw Jan 4, 2017

Choose a reason for hiding this comment

mathieu1 Jan 8, 2017

Choose a reason for hiding this comment

bearnshaw commented Jan 4, 2017

coveralls commented Jan 7, 2017 • edited Loading

coveralls commented Jan 7, 2017 • edited Loading

ghost commented Jul 2, 2019

guilhermeprokisch commented Sep 10, 2019

mwos-sl commented Sep 19, 2019

nagmo commented Nov 7, 2019

isthisthat commented Dec 17, 2019

jhulten commented Feb 21, 2020

xzenggit commented Apr 1, 2020

anibalsolon commented Apr 3, 2020

willyraedy commented Apr 30, 2020

sc305495 commented May 21, 2020

digitizdat commented Aug 6, 2020

hannonq commented Sep 28, 2020

santiag0m commented Oct 10, 2020

andresinostroza13 commented May 10, 2021

alfredvijender commented Jun 30, 2021

MichelML commented Sep 13, 2021

gbossy commented Sep 23, 2021

salomonMuriel commented Nov 12, 2021

TheFenrisLycaon left a comment

Choose a reason for hiding this comment

gustavorps commented Oct 3, 2022

codeAshu commented Oct 4, 2022

rudralabs commented Oct 8, 2022

LuizFDuarte commented Mar 17, 2023

drius-22 commented Jun 18, 2023

KNehe commented Jul 10, 2023

monkut commented Jul 12, 2023

mathieu1 commented May 27, 2016 •

edited

Loading

mathom commented Jul 6, 2016 •

edited

Loading

coveralls commented Jan 7, 2017 •

edited

Loading

coveralls commented Jan 7, 2017 •

edited

Loading