tl;dr: Use pip, git(hub) and tags to share packages privately amongst your team
Problem Statement We have libraries that we need to share across multiple applications. Using Git submodules, makes me cry — there has to be a better way (and there is)
I am going to break this up into three sections:
- Packaging Python (this needs to happen regardless of how you share)
- Sharing Publicly (with the world via PyPi)
- Sharing Privately (within your organization, with pip and git)
This section needs to happen regardless if you share in the public domain or privately
Python (our beloved language with bells and whistles) has
setuptools built in that we can use to package our utility.
I really do not having anything to say that hasn’t been said before. I found this tutorial to be excellent:
But the gist of it is:
- create a
setup.py(to define whats to be shared using
- Gotsta have README.rst (sharing code with documenting it is useless)
- Test it locally
- Use Semantic Versioning (or you are a bad person)
Sharing in the public domain
Okay, you wrote an amazing utility and want to share it with the world. I really was blown away how easy it was to do this with thanks to PyPi (Python Package Index) (aka the Cheeseshop)
For this example, I wrote a package called
human_dates That takes some of the syntax sugar from the ruby and rails world and brings it over to python land.
You can find the code here
And you will see the light if you continue reading the packaging tutorial from the first section
Once you have you package tested locally, its as simple as:
And now anyone can do the following:
And then in a python console, you can:
1 2 3 4 5 6 7
AWESOME! I really enjoyed this experience. I found the overhead very light for creating reusable components.
Also — if this still sounds like to much work for you — the least you can do is just write and share a gist of your snippet.
Sharing in the private domain (within your company)
Okay, now things get a bit more interesting. You have a key component that is used in multiple code bases. What is the best way to share that. But we need to, a la Gandalf: “Keep it secret, keep it safe”
- The sharing should be done via pip, and the requirements.txt file
- I do not want to set up my own pypi server
- It needs to be secure within our organization
- But, needs to be deployable to 3rd party PaaS providers like heroku or elastic beanstalk
To accomplish this, I am using pip’s ability to interact with git over ssh. That and git tagging. Here are the steps that I took …
- Create a private git repo (I am using github)
Make your package, just like we did before. For this example I will share piece of code that I use to inspect into dictionaries. here is the project (note: its public to share with you guys — but it the real use case it MUST be private.)
push to github. It needs to have all the same trappings of a python package you would push to PyPi
When working with git and pip, you need to go through an additional step of explicitly tagging your versions
- For YOU to install the most current version from HEAD, we can now do the following:
- For YOU to install the a specific version, we can now do the following:
— or —
in your requirements.txt file
At this point, your colleagues and deployment machines WILL NOT be able to access this package
Time to provision them …
For your colleagues to access this library:
- First they will need to have access to your github repository. So make sure you have added her/them as collaborators. Best handled on the github website
- They will need to turn on ssh agent forwarding. Following instructions (here)
For your deployment machines to access the library. You will need to follow the following two steps:
Note: As an alternative you can host your own secure PyPi server.
What I really like about the GitHub approach is ( a ) all of the ACL heavy lift is handled by GitHub, and ( b ) they are responsible for keeping their server up.
Cool — this is working for me. Love to hear others thoughts and successful approaches.
Other interesting articles while researching this …