tushman.io

The musings of an insecure technologist

Sharing Code Across Applications With Python

tl;dr: Use pip, git(hub) and tags to share packages privately amongst your team

Problem Statement We have libraries that we need to share across multiple applications. Using Git submodules, makes me cry — there has to be a better way (and there is)

I am going to break this up into three sections:

Packaging Python

This section needs to happen regardless if you share in the public domain or privately

Python (our beloved language with bells and whistles) has setuptools built in that we can use to package our utility.

I really do not having anything to say that hasn’t been said before. I found this tutorial to be excellent:

http://www.scotttorborg.com/python-packaging/minimal.html

But the gist of it is:

  1. create a setup.py (to define whats to be shared using setuptools)
  2. Gotsta have README.rst (sharing code with documenting it is useless)
  3. Test it locally
  4. Use Semantic Versioning (or you are a bad person)

Sharing in the public domain

Okay, you wrote an amazing utility and want to share it with the world. I really was blown away how easy it was to do this with thanks to PyPi (Python Package Index) (aka the Cheeseshop)

For this example, I wrote a package called human_dates That takes some of the syntax sugar from the ruby and rails world and brings it over to python land.

You can find the code here

And you will see the light if you continue reading the packaging tutorial from the first section

Once you have you package tested locally, its as simple as:

1
2

$ python setup.py register
$ python setup.py sdist upload

And now anyone can do the following:

1

$ pip install human_dates

And then in a python console, you can:

1
2
3
4
5
6
7

from human_dates import time_ago_in_words, beginning_of_day

print time_ago_in_words()
#prints "just now"

print time_ago_in_words(beginning_of_day())
# prints 8 hours ago

AWESOME! I really enjoyed this experience. I found the overhead very light for creating reusable components.

Also — if this still sounds like to much work for you — the least you can do is just write and share a gist of your snippet.

Sharing in the private domain (within your company)

Okay, now things get a bit more interesting. You have a key component that is used in multiple code bases. What is the best way to share that. But we need to, a la Gandalf: “Keep it secret, keep it safe”

Our Requirements:

  • The sharing should be done via pip, and the requirements.txt file
  • I do not want to set up my own pypi server
  • It needs to be secure within our organization
  • But, needs to be deployable to 3rd party PaaS providers like heroku or elastic beanstalk

To accomplish this, I am using pip’s ability to interact with git over ssh. That and git tagging. Here are the steps that I took …

  • Create a private git repo (I am using github)

  • Make your package, just like we did before. For this example I will share piece of code that I use to inspect into dictionaries. here is the project (note: its public to share with you guys — but it the real use case it MUST be private.)

  • push to github. It needs to have all the same trappings of a python package you would push to PyPi

  • When working with git and pip, you need to go through an additional step of explicitly tagging your versions

1
2

git tag -a v0.1.0 -m 'version 0.1.0 initial version'
git push --tags
  • For YOU to install the most current version from HEAD, we can now do the following:
1

pip install git+ssh://git@github.com/jtushman/dict_digger.git
  • For YOU to install the a specific version, we can now do the following:
1

pip install git+ssh://git@github.com/jtushman/dict_digger.git@v0.1.0

— or —

in your requirements.txt file

requirements.txt
1
2

requests==1.2.2
git+ssh://git@github.com/jtushman/dict_digger.git@v0.1.0

At this point, your colleagues and deployment machines WILL NOT be able to access this package

Time to provision them …

  • For your colleagues to access this library:

    1. First they will need to have access to your github repository. So make sure you have added her/them as collaborators. Best handled on the github website
    2. They will need to turn on ssh agent forwarding. Following instructions (here)

  • For your deployment machines to access the library. You will need to follow the following two steps:

    1. Create a Machine User
    2. Add the ssh keys to your deploy machine. If you are using heroku. Its as simple as this

Note: As an alternative you can host your own secure PyPi server.

What I really like about the GitHub approach is ( a ) all of the ACL heavy lift is handled by GitHub, and ( b ) they are responsible for keeping their server up.

Cool — this is working for me. Love to hear others thoughts and successful approaches.

Other interesting articles while researching this …

Comments