Problem to solve

Python developers need a mechanism to create, share, and consume packages that contain compiled code and other content in projects that consume these packages. PyPI, an open source project maintained by the Python Packaging Authority, is the standard for how to define, create, host and consume Python packages.

By integrating with PyPI, GitLab will provide a centralized location to store and view those packages, in the same place as their source code and pipelines.

This epic contributes to our vision, by providing a single application CI/CD experience for our customers and our users.

Intended users

• Software Developer
• DevOps Engineer

Proposal

Provide support for users coding in Python by integrating with PyPI and allowing developers to publish, share and consume Python packages alongside their source code and CI/CD pipelines.

Proposed MVC Scope

• Users can add GitLab as a remote repository.
• Users are able to add a project PyPI Repository url in an authenticated manner using GitLab username + token via:
  • ~/.pip/pip.conf
  • requirements.txt
• Users are able to publish a Python package to GitLab:
  • With sdist: python setup.py sdist upload -r <gitlab_remote>
  • With wheel: python setup.py bdist_wheel upload -r <gitlab_remote>
  • With twine: twine upload -r pypi <filename>
• Users are able to pull a Python package (pip install <package_name>)
• Users are able to delete a Python package (For the MVC - we will only allow users to delete packages via the UI or API)
• The MVC will be limited to the backend work and we will address the UI in subsequent milestones.

Further details

User stories

Administrator

• I as an administrator of GitLab, need the ability to enable/disable the Package Registry, so that I can ensure the developers in my organization have access to the features that they are supposed to.
• I as an administrator of Gitlab, need the ability to configure object storage for the GitLab Package Registry, including the PyPI Repository, so that I can optimize how my organization utilizes storage.

Developer

• I as a developer, need the ability to configure PyPI to point to GitLab as a remote repository, so that I can push, pull and view my Python packages with GitLab.
• I as a developer, need the ability to setup authentication between GitLab and PyPI using my personal access token, so that I can push and pull packages to the GitLab PyPI Repository.
• I as a developer, need the ability to run PyPI primary commands from the CLI to push, pull and update Python packages in the GitLab PyPI Repository at the project level.
• I as a developer, need the ability to view basic meta data about packages from within the GitLab UI, so that I can verify package info and ensure my project is using the correct dependencies.
• I as a developer need the ability to view GitLab metadata, such as pipeline_id, branch and commit associated with a package, so that I can understand how a package was built, by whom and when.
• I as a developer, need the ability to delete packages from within the GitLab UI, so that I can remove old packages and ensure they are not accidentally used in my project.

Reporter

• I as a project-stakeholder need the ability to view and pull packages from the PyPI Repository, so that I can view, inspect and download Python packages.

Example UI (not included in the MVC)

Metadata (for the UI)

• Name
• Version
• Platform
• Summary
• Keywords
• Homepage
• Download URL
• Author
• Author email
• License

Permissions and Security

The permissions should follow the same levels as all other package registries

Project Permissions: UI

Action	Guest	Reporter	Developer	Maintainer	Owner
Pull from Maven, NPM, Conan, NuGet, PyPI		x	x	x	x
Publish to Maven, NPM, Conan, NuGet, PyPI			x	x	x

Project Permissions: API

Action	Guest	Reporter	Developer	Maintainer	Owner
List project packages (5)				x	x
Get a project package				x	x
List package files				x	x
Delete a project package				x	x

Group Permissions: API

Action	Guest	Reporter	Developer	Maintainer	Owner
[List the packages of a group				x	x

Instance Level Permissions

Action	Guest	Reporter	Developer	Maintainer	Owner
Enable the Packages feature					x
Migrate local packages to object storage					x
Disable the Packages feature					x

Documentation

• Add a new section to the package registry user guides for the PyPI Repository at https://docs.gitlab.com/ee/user/project/packages/pypi_repository.html
• Packages API
• Packages Admin

What does success look like, and how can we measure that?

The goal of the Package Group is to ensure that in 3 years, 90% of our customers are using GitLab as their sole package registry. Success for this issue will be that we begin to empower Python developers to utilize GitLab for publishing and sharing packages.

Outcomes

• We'd like to see a 5% increase in overall adoption of the GitLab Package Registry with the launch of this MVC
• We expect that number to rise to 10%, once we enable CI_JOB_TOKEN support
• As this is an MVC, we expect to see an increased volume of issues for the feature.

Data and metrics

• Count of packages pushed to the PyPI Repository
• Count of packages pulled from the PyPI Repository
• Count of packages deleted from the PyPI Repository
• Count of repository setup commands copied
• Count of package install commands copied

Links / references

• JFrog's PyPI Integration
• Sonatype's PyPI Integration