From 67f1323dd936d5bf775e7d0a57a7bcad5f09dc69 Mon Sep 17 00:00:00 2001 From: Suzanne Selhorn <sselhorn@gitlab.com> Date: Fri, 6 Nov 2020 22:23:19 +0000 Subject: [PATCH] Docs: Edited PyPI topic Related to: https://gitlab.com/gitlab-org/gitlab/-/issues/247950 --- doc/api/README.md | 2 +- doc/user/packages/package_registry/index.md | 1 - doc/user/packages/pypi_repository/index.md | 344 ++++++++++---------- 3 files changed, 174 insertions(+), 173 deletions(-) diff --git a/doc/api/README.md b/doc/api/README.md index 2a5c473dcc69..63a936c258de 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -184,7 +184,7 @@ to authenticate with the API: - [Maven Repository](../user/packages/maven_repository/index.md#authenticate-with-a-ci-job-token-in-maven) - [NPM Repository](../user/packages/npm_registry/index.md#authenticate-with-a-ci-job-token) - [Nuget Repository](../user/packages/nuget_repository/index.md) - - [PyPI Repository](../user/packages/pypi_repository/index.md#publish-a-pypi-package-by-using-cicd) + - [PyPI Repository](../user/packages/pypi_repository/index.md#authenticate-with-a-ci-job-token) - [Generic packages](../user/packages/generic_packages/index.md#publish-a-generic-package-by-using-cicd) - [Get job artifacts](job_artifacts.md#get-job-artifacts) - [Pipeline triggers](pipeline_triggers.md) (using the `token=` parameter) diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index a279779e2e77..56fd4a02ba03 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -38,7 +38,6 @@ Learn more about using CI/CD to build: - [Composer packages](../composer_repository/index.md#publish-a-composer-package-by-using-cicd) - [NuGet packages](../nuget_repository/index.md#publish-a-nuget-package-by-using-cicd) - [Conan packages](../conan_repository/index.md#publish-a-conan-package-by-using-cicd) -- [PyPI packages](../pypi_repository/index.md#publish-a-pypi-package-by-using-cicd) - [Generic packages](../generic_packages/index.md#publish-a-generic-package-by-using-cicd) If you use CI/CD to build a package, extended activity information is displayed diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index 7df0f5ef1ceb..bc77bb2ccc59 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -12,67 +12,75 @@ info: To determine the technical writer assigned to the Stage/Group associated w Publish PyPI packages in your project’s Package Registry. Then install the packages whenever you need to use them as a dependency. -The GitLab PyPI Repository works with: +The Package Registry works with: - [pip](https://pypi.org/project/pip/) - [twine](https://pypi.org/project/twine/) ## Build a PyPI package -This section covers creating a new example PyPI package to upload. This is a -quickstart to test out the **GitLab PyPI Registry**. If you already understand -how to build and publish your own packages, move on to the [next section](#authenticate-with-the-package-registry). +This section explains how to create a PyPI package. + +If you already use PyPI and know how to build your own packages, go to the +[next section](#authenticate-with-the-package-registry). ### Install pip and twine -You need a recent version of [pip](https://pypi.org/project/pip/) and [twine](https://pypi.org/project/twine/). +Install a recent version of [pip](https://pypi.org/project/pip/) and +[twine](https://pypi.org/project/twine/). ### Create a project -Understanding how to create a full Python project is outside the scope of this -guide, but you can create a small package to test out the registry. Start by -creating a new directory called `MyPyPiPackage`: +Create a test project. -```shell -mkdir MyPyPiPackage && cd MyPyPiPackage -``` +1. Open your terminal. +1. Create a directory called `MyPyPiPackage`, and then go to that directory: -After creating this, create another directory inside: + ```shell + mkdir MyPyPiPackage && cd MyPyPiPackage + ``` -```shell -mkdir mypypipackage && cd mypypipackage -``` +1. Create another directory and go to it: -Create two new files inside this directory to set up the basic project: + ```shell + mkdir mypypipackage && cd mypypipackage + ``` -```shell -touch __init__.py -touch greet.py -``` +1. Create the required files in this directory: -Inside `greet.py`, add the following code: + ```shell + touch __init__.py + touch greet.py + ``` -```python -def SayHello(): - print("Hello from MyPyPiPackage") - return -``` +1. Open the `greet.py` file, and then add: -Inside the `__init__.py` file, add the following: + ```python + def SayHello(): + print("Hello from MyPyPiPackage") + return + ``` -```python -from .greet import SayHello -``` +1. Open the `__init__.py` file, and then add: -Now that the basics of our project is completed, we can test that the code runs. -Start the Python prompt inside your top `MyPyPiPackage` directory. Then run: + ```python + from .greet import SayHello + ``` -```python ->>> from mypypipackage import SayHello ->>> SayHello() -``` +1. To test the code, in your `MyPyPiPackage` directory, start the Python prompt. -You should see an output similar to the following: + ```shell + python + ``` + +1. Run this command: + + ```python + >>> from mypypipackage import SayHello + >>> SayHello() + ``` + +A message indicates that the project was set up successfully: ```plaintext Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18) @@ -83,83 +91,81 @@ Type "help", "copyright", "credits" or "license" for more information. Hello from MyPyPiPackage ``` -After we've verified that the sample project is working as previously described, -we can next work on creating a package. - ### Create a package -Inside your `MyPyPiPackage` directory, we need to create a `setup.py` file. Run -the following: - -```shell -touch setup.py -``` - -This file contains all the information about our package. For more information -about this file, see [creating setup.py](https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py). -Becaue GitLab identifies packages based on -[Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), -ensure your package name meets these requirements. See the [installation section](#publish-a-pypi-package-by-using-cicd) -for more details. - -For this guide, we don't need to extensively fill out this file. Add the -following to your `setup.py`: - -```python -import setuptools - -setuptools.setup( - name="mypypipackage", - version="0.0.1", - author="Example Author", - author_email="author@example.com", - description="A small example package", - packages=setuptools.find_packages(), - classifiers=[ - "Programming Language :: Python :: 3", - "License :: OSI Approved :: MIT License", - "Operating System :: OS Independent", - ], - python_requires='>=3.6', -) -``` - -Save the file, and then execute the setup: - -```shell -python3 setup.py sdist bdist_wheel -``` - -If successful, you should be able to see the output in a newly created `dist` -folder. Run: +After you create a project, you can create a package. + +1. In your terminal, go to the `MyPyPiPackage` directory. +1. Create a `setup.py` file: + + ```shell + touch setup.py + ``` + + This file contains all the information about the package. For more information + about this file, see [creating setup.py](https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py). + Because GitLab identifies packages based on + [Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), + ensure your package name meets these requirements. See the [installation section](#authenticate-with-a-ci-job-token) + for details. + +1. Open the `setup.py` file, and then add basic information: + + ```python + import setuptools + + setuptools.setup( + name="mypypipackage", + version="0.0.1", + author="Example Author", + author_email="author@example.com", + description="A small example package", + packages=setuptools.find_packages(), + classifiers=[ + "Programming Language :: Python :: 3", + "License :: OSI Approved :: MIT License", + "Operating System :: OS Independent", + ], + python_requires='>=3.6', + ) + ``` + +1. Save the file. +1. Execute the setup: + + ```shell + python3 setup.py sdist bdist_wheel + ``` + +The output should be visible in a newly-created `dist` folder: ```shell ls dist ``` -And confirm your output matches the below: +The output should appear similar to the following: ```plaintext mypypipackage-0.0.1-py3-none-any.whl mypypipackage-0.0.1.tar.gz ``` -The package is now all set up and is ready to be uploaded to the -_GitLab PyPI Package Registry_. Before we do so, we next need to set up -authentication. +The package is now ready to be published to the Package Registry. ## Authenticate with the Package Registry -### Authenticate with a personal access token +Before you can publish to the Package Registry, you must authenticate. + +To do this, you can use: -You need the following: +- A [personal access token](../../../user/profile/personal_access_tokens.md) + with the scope set to `api`. +- A [deploy token](./../../project/deploy_tokens/index.md) with the scope set to + `read_package_registry`, `write_package_registry`, or both. +- A [CI job token](#authenticate-with-a-ci-job-token). -- A personal access token. You can generate a - [personal access token](../../../user/profile/personal_access_tokens.md) - with the scope set to `api` for repository authentication. -- A suitable name for your source. -- Your project ID, which is found on the home page of your project. +### Authenticate with a personal access token -Edit your `~/.pypirc` file and add the following: +To authenticate with a personal access token, edit the `~/.pypirc` file and add: ```ini [distutils] @@ -167,22 +173,17 @@ index-servers = gitlab [gitlab] -repository = https://gitlab.com/api/v4/projects/<project_id>/packages/pypi +repository = https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi username = __token__ password = <your personal access token> ``` -### Authenticate with a deploy token - -You need the following: +- `username` must be `__token__` exactly. +- Your project ID is on your project's home page. -- A deploy token. You can generate a [deploy token](./../../project/deploy_tokens/index.md) - with the `read_package_registry` or `write_package_registry` scopes for - repository authentication. -- A suitable name for your source. -- Your project ID, which is found on the home page of your project. +### Authenticate with a deploy token -Edit your `~/.pypirc` file and add the following: +To authenticate with a deploy token, edit your `~/.pypirc` file and add: ```ini [distutils] @@ -190,22 +191,57 @@ index-servers = gitlab [gitlab] -repository = https://gitlab.com/api/v4/projects/<project_id>/packages/pypi +repository = https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi username = <deploy token username> password = <deploy token> ``` +Your project ID is on your project's home page. + +### Authenticate with a CI job token + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202012) in GitLab 13.4. + +To work with PyPI commands within [GitLab CI/CD](./../../../ci/README.md), you +can use `CI_JOB_TOKEN` instead of a personal access token or deploy token. + +For example: + +```yaml +image: python:latest + +run: + script: + - pip install twine + - python setup.py sdist bdist_wheel + - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url https://gitlab.example.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi dist/* +``` + +You can also use `CI_JOB_TOKEN` in a `~/.pypirc` file that you check in to +GitLab: + +```ini +[distutils] +index-servers = + gitlab + +[gitlab] +repository = https://gitlab.example.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/pypi +username = gitlab-ci-token +password = ${env.CI_JOB_TOKEN} +``` + ## Publish a PyPI package When publishing packages, note that: -- The maximum allowed size is 50 Megabytes. +- The maximum allowed size is 50 MB. - You can't upload the same version of a package multiple times. If you try, you'll receive the error `Validation failed: File name has already been taken`. ### Ensure your version string is valid -If your version string (for example, `0.0.1`) is invalid, it will be rejected. +If your version string (for example, `0.0.1`) isn't valid, it will be rejected. GitLab uses the following regex to validate the version string. ```ruby @@ -220,120 +256,86 @@ GitLab uses the following regex to validate the version string. )\z}xi ``` -You can experiment with the regex and try your version strings using this +You can experiment with the regex and try your version strings by using this [regular expression editor](https://rubular.com/r/FKM6d07ouoDaFV). -For more details about the regex used, review this [documentation](https://www.python.org/dev/peps/pep-0440/#appendix-b-parsing-version-strings-with-regular-expressions). +For more details about the regex, review this [documentation](https://www.python.org/dev/peps/pep-0440/#appendix-b-parsing-version-strings-with-regular-expressions). ### Publish a PyPI package by using twine -If you were following the steps on this page, the `MyPyPiPackage` package should -be ready to be uploaded. Run the following command: +To publish a PyPI package, run a command like: ```shell python3 -m twine upload --repository gitlab dist/* ``` -If successful, you should see the following: +This message indicates that the package was published successfully: ```plaintext -Uploading distributions to https://gitlab.com/api/v4/projects/<your_project_id>/packages/pypi +Uploading distributions to https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/pypi Uploading mypypipackage-0.0.1-py3-none-any.whl 100%|███████████████████████████████████████████████████████████████████████████████████████████| 4.58k/4.58k [00:00<00:00, 10.9kB/s] Uploading mypypipackage-0.0.1.tar.gz 100%|███████████████████████████████████████████████████████████████████████████████████████████| 4.24k/4.24k [00:00<00:00, 11.0kB/s] ``` -This indicates that the package was uploaded successfully. You can then navigate -to your project's **Packages & Registries** page and see the uploaded packages. +To view the published package, go to your project's **Packages & Registries** +page. -If you would rather not use a `.pypirc` file to define your repository source, -you can upload to the repository with the authentication inline: +If you didn't use a `.pypirc` file to define your repository source, you can +publish to the repository with the authentication inline: ```shell -TWINE_PASSWORD=<personal_access_token or deploy_token> TWINE_USERNAME=<username or deploy_token_username> python3 -m twine upload --repository-url https://gitlab.com/api/v4/projects/<project_id>/packages/pypi dist/* +TWINE_PASSWORD=<personal_access_token or deploy_token> TWINE_USERNAME=<username or deploy_token_username> python3 -m twine upload --repository-url https://gitlab.example.com/api/v4/projects/<project_id>/packages/pypi dist/* ``` -If you didn't use the steps on this page, you need to ensure your package has -been properly built, and that you [created a PyPI package with `setuptools`](https://packaging.python.org/tutorials/packaging-projects/). +If you didn't follow the steps on this page, ensure your package was properly +built, and that you [created a PyPI package with `setuptools`](https://packaging.python.org/tutorials/packaging-projects/). -You can then upload your package using the following command: +You can then upload your package by using the following command: ```shell python -m twine upload --repository <source_name> dist/<package_file> ``` -Where: - - `<package_file>` is your package filename, ending in `.tar.gz` or `.whl`. - `<source_name>` is the [source name used during setup](#authenticate-with-the-package-registry). -### Publish a PyPI package by using CI/CD - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202012) in GitLab 13.4. - -To work with PyPI commands within [GitLab CI/CD](./../../../ci/README.md), you -can use `CI_JOB_TOKEN` in place of the personal access token or deploy a token -in your commands. - -For example: - -```yaml -image: python:latest - -run: - script: - - pip install twine - - python setup.py sdist bdist_wheel - - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url https://gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi dist/* -``` - -You can also use `CI_JOB_TOKEN` in a `~/.pypirc` file that you check into GitLab: - -```ini -[distutils] -index-servers = - gitlab - -[gitlab] -repository = https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/pypi -username = gitlab-ci-token -password = ${env.CI_JOB_TOKEN} -``` - ## Install a PyPI package -Install the latest version of a package using the following command: +To install the latest version of a package, use the following command: ```shell -pip install --extra-index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<project_id>/packages/pypi/simple --no-deps <package_name> +pip install --extra-index-url https://__token__:<personal_access_token>@gitlab.example.com/api/v4/projects/<project_id>/packages/pypi/simple --no-deps <package_name> ``` -Where: - - `<package_name>` is the package name. - `<personal_access_token>` is a personal access token with the `read_api` scope. - `<project_id>` is the project ID. -If you were following the guide above and want to test installing the -`MyPyPiPackage` package, you can run the following: +If you were following the guide and want to install the +`MyPyPiPackage` package, you can run: ```shell -pip install mypypipackage --no-deps --extra-index-url https://__token__:<personal_access_token>@gitlab.com/api/v4/projects/<your_project_id>/packages/pypi/simple +pip install mypypipackage --no-deps --extra-index-url https://__token__:<personal_access_token>@gitlab.example.com/api/v4/projects/<your_project_id>/packages/pypi/simple ``` -This should result in the following: +This message indicates that the package was installed successfully: ```plaintext -Looking in indexes: https://__token__:****@gitlab.com/api/v4/projects/<your_project_id>/packages/pypi/simple +Looking in indexes: https://__token__:****@gitlab.example.com/api/v4/projects/<your_project_id>/packages/pypi/simple Collecting mypypipackage - Downloading https://gitlab.com/api/v4/projects/<your_project_id>/packages/pypi/files/d53334205552a355fee8ca35a164512ef7334f33d309e60240d57073ee4386e6/mypypipackage-0.0.1-py3-none-any.whl (1.6 kB) + Downloading https://gitlab.example.com/api/v4/projects/<your_project_id>/packages/pypi/files/d53334205552a355fee8ca35a164512ef7334f33d309e60240d57073ee4386e6/mypypipackage-0.0.1-py3-none-any.whl (1.6 kB) Installing collected packages: mypypipackage Successfully installed mypypipackage-0.0.1 ``` -GitLab looks for packages using -[Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), -so the characters `-`, `_`, and `.` are all treated the same and repeated characters are removed. +### Package names + +GitLab looks for packages that use +[Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names). +The characters `-`, `_`, and `.` are all treated the same, and repeated +characters are removed. + A `pip install` request for `my.package` looks for packages that match any of the three characters, such as `my-package`, `my_package`, and `my....package`. -- GitLab