Skip to content

[python] Made the script for building Docker images for Python work #8153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 1, 2020

Conversation

massakam
Copy link
Contributor

Motivation

Currently, enabling SNAPPY compression on the Python client causes a segmentation fault. I think this is because the Docker images for building the Python clients uploaded to Docker Hub do not include the SNAPPY library.

We need to run pulsar-client-cpp/docker/create-images.sh in order to build the latest Docker images that include the SNAPPY library, but this script fails for two reasons:

  1. Log4cxx compilation fails. Currently the source of log4cxx is from the master branch, but if we get the source of tag v0.11.0 and compile it, it will succeed.
In file included from logstream.cpp:24:0:
../../../src/main/include/log4cxx/private/log4cxx_private.h:43:31: error: token "@" is not valid in preprocessor expressions
 #define LOG4CXX_INIT_IOS_BASE @INIT_IOS_BASE@
                               ^
logstream.cpp:38:5: note: in expansion of macro ‘LOG4CXX_INIT_IOS_BASE’
 #if LOG4CXX_INIT_IOS_BASE
     ^
  1. Building the image pulsar-build/manylinux-cp34-cp34m fails. This is probably because Python 3.4 has already reached EOL in 2014 and is no longer included in the base image quay.io/pypa/manylinux1_x86_64.
Step 8/32 : RUN ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}
 ---> Running in 0e1d652c2c29
ln: creating symbolic link `/opt/python/cp34-cp34m/include/python3.4' to `/opt/python/cp34-cp34m/include/python3.4m': No such file or directory
The command '/bin/sh -c ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}' returned a non-zero code: 1

Modifications

  1. Get the source of Log4cxx from tag v0.11.0 when building the Docker images.
  2. Do not build the Docker image for Python 3.4.

@massakam massakam added type/bug The PR fixed a bug or issue reported a bug area/build component/python release/2.6.2 labels Sep 28, 2020
@massakam massakam added this to the 2.7.0 milestone Sep 28, 2020
@massakam massakam self-assigned this Sep 28, 2020
@massakam
Copy link
Contributor Author

/pulsarbot run-failure-checks

@sijie sijie merged commit e65875b into apache:master Oct 1, 2020
@massakam massakam deleted the fix-docker-image-for-python branch October 2, 2020 00:55
@massakam
Copy link
Contributor Author

massakam commented Oct 2, 2020

Pushed the following images to Docker Hub.

apachepulsar/pulsar-build:manylinux-cp27-cp27m
apachepulsar/pulsar-build:manylinux-cp27-cp27mu
apachepulsar/pulsar-build:manylinux-cp35-cp35m
apachepulsar/pulsar-build:manylinux-cp36-cp36m
apachepulsar/pulsar-build:manylinux-cp37-cp37m
apachepulsar/pulsar-build:manylinux-cp38-cp38

lbenc135 pushed a commit to lbenc135/pulsar that referenced this pull request Oct 3, 2020
…pache#8153)

### Motivation

Currently, enabling SNAPPY compression on the Python client causes a segmentation fault. I think this is because the Docker images for building the Python clients uploaded to Docker Hub do not include the SNAPPY library.

We need to run `pulsar-client-cpp/docker/create-images.sh` in order to build the latest Docker images that include the SNAPPY library, but this script fails for two reasons:

1. Log4cxx compilation fails. Currently the source of log4cxx is from the master branch, but if we get the source of tag `v0.11.0` and compile it, it will succeed.
```
In file included from logstream.cpp:24:0:
../../../src/main/include/log4cxx/private/log4cxx_private.h:43:31: error: token "@" is not valid in preprocessor expressions
 #define LOG4CXX_INIT_IOS_BASE @INIT_IOS_BASE@
                               ^
logstream.cpp:38:5: note: in expansion of macro ‘LOG4CXX_INIT_IOS_BASE’
 #if LOG4CXX_INIT_IOS_BASE
     ^
```
2. Building the image `pulsar-build/manylinux-cp34-cp34m` fails. This is probably because Python 3.4 has already reached EOL in 2014 and is no longer included in the base image `quay.io/pypa/manylinux1_x86_64`.
```
Step 8/32 : RUN ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}
 ---> Running in 0e1d652c2c29
ln: creating symbolic link `/opt/python/cp34-cp34m/include/python3.4' to `/opt/python/cp34-cp34m/include/python3.4m': No such file or directory
The command '/bin/sh -c ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}' returned a non-zero code: 1
```

### Modifications

1. Get the source of Log4cxx from tag v0.11.0 when building the Docker images.
2. Do not build the Docker image for Python 3.4.
wolfstudy pushed a commit that referenced this pull request Oct 30, 2020
…8153)

### Motivation

Currently, enabling SNAPPY compression on the Python client causes a segmentation fault. I think this is because the Docker images for building the Python clients uploaded to Docker Hub do not include the SNAPPY library.

We need to run `pulsar-client-cpp/docker/create-images.sh` in order to build the latest Docker images that include the SNAPPY library, but this script fails for two reasons:

1. Log4cxx compilation fails. Currently the source of log4cxx is from the master branch, but if we get the source of tag `v0.11.0` and compile it, it will succeed.
```
In file included from logstream.cpp:24:0:
../../../src/main/include/log4cxx/private/log4cxx_private.h:43:31: error: token "@" is not valid in preprocessor expressions
 #define LOG4CXX_INIT_IOS_BASE @INIT_IOS_BASE@
                               ^
logstream.cpp:38:5: note: in expansion of macro ‘LOG4CXX_INIT_IOS_BASE’
 #if LOG4CXX_INIT_IOS_BASE
     ^
```
2. Building the image `pulsar-build/manylinux-cp34-cp34m` fails. This is probably because Python 3.4 has already reached EOL in 2014 and is no longer included in the base image `quay.io/pypa/manylinux1_x86_64`.
```
Step 8/32 : RUN ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}
 ---> Running in 0e1d652c2c29
ln: creating symbolic link `/opt/python/cp34-cp34m/include/python3.4' to `/opt/python/cp34-cp34m/include/python3.4m': No such file or directory
The command '/bin/sh -c ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}' returned a non-zero code: 1
```

### Modifications

1. Get the source of Log4cxx from tag v0.11.0 when building the Docker images.
2. Do not build the Docker image for Python 3.4.

(cherry picked from commit e65875b)
huangdx0726 pushed a commit to huangdx0726/pulsar that referenced this pull request Nov 13, 2020
…pache#8153)

### Motivation

Currently, enabling SNAPPY compression on the Python client causes a segmentation fault. I think this is because the Docker images for building the Python clients uploaded to Docker Hub do not include the SNAPPY library.

We need to run `pulsar-client-cpp/docker/create-images.sh` in order to build the latest Docker images that include the SNAPPY library, but this script fails for two reasons:

1. Log4cxx compilation fails. Currently the source of log4cxx is from the master branch, but if we get the source of tag `v0.11.0` and compile it, it will succeed.
```
In file included from logstream.cpp:24:0:
../../../src/main/include/log4cxx/private/log4cxx_private.h:43:31: error: token "@" is not valid in preprocessor expressions
 #define LOG4CXX_INIT_IOS_BASE @INIT_IOS_BASE@
                               ^
logstream.cpp:38:5: note: in expansion of macro ‘LOG4CXX_INIT_IOS_BASE’
 #if LOG4CXX_INIT_IOS_BASE
     ^
```
2. Building the image `pulsar-build/manylinux-cp34-cp34m` fails. This is probably because Python 3.4 has already reached EOL in 2014 and is no longer included in the base image `quay.io/pypa/manylinux1_x86_64`.
```
Step 8/32 : RUN ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}
 ---> Running in 0e1d652c2c29
ln: creating symbolic link `/opt/python/cp34-cp34m/include/python3.4' to `/opt/python/cp34-cp34m/include/python3.4m': No such file or directory
The command '/bin/sh -c ln -s /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}m /opt/python/${PYTHON_SPEC}/include/python${PYTHON_VERSION}' returned a non-zero code: 1
```

### Modifications

1. Get the source of Log4cxx from tag v0.11.0 when building the Docker images.
2. Do not build the Docker image for Python 3.4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/build release/2.6.2 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants