How to run a python script with another virtual environment dependencies in a DAG in Airflow?

72
December 11, 2019, at 00:40 AM

How to run a python script with another virtual environment dependencies in a DAG in Airflow? Not using the main python environment where airflow is installed.

I have a docker container running the puckel image and I'm trying to create a DAG which consists of a blob sensor and a bash operator. The bash operator is meant to run a python script, which has dependencies of specific packages different than the main python installation where Airflow is installed.

The way I'm doing it is by building a (python 3.7) virtual env in the image and place it in "~/new_env/" with a version of azure-storage-blob=12.1.0. In the BashOperator I have the following definition:

BashOperator(
        task_id='task_id_1',
        bash_command=f'{PATH_TO_VIRTUAL_ENV}/bin/python {PATH_TO_PYTHON_SCRIPT_1}/some_py_script.py'
    )

The python environment where Airflow is running has azure-storage-blob=1.3.0. The script gets stuck in running.

Is this not a good approach? if not, has anyone solved this issue or use case?

Thanks

READ ALSO
Error when opening executable in raspberry pi [closed]

Error when opening executable in raspberry pi [closed]

Want to improve this question? Update the question so it's on-topic for Stack Overflow

96
Getting error about argument when calling method

Getting error about argument when calling method

I have feature file from where I am trying to get the email :

59
How to save results of lenghty calculation with multiple parameters?

How to save results of lenghty calculation with multiple parameters?

I am doing some lengthy calculations with multiple parametersI would like to save and retrieve the results of these calculations

74
Deleting elements from list not results as expected [duplicate]

Deleting elements from list not results as expected [duplicate]

I'm new to python, just exploring list in python

47