Categories
Python

Conda Python Environments

1. Creating Environments

conda create --name py2 python=2.7
conda create --name py3 python=3.7

2. Switch to Python 2

source activate py2

Deactivate Conda Python 2

source deactivate

3. Switch to Python 3

source activate py3

Deactivate Conda Python 3

source deactivate
Categories
Python Web Server

Run shell script in Apache Airflow

Create the script.py inside the ../airflow/dags folder

from builtins import range
from datetime import timedelta

import airflow
from airflow.models import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.dummy_operator import DummyOperator

args = {
    'owner': 'airflow',
    'start_date': airflow.utils.dates.days_ago(2),
}

dag = DAG(
    dag_id='dagid1',
    default_args=args,
    schedule_interval='0 0 * * *',
    dagrun_timeout=timedelta(minutes=60),
)

# [START howto_operator_bash]
create_command = """
echo 'Hello world';
echo 'Welcome';"""

run_this = BashOperator(
    task_id='taskid1',
    bash_command=create_command,
    dag=dag,
)
Categories
Python

Apache Airflow Beginners guide

export AIRFLOW_HOME=~/airflow

# install from pypi using pip
pip install apache-airflow

# initialize the database
airflow initdb

# start the web server, default port is 8080
airflow webserver -p 8080

# start the scheduler
airflow scheduler

# visit localhost:8080 in the browser and magnum pharmaceuticals enable the example dag in the home page

Nginx reverse proxy

Airflow reverse proxy without rewrite

server {
  listen 80;
  server_name lab.example.com;

  location /myorg/airflow/ {
      proxy_pass http://localhost:8080;
      proxy_set_header Host $host;
      proxy_redirect off;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
  }
}

Airflow reverse proxy with rewrite

server {
    listen 80;
    server_name lab.example.com;

    location /myorg/workflow/ {
        rewrite ^/myorg/workflow/(.*)$ /$1 break;  # remove prefix from http header
        proxy_pass http://localhost:5555;
        proxy_set_header Host $host;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}
Categories
Python

Python Virtual Environment Wrapper

$ pip install virtualenvwrapper
$ which virtualenvwrapper.sh
/usr/local/bin/virtualenvwrapper.sh

Adding path

$ export WORKON_HOME=$HOME/.virtualenvs   # Optional
$ export PROJECT_HOME=$HOME/projects      # Optional
$ source /usr/local/bin/virtualenvwrapper.sh

Reloading Shell

$ source ~/.bashrc
$ echo $WORKON_HOME
/Users/mervinpraison/.virtualenvs

Functions

$ mkvirtualenv my-new-project
(my-new-project) $

To stop using the environment

(my-new-project) $ deactivate
$
$ workon
my-new-project

To activate the environment

$ workon my-new-project

-p parameter to choose the Python Version

$ virtualenv -p $(which python3) blog_virtualenv
Categories
Python Uncategorized

Python Redirect Path

import requests 
s = requests.Session()
response = s.get("https://google.com/")
cookies = dict(response.cookies)
if response.history:
    print ("Request was redirected")
    for resp in response.history:
        print (resp.status_code, resp.url)
    print ("Final destination:")
    print (response.status_code, response.url)
else:
    print ("Request was not redirected")