![]() Postgres_conn_id='remote-db-conn', # NOTE: host='localhost' Ssh_hook = SSHHook(ssh_conn_id='bastion-ssh-conn', keepalive_interval=60) I was having this same issue when opening the tunnel and trying to connect to the database in separate tasks, but got it working by doing both in the same task (Airflow doesn't persist state between task runs): def select_from_tunnel_db(): If more depth is needed, please let me know. At this time, I am unable to manually put all of the connections in because there will be > 100 once this system deploys. I am truly confused why the python code would work while exec-ed into the system, but not when I run it via my airflow DAG. I also followed the instructions here and got to the point where I can docker exec into my container and manually type the above python snippet and get a working connection.Īdditionally, I have read the airflow documentation here which covers SSH connection operators, but those only support bash commands, I will need my python function to run. FERNET_KEY=46BKJoQYlPPOexq0OhDZnIlNepKFf87WFwLbfzqDDho= dags/docker/sql/create.sql:/docker-entrypoint-initdb.d/init.sqlĭockerfile. ~/.whale/pgdata:/var/lib/postgresql/data/pgdata However, this does not work because my containers lose the ability to communicate with one another. I realize that docker runs and binds to a port and is not actually apart of the host system, so I used network_mode="host" to help mitigate this issue. However, within Docker, it does not work. This works as expected and I can do whatever I need from there. Local_bind_address=(localhost, localport) Remote_bind_address=(localhost, db_port), Server_conn = sshtunnel.SSHTunnelForwarder( Normally, I'd do it the following way: for server in servers: I have a list of servers which I use to create a tunnel, query the database, and close the connection. Only running the function on my local machine works fine. My program is unable to create an SSH tunnel while inside of my docker container running apache airflow.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |