Airflow: Why is there a start_date for operators?
Date : March 29 2020, 07:55 AM
like below fixes the issue Regarding start_date on task instance, personally I have never used this, I always just have a single DAG start_date. However from what I can see this would allow you to specify certain tasks to start at a different time from the main DAG. It appears this is a legacy feature and from reading the FAQ they recommend using time sensors for that type of thing instead and just having one start_date for all tasks passed through the DAG.
|
how to create custom operators in airflow and use them in airflow template which is running through cloud composer(in go
Tag : python , By : user143729
Date : March 29 2020, 07:55 AM
|
Airflow - No module named 'airflow.contrib.operators.mssql_to_gcs'
Date : March 29 2020, 07:55 AM
|
Installing (python3) airflow does not create airflow directory
Date : March 29 2020, 07:55 AM
I hope this helps you . Found that unlike with python2, airflow in python3 does not create the ~/airflow dir until the first time airflow is run [airflow@airflowetl ~]$ airflow initdb
[2019-10-18 13:36:56,916] {__init__.py:51} INFO - Using executor SequentialExecutor
DB: sqlite:////home/airflow/airflow/airflow.db
[2019-10-18 13:36:57,438] {db.py:369} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema
INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
/home/airflow/.local/lib/python3.6/site-packages/alembic/ddl/sqlite.py:39: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
"Skipping unsupported ALTER for "
....
INFO [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
WARNI [airflow.utils.log.logging_mixin.LoggingMixin] cryptography not found - values will not be stored encrypted.
Done.
[airflow@airflowetl ~]$ cd
[airflow@airflowetl ~]$ ls
airflow
[airflow@airflowetl ~]$ tree airflow/
airflow/
├── airflow.cfg
├── airflow.db
├── logs
│ └── scheduler
│ ├── 2019-10-18
│ └── latest -> /home/airflow/airflow/logs/scheduler/2019-10-18
└── unittests.cfg
4 directories, 3 files
[airflow@airflowetl ~]$ mkdir airflow/dags
[airflow@airflowetl ~]$ cd
[airflow@airflowetl ~]$ pwd
/home/airflow
[airflow@airflowetl ~]$ find . -name airflow
./.local/lib/python3.6/site-packages/airflow
./.local/lib/python3.6/site-packages/airflow/bin/airflow
./.local/lib/python3.6/site-packages/airflow/www/templates/airflow
./.local/lib/python3.6/site-packages/airflow/www_rbac/templates/airflow
./.local/bin/airflow
[airflow@airflowetl ~]$ echo "PATH=$PATH:~/.local/bin" >> ~/.bashrc
[airflow@airflowetl ~]$ source ~/.bashrc
|
Airflow: Using macro outside of operators
Date : March 29 2020, 07:55 AM
|