logo
down
shadow

How to configure Luigi task retry correctly?


How to configure Luigi task retry correctly?

Content Index :

How to configure Luigi task retry correctly?
Tag : python , By : FriendL
Date : November 28 2020, 11:01 PM

will be helpful for those in need This is an old issue of Luigi - where successful retried tasks were not marked as such when failed and then succeeded on retry: https://github.com/spotify/luigi/issues/1932
It was fixed in version 2.7.2: https://github.com/spotify/luigi/releases/tag/2.7.2

Comments
No Comments Right Now !

Boards Message :
You Must Login Or Sign Up to Add Your Comments .

Share : facebook icon twitter icon

Retry task without increment retry counter


Tag : development , By : Cowtung
Date : March 29 2020, 07:55 AM
I think the issue was by ths following , Set max_retries = None and it will never stop retrying. Pass it as a keyword to the task decorator or to the retry call itself.
Documentation

use luigi.DateParameter in a run function of a Luigi Orchestrator Task


Tag : python , By : Shitic
Date : March 29 2020, 07:55 AM
Does that help Your code is incomplete but I'll guess the rest is like the following. You must have an error somewhere because it works: DateParameter returns a value that is a python date. See luigi source code for details.
My tasks/foo.py:
from datetime import date, timedelta
import luigi


class AggregateArtists(luigi.Task):
    date = luigi.DateParameter(default=date.today() - timedelta(days=1))

    def output(self):
        return luigi.LocalTarget("/tmp/foobar.txt")

    def run(self):
        with self.output().open('w') as out_file:
            out_file.write(self.date.strftime("%Y%m%d") + "\n")


if __name__ == "__main__":
    luigi.run()
$ python tasks/foo.py AggregateArtists --local-scheduler

DEBUG: Checking if AggregateArtists(date=2015-12-03) is complete
INFO: Scheduled AggregateArtists(date=2015-12-03) (PENDING)
INFO: Done scheduling tasks
INFO: Running Worker with 1 processes
DEBUG: Asking scheduler for work...
DEBUG: Pending tasks: 1
INFO: [pid 21831] Worker Worker(salt=179482616, workers=1, host=matagus-laptop, username=matagus, pid=21831) running   AggregateArtists(date=2015-12-03)
INFO: [pid 21831] Worker Worker(salt=179482616, workers=1, host=matagus-laptop, username=matagus, pid=21831) done      AggregateArtists(date=2015-12-03)
DEBUG: 1 running tasks, waiting for next task to finish
DEBUG: Asking scheduler for work...
INFO: Done
INFO: There are no more tasks to run at this time
INFO: Worker Worker(salt=179482616, workers=1, host=matagus-laptop, username=matagus, pid=21831) was stopped. Shutting down Keep-Alive thread
$ cat /tmp/foobar.txt 
20151203

celery worker does not retry task after calling retry()


Tag : python , By : Christopher
Date : March 29 2020, 07:55 AM

Where did the Luigi task go?


Tag : python , By : John Q.
Date : March 29 2020, 07:55 AM

Luigi : Error while Parallelizing luigi task by using build function


Tag : function , By : uxn
Date : March 29 2020, 07:55 AM
Related Posts Related QUESTIONS :
  • Tensorflow Estimator API save image summary in eval mode
  • How to Pack with PyQt - how to make QFrame/Layout adapt to content
  • How do I get certain Time Range in Python
  • python doubly linked list - insertAfter node
  • Open .h5 file in Python
  • Joining a directory name with a binary file name
  • python, sort list with two arguments in compare function
  • Is it possible to print from Python using non-ANSI colors?
  • Pandas concat historical data using date minus some number of days
  • CV2: Import Error in Python OpenCV
  • Is it possible to do this loop in a one-liner?
  • invalid literal for int() with base 10: - django
  • Why does my code print a value that I have not assigned as yet?
  • the collatz func in automate boring stuff with python
  • How to find all possible combinations of parameters and funtions
  • about backpropagation deep neural network in tensorflow
  • Sort strings in pandas
  • How do access my flask app hosted in docker?
  • Replace the sentence include some text with Python regex
  • Counting the most common element in a 2D List in Python
  • logout a user from the system using a function in python
  • mp4 metadata not found but exists
  • Django: QuerySet with ExpressionWrapper
  • Pandas string search in list of dicts
  • Decryption from RSA encrypted string from sqlite is not the same
  • need of maximum value in int
  • a list of several tuples, how to extract the same of the first two elements in the small tuple in the large tuple
  • Display image of 2D Sinewaves in 3D
  • how to prevent a for loop from overwriting a dictionary?
  • How To Fix: RuntimeError: size mismatch in pyTorch
  • Concatenating two Pandas DataFrames while maintaining index order
  • Why does this not run into an infinite loop?
  • Python Multithreading no current event loop
  • Element Tree - Seaching for specific element value without looping
  • Ignore Nulls in pandas map dictionary
  • How do I get scrap data from web pages using beautifulsoup in python
  • Variable used, golobal or local?
  • I have a regex statement to pull all numbers out of a text file, but it only finds 77 out of the 81 numbers in the file
  • How do I create a dataframe of jobs and companies that includes hyperlinks?
  • Detect if user has clicked the 'maximized' button
  • Does flask_login automatically set the "next" argument?
  • Indents in python 3
  • How to create a pool of threads
  • Pandas giving IndexError on one dataframe but not on another similar dataframe
  • Django Rest Framework - Testing client.login doesn't login user, ret anonymous user
  • Running dag without dag file in airflow
  • Filling across a specified dimension of a numpy array
  • Python populating dataframe in pandas from text files
  • How to interpolate a single ("non-piecewise") cubic spline from a set of data points?
  • Divide 2 integers (leetcode 29) - recursion issue
  • Can someone explain why do I get this output in Python?
  • How do I scrape pdf and html from search results without obvious url
  • Is there a way to automatically make a "collage" of plots with matplotlib?
  • How to combine multiple rows in pandas with shared column values
  • How do I get LOAD_CLASSDEREF instruction after dis.dis?
  • Django - How to add items to Bootstrap dropdown?
  • Linear Regression - Does the below implementation of ridge regression finding coefficient term using gradient method is
  • How to drop all rows in pandas dataframe with negative values?
  • Most Efficient Way to Find Closest Date Between 2 Dataframes
  • Execution error when Passing arguments to a python script using os.system. The script takes sys.argv arguments
  • shadow
    Privacy Policy - Terms - Contact Us © scrbit.com