logo
down
shadow

applying window function to big data set (how to optimize?)


applying window function to big data set (how to optimize?)

Content Index :

applying window function to big data set (how to optimize?)
Tag : development , By : user180941
Date : November 25 2020, 07:06 PM

wish help you to fix your issue I have to do some data analysis on a table with 400+ million rows. I got this to work on a small sample but I'm sure it will run out of memory in production. , You can use lag to do this.
select *
from (select t.*
      ,lag(status_2) over(partition by serial_no order by date) as prev_status_2
      ,lag(date) over(partition by serial_no order by date) as prev_date
      from tbl t  
     ) t 
where status_1 = 'in_transit' and prev_status_2 = 'x'

Comments
No Comments Right Now !

Boards Message :
You Must Login Or Sign Up to Add Your Comments .

Share : facebook icon twitter icon

Applying a function to subsets of data in a data frame using a moving window in R


Tag : r , By : Michael T.
Date : March 29 2020, 07:55 AM
Hope this helps As you can see, there are lots of ways to go. I think you could do this with a series of loops like @Codoremifa showed you or with some handy add-on packages such as data.table that @RInatM walked you through. I made an example working with the sapply function to loop through the data.
First, I calculated the distance between each pair of points in sequence for the whole dataset based on your code. I used with to avoid having to use dollar sign notation or the extract function [. You can see the vector output pairdist is 1 unit shorter than the number of rows in the dataset.
pairdist = sapply(2:nrow(bird), function(x) with(bird, trackDistance(Longitude[x-1], Latitude[x-1], 
                                 Longitude[x], Latitude[x], longlat=TRUE) ))
totdist= sapply(seq(1,length(pairdist)-3, by = 4), function(x) sum(pairdist[x:(x+3)]))
straight = sapply(seq(1, nrow(bird)-4, by = 4), function(x) with(bird,trackDistance(Longitude[x],
                                                                    Latitude[x], 
                                 Longitude[x+4], Latitude[x+4], longlat=TRUE) ))
bird$Sinuosity = c(NA, rep(totdist/straight, each = 4), 
                rep(NA, length(pairdist)-4*floor(length(pairdist)/4)))

Applying function to data.frame generates NAs while applying it to columns works


Tag : r , By : Jason Haar
Date : March 29 2020, 07:55 AM
this will help The operation will work as intended if you use sapply instead instead of apply:
sapply(frame, weight.fun)
#         x  y
#  [1,] 290 10
#  [2,] 790 10
#  [3,] 410 10
#  [4,] 860 10
#  [5,] 910 10
#  [6,]  50 20
#  [7,] 500 20
#  [8,] 830 20
#  [9,] 510 20
# [10,] 420 20
as.matrix(frame)
#        x    y  
#  [1,] "29" "1"
#  [2,] "79" "1"
#  [3,] "41" "1"
#  [4,] "86" "1"
#  [5,] "91" "1"
#  [6,] " 5" "2"
#  [7,] "50" "2"
#  [8,] "83" "2"
#  [9,] "51" "2"
# [10,] "42" "2"

Applying window function in one query without CTE


Tag : sql-server , By : chintown
Date : March 29 2020, 07:55 AM
it helps some times Demo with some test data on how this works
 select top (1) with ties
                p1.Gid  Gid,
                p1.id id,
                p1.prod prod,
                p1.orderdate orderdate,
                p2.shipdate shipdate       
          from shpro p1 inner join shpro p2
          on p1.id=p2.id
          where cast(p1.orderdate as DATE)>GETDATE() and cast(p1.shipdate as DATE)<GETDATE()-1
          order by
          Rank() over (partition by p1.prod order by p1.id desc) 
  CREATE TABLE #TEST
    (
    Id INT,
    Name VARCHAR(10)
    )

    Insert Into #Test
    select 1,'A'
    Union All
    Select 1,'B'
    union all
    Select 1,'C'
    union all
    Select 2,'D'



select top 1 with ties id,name
from #test
order by row_number() over (partition by id order by id)

Fitting data to Faddeeva function using python's optimize.leastsq() and optimize.curve_fit


Tag : python , By : adrianmooreuk
Date : March 29 2020, 07:55 AM
Hope this helps Hello Stackoverflow community, , The third comment by PRMoureu on my question fixed the problem.

FFT - Applying window on PCM data


Tag : c , By : ganok_tor
Date : March 29 2020, 07:55 AM
Related Posts Related QUESTIONS :
  • How to add CVS directories recursively
  • Web App Beta
  • Virtual machine supporting multiple displays
  • LINQ to SQL strings to enums
  • How do I configure eclipse (zend studio 6) to hint and code complete several languages?
  • Recommended Fonts for Programming?
  • Automating VMWare or VirtualPC
  • How do I use more than one OpenID?
  • Domain Specific Language resources
  • Vista or XP for Dev Machine
  • Where can I get the Windows Workflow "wca.exe" application?
  • How to learn ADO.NET
  • What is a good barebones CMS or framework?
  • What's the Developer Express equivalent of System.Windows.Forms.LinkButton?
  • Is it acceptable for invalid XHTML?
  • Anyone soloing using fogbugz?
  • Using MBUnit in TeamCity
  • How to setup a crontab to execute at specific time
  • Linking two Office documents
  • How can I detect if a browser is blocking a popup?
  • Apply filter to t-foreach
  • Lotus Notes convert formula to agent
  • Problem with configuration of odoo for working with pycharm
  • Unexpected number of arguments when calling EM_JS function
  • What does the function state_fidelity() do in Qiskit?
  • Azure function with IoT Hub trigger get send device
  • Cluster similar words using word2vec
  • Add elements into an array then find distinct,
  • Angular8 - error TS2304: Cannot find name
  • Manually polling streams in future implementation
  • How do you perform receipt validation for Android on Xamarin Forms
  • What is the rule behind instruction count in Intel PIN?
  • How to draw some text when click a button in wxWidgets?
  • Restrict entity types in Spacy NER
  • How to animate CN1 Slider progress on load
  • Issue with bwa mem process not running on all output files from previous process
  • Delete videos from playlist using YouTube API
  • How to restart the Hyperledger Composer Playground locally
  • How to configure email alerts in Zabbix Server?
  • Simulator is not working for smart home action with all configured session
  • How parse data using join on Objection.js
  • Assign an array to a property in a Chapel Class
  • Netlogo: How can I obtain the accumulate value in Netlogo?
  • How to populate datasource for listview using api response in react native?
  • Why does gmail API when using history.list method send message ids without the field what action has been preformed on t
  • How to run an append query in ms access vba as part of a transaction
  • Wrong dates shown in Jekyll-based blog
  • How to concatenate two document lists in a webMethods flow service?
  • How to properly set up a site map addition in a customization project
  • IBM Content Collector error calling external Web Service
  • Is (0*1*)* equal to (0 | 1)*?
  • How do I configure phpMyAdmin to start with a blank sql query from the SQL tab?
  • "Objects are not valid as a React child" Redux error when conditionally connecting a component?
  • Displacy results are not showing
  • Strapi / Graphql. What am i doing wrong?
  • How to add Search bar for django template?
  • lookup in presto using single column against a range in lookup table
  • How can you stop videos being stolen on website?
  • Can't code substitution happen in Hybrid Flow?
  • Removed widget remains on parent
  • shadow
    Privacy Policy - Terms - Contact Us © scrbit.com