logo
down
shadow

Fast CSV parser in C++


Fast CSV parser in C++

Content Index :

Fast CSV parser in C++
Tag : cpp , By : arbeitandy
Date : December 05 2020, 12:24 PM


Comments
No Comments Right Now !

Boards Message :
You Must Login Or Sign Up to Add Your Comments .

Share : facebook icon twitter icon

Fast, lightweight XML parser


Tag : java , By : hellboy32
Date : March 29 2020, 07:55 AM
should help you out No, you should not try to write your own XML parser for this.
SAX itself is very lightweight and fast, so I'm not sure why think it's too much. Also using a string buffer would actually be much less scalable then using SAX because SAX doesn't require you to load the whole XML file into memory to use it. I've used SAX to parse through multigigabyte XML files, which you wouldn't be able to do using string buffers on a 32 bit machine.
Document d = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(new FileInputStream("file.xml"));

Delphi - Fast(er) XML parser


Tag : xml , By : Joe
Date : March 29 2020, 07:55 AM
help you fix your problem First let me tell you that you're optimizing the wrong thing here: unless you're doing this for recreational purposes, then your approach is wrong. XML is not a difficult format but it does have it quirks and it takes it's liberties. It's a format designed for data exchange between foreign applications, so the emphasis needs to be put on COMPATIBILITY, not on SPEED! What good is a non-standard ultra-fast parser that gives the wrong result when confronted with a slightly altered XML file?
If you can find a XML parsing LIBRARY that's guaranteed to be compatible with anything out there that can parse your data at HALF the speed your HDD can read it, then simply implement a producer-consumer multi-threaded application where one thread constantly reads the data from disk while the other two simply do the parsing. In the end you'll only be limited by the speed of the HDD while maintaining compatibility. If you're only looking for speed you're liable to make mistakes, skip XML features, depend on certain particularities of the sample XML file you're dealing with. Your application is likely to break for numerous reasons.
program Project28;

{$APPTYPE CONSOLE}

uses SysUtils, DateUtils, Windows;

const SampleData =
    '<node>'#13#10+
    '  <datatype1>randomdata</datatype1>'#13#10+
    '  <datatype2>randomdata</datatype2>'#13#10+
    '  <datatype3>randomdata</datatype3>'#13#10+
    '  <datatype4>randomdata</datatype4>'#13#10+
    '  <datatype5>randomdata</datatype5>'#13#10+
    '  <datatype6>randomdata</datatype6>'#13#10+
    '  <datatype7>randomdata</datatype7>'#13#10+
    '  <datatype8>randomdata</datatype8>'#13#10+
    '  <datatype9>randomdata</datatype9>'#13#10+
    '  <datatype10>randomdata</datatype10>'#13#10+
    '  <datatype11>randomdata</datatype11>'#13#10+
    '  <datatype12>randomdata</datatype12>'#13#10+
    '  <datatype13>randomdata</datatype13>'#13#10+
    '  <datatype14>randomdata</datatype14>'#13#10+
    '  <datatype15>randomdata</datatype15>'#13#10+
    '  <datatype16>randomdata</datatype16>'#13#10+
    '  <datatype17>randomdata</datatype17>'#13#10+
    '  <datatype18>randomdata</datatype18>'#13#10+
    '  <datatype19>randomdata</datatype19>'#13#10+
    '  <datatype20>randomdata</datatype20>'#13#10+
    '</node>'#13#10;
const NodeIterations = 10000;

type
  TDummyRecord = record
    D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13,
      D14, D15, D16, D17, D18, D19, D20: string;
  end;
  TDummyRecordArray = array[1..NodeIterations] of TDummyRecord;

procedure ParseDummyXMLToRecordArray(const InputText:string; var A: TDummyRecordArray);
var PInputText: PChar;
    cPos, TextLen: Integer;
    C: Char;
    State: Integer;

    tag_starts_at: Integer;
    last_payload_starts_at: Integer;
    FlagEndTag: Boolean;

    NodeName, Payload: string;

    cNode: Integer;

const st_not_in_node = 1;
      st_in_node = 2;
begin
  cPos := 1;
  TextLen := Length(InputText);
  PInputText := @InputText[1];
  State := st_not_in_node;
  last_payload_starts_at := 1;
  cNode := 0;

  // This is the lexer/parser loop. It's a finite-state machine with only
  // two states: st_not_in_node and st_in_node
  while cPos < TextLen do
  begin
    C := PInputText[cPos-1];
    case State of

      // What happens when we're NOT currently inside a node?
      // Not much. We only jump to st_in_node if we see a "<"
      st_not_in_node:
        case C of
          '<':
            begin
              // A node starts here. Switch state and set up some simple
              // flags.
              state := st_in_node;
              tag_starts_at := cPos + 1;
              FlagEndTag := False;
            end;
        end;

      // What happens while inside a node? Again, not much. We only care about
      // the "/" - as it signals an closing tag, and we only care about the
      // ">" because that means the end of the ndoe.
      st_in_node:
        case C of
          '/': FlagEndTag := True;
          '>':
            begin
              // This is where the magic haepens. We're in one of possibly two states:
              // We're ither seeing the first <name> of a pair, or the second </name>
              //
              if FlagEndTag then
                begin
                  // This is the closing pair of a tag pair, ie, it's the </NodeName> What we'll do
                  // depends on what node is closing, so we retreive the NodeName:
                  NodeName := System.Copy(InputText, tag_starts_at+1, cPos - tag_starts_at-1);
                  if NodeName <> 'node' then // SAMPLE-DATA-SPECIFIC: I know I don't care about "node" tags.
                  begin
                    // SAMPLE-DATA-SPECIFIC: I know there are only two kinds of nodes:
                    // "node" and "datatypeN". I retreive the PAYLOAD for the node because
                    // I know it's not "ndoe" and I know I'll need it.
                    Payload := System.Copy(InputText,last_payload_starts_at, tag_starts_at - last_payload_starts_at -1);
                    // Make sure we're dealing with a valid node
                    if (cNode > 0) and (cNode <= High(A)) then
                      begin
                        // Based on NodeName, copy the Payload into the appropriate field.
                        if NodeName = 'datatype1' then A[cNode].D1 := Payload
                        else if NodeName = 'datatype2' then A[cNode].D2 := Payload
                        else if NodeName = 'datatype3' then A[cNode].D3 := Payload
                        else if NodeName = 'datatype4' then A[cNode].D4 := Payload
                        else if NodeName = 'datatype5' then A[cNode].D5 := Payload
                        else if NodeName = 'datatype6' then A[cNode].D6 := Payload
                        else if NodeName = 'datatype7' then A[cNode].D7 := Payload
                        else if NodeName = 'datatype8' then A[cNode].D8 := Payload
                        else if NodeName = 'datatype9' then A[cNode].D9 := Payload
                        else if NodeName = 'datatype10' then A[cNode].D10 := Payload
                        else if NodeName = 'datatype11' then A[cNode].D11 := Payload
                        else if NodeName = 'datatype12' then A[cNode].D12 := Payload
                        else if NodeName = 'datatype13' then A[cNode].D13 := Payload
                        else if NodeName = 'datatype14' then A[cNode].D14 := Payload
                        else if NodeName = 'datatype15' then A[cNode].D15 := Payload
                        else if NodeName = 'datatype16' then A[cNode].D16 := Payload
                        else if NodeName = 'datatype17' then A[cNode].D17 := Payload
                        else if NodeName = 'datatype18' then A[cNode].D18 := Payload
                        else if NodeName = 'datatype19' then A[cNode].D19 := Payload
                        else if NodeName = 'datatype20' then A[cNode].D20 := Payload
                        else
                          raise Exception.Create('Unknown node: ' + NodeName);
                      end
                    else
                      raise Exception.Create('cNode out of bounds.');
                  end;
                  // Repeat :-)
                  state := st_not_in_node;
                end
              else
                begin
                  // Node start. Retreive node name. I only care about the start of the "NODE" - if I see that
                  // I'll increment the current node counter so I'll go on filling the next position in the array
                  // with whatever I need.
                  NodeName := System.Copy(InputText, tag_starts_at, cPos - tag_starts_at);
                  last_payload_starts_at := cPos+1;
                  if NodeName = 'node' then Inc(cNode);
                  state := st_not_in_node;
                end;
            end;
        end;
    end;
    Inc(cPos);
  end;
end;

var DataString: string;
    SB: TStringBuilder;
    i: Integer;
    DummyArray: TDummyRecordArray;
    T1, T2, F: Int64;

begin
  try
    try
      // Prepare the sample string; 10.000 iterations of the sample data.
      SB := TStringBuilder.Create;
      try
        for i:=1 to NodeIterations do
          SB.Append(SampleData);
        DataString := SB.ToString;
      finally SB.Free;
      end;

      // Invoke the simple parser using the string constant.
      QueryPerformanceCounter(T1);

      ParseDummyXMLToRecordArray(DataString, DummyArray);

      QueryPerformanceCounter(T2);
      QueryPerformanceFrequency(F);
      WriteLn(((T2-T1) * 1000) div F);

      // Test parse validity.
      for i:=1 to NodeIterations do
      begin
        if DummyArray[i].D1 <> 'randomdata' then raise Exception.Create('Bug. D1 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D2 <> 'randomdata' then raise Exception.Create('Bug. D2 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D3 <> 'randomdata' then raise Exception.Create('Bug. D3 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D4 <> 'randomdata' then raise Exception.Create('Bug. D4 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D5 <> 'randomdata' then raise Exception.Create('Bug. D5 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D6 <> 'randomdata' then raise Exception.Create('Bug. D6 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D7 <> 'randomdata' then raise Exception.Create('Bug. D7 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D8 <> 'randomdata' then raise Exception.Create('Bug. D8 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D9 <> 'randomdata' then raise Exception.Create('Bug. D9 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D10 <> 'randomdata' then raise Exception.Create('Bug. D10 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D11 <> 'randomdata' then raise Exception.Create('Bug. D11 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D12 <> 'randomdata' then raise Exception.Create('Bug. D12 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D13 <> 'randomdata' then raise Exception.Create('Bug. D13 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D14 <> 'randomdata' then raise Exception.Create('Bug. D14 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D15 <> 'randomdata' then raise Exception.Create('Bug. D15 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D16 <> 'randomdata' then raise Exception.Create('Bug. D16 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D17 <> 'randomdata' then raise Exception.Create('Bug. D17 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D18 <> 'randomdata' then raise Exception.Create('Bug. D18 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D19 <> 'randomdata' then raise Exception.Create('Bug. D19 doesn''t have the proper value, i=' + IntToStr(i));
        if DummyArray[i].D20 <> 'randomdata' then raise Exception.Create('Bug. D20 doesn''t have the proper value, i=' + IntToStr(i));
      end;

    except on E: Exception do Writeln(E.ClassName, ': ', E.Message);
    end;
  finally
    WriteLn('ENTER to Exit');
    ReadLn;
  end;
end.

Fast parser for logs


Tag : perl , By : Lauren Kirschner
Date : March 29 2020, 07:55 AM
Any of those help I answered this in my response to your comment on my solution to your previous question.
Here is the answer I gave before, together with the new data that you have shown in this question.
use strict;
use warnings;

my $string = 'id=firewall time="2010-05-09 16:07:21 UTC" 1.1.1.1 ...';

my @fields = $string =~ / (?: "[^"]*" | \S )+ /xg;

print "$_\n" for @fields;
id=firewall
time="2010-05-09 16:07:21 UTC"
1.1.1.1
...

Is there a fast parser for date


Tag : r , By : Josh
Date : March 29 2020, 07:55 AM
will help you For datetimes fasttime provides very fast parsing to POSIXct , Given
## the following two (here three) lines are all of fasttime's R/time.R
fastPOSIXct <- function(x, tz=NULL, required.components = 3L)
  .POSIXct(if (is.character(x)) .Call("parse_ts", x, required.components)
           else .Call("parse_ts", as.character(x), required.components), tz)
## so we suggest to just use it, and convert later
fastDate <- function(x, tz=NULL)
  as.Date(fastPOSIXct(x, tz=tz))
R> library(microbenchmark)
R> library(fasttime)
R> d <- rep("2010-11-12", n=1e4)
R> microbenchmark(fastDate(d), as.Date(d), times=100)
Unit: microseconds
        expr    min      lq    mean  median      uq     max neval cld
 fastDate(d) 47.469 48.8605 54.3232 55.7270 57.1675 104.447   100  a 
  as.Date(d) 77.194 79.4120 85.3020 85.2585 87.3135 121.979   100   b

R> 

Fast CSV parser with low GC load


Tag : java , By : browe
Date : March 29 2020, 07:55 AM
Related Posts Related QUESTIONS :
  • Convert QString to QJsonArray
  • Data exchange finished in CPropertyPage::OnOK?
  • Template member specialization in template class
  • Is it not possible to assign a struct to an index of a vector?
  • Why is empty unordered_map.find not returning end()?
  • Template argument deduction for inheriting specializations
  • dlopen undefined reference
  • Member function of class with template arguments and default arguments outside class
  • Is it possible to implement a non-owning "slightly smart" pointer on top of standard weak pointers?
  • how to configure the AcquireCredentialsHandleA correctly
  • Using private versions of global extern variables with OpenMP
  • Eigen Block wrong amount of columns and rows
  • Memory alignment rules in inheritance
  • Is nullptr falsy?
  • tm_wday returns a large integer outside 0-6 range
  • Scope a using declaration, inside a header
  • How to specify constructor's template arguments inside a new expression?
  • Sort an array via x86 Assembly (embedded in C++)?? Possible?
  • How to Replace only Part of the Variable using #define
  • How do you compare the performace of valarrays vs built-in arrays?
  • Is it normal for C++ static initialization to appear twice in the same backtrace?
  • c++ generate a good random seed for psudo random number generators
  • Why isn't my operator overloading working properly?
  • Getting meaningful error messages from fstream's in C++
  • C++: Converting Julian dates to Gregorian
  • Could someone explain this interesting behaviour with Sleep(1)?
  • Is it possible to roll a significantly faster version of modf
  • Updating pointer using signals and slots
  • How are classes more secure than structures?
  • finding "distance" between two pixel's colors
  • C++ Greatest Number Verification
  • Why does my token return NULL and how can I fix it?(c++)
  • C++ enforce conditions on inherited classes
  • what happened if an exception is not captured?
  • Redundant naming in C/C++ typedefs/structs
  • question about STL thread-safe and STL debugging
  • killing a separate thread having a socket
  • Returning the size of available virtual memory at run-time in C++
  • Parallel computing for integrals
  • How do I force my std::map to deallocate memory used?
  • C++ Templates: implicit conversion, no matching function for call to ctor
  • Adding python script to c++ project
  • C++ private pointer "leaking"?
  • Initializing Primitive Array to One Value
  • how is push_back implemented in STL vector?
  • C++ Profiling: KiFastSystemCallRet
  • Multiple rows with a single INSERT in SQLServer 2008
  • Use super class's address/pointer in initialization list
  • double fork using vfork
  • Convert char pointer (char*) to struct
  • Does anyone have good tips for using pc-lint?
  • C++ How fast is passing around objects?
  • template type's ctor arguments at instantiation time
  • Get list of windows in taskbar on Windows 7?
  • Curl connection options
  • Best cross-platform solution for network server?
  • simple c++ file opening issue
  • Switching from C++ (with a lot of STL use) to C for interpreter building
  • How can I access the JPEG image pixels as a 3D array like we do in MATLAB?
  • What wording in the C++ standard allows static_cast<non-void-type*>(malloc(N)); to work?
  • shadow
    Privacy Policy - Terms - Contact Us © scrbit.com