You mentioned that in your deserialization function that you’re converting from a denormalized csv to an in-memory list.

Basically, from what I’ve read, you’ve managed to optimize the size on disk of data by transitioning from xml to binary (w/ gzip thrown on top). I agree that this is probably a good thing provided the data is small enough that the overhead of decompression doesn’t hurt. That being said, there are some very fast XML parsers available that should be able to rip through 500M of data in well under 100s.

However, I’d be curious to know if the in-memory footprint is still upwards of 500M?

Partitioning the data is often a good position to take with problems like this. It’s one of the core tenets behind Map/Reduce and Google has obviously shown it to scale well. Even with much smaller data sets it’s nice to be able to have the flexibility to say ‘I haven’t used this data in awhile I’m going to evict it from memory’, choosing to pay the penalty to reload at a later time. It’s unfortunate that this strategy doesn’t work for your particular application.

Cheers.

I have been an xml guy for years, and I love working with it because it is so functional. With that said, I did not want to move away from xml because I am so familiar with it. In .NET if you want to be able to edit an xml object you have to use a XmlDocument object. This slows things down and the XPath we used to search through the lump was just too slow.

If you only need read access to an xml file, then you can use faster parsers like an XPathDocument and XPathNavigator…unfortunately, we couldn’t go “read only”.

I haven’t heard of Map/Reduce before…gonna have to look that one up. Thanks!

David

David

About the 250MB vs 500MB… could it be an encoding related issue?

I mean, doesn’t Unicode encoded strings (like UTF-8) use double bytes for characters, when other encodings (like ISO-8859-1) use only one?

If i’m understanding it right, that means any utf-8 encoded file has double data space used than any same iso-8859-1′d file.

Maybe when serialized that penalty is lost if your XML file is unicode.
I’m just a high level programmer, not a byte expert, so just guessing.

I do know some encoding formats are better than others, but I have never really sat down and thought about it.

In my case, I just trust the .NET serializer to make the right choice, but if space is at a premium, that might be a good way to be more efficient.

Thanks for the advice!

David

Please, let me tell you a few tips about that encoding thing i’ve learned working with XML in Microsoft enviroments.

As i told you before, we speak spanish here. So, we got a few non-US characters, like the “ñ” and the tilded vowels. Those characters gave us always encoding problems. That’s why and where UTF-8 comes in: it’s supposed to be universal. Therefore, it’s spected that contemporary XML parsers work in UTF-8 by default, meaning that your file may be utf-8 encoded (and have a size of 500M when using 250M chars).

It’s easy to know this, by opening the file in Notepad, selecting “Save As”, and checking what file encoding does it shows by default; that’s the encoding of your file.
Unfortunately, that’s not that “easy” to check that way with a 500MB text file (notepad will surely hang).

Visual Studio 6, and windows up to server 2003 (don’t know about vista or 2008), handle files in ANSI by default (wich is but an ASCII with MSDOS characters and 16 colours). To write an UTF-8 file, you gotta do a few things.
Therefore, if you save an UTF-8 encoded XML into an ANSI file, it does not work (the system missuderstand some chars).
Also, by default, MSXML up to version 3 (maybe 6 as well, gotta try) wrote XML code in UTF-16 encoding (wich is ANSI compatible, as well as ISO-8859-1).
That’s how the things worked before .NET in the Microsoft’s XML world: you HAD to put this parameters by hand, and know how to do it.

I assume .NET parsers do work in UTF-8 by now, because of the mess that was all that encoding thing, and therefore you might have an utf-8 file.

Don’t know about your version of Visual Studio, nor you framework version, but is good to know this kind of issues everytime you work with an XML file and Microsoft technologies. I learned all this stuff in the hard way.

Best.

Now, i’m a bit new on this, so i got 1 quick question.

I get all this code to work, but i need my program to write to the file, close it, then continue on the same file later in the code.

when i use it as it is it only overwrited the previous data when i try to continue on it.

I tried changing the “FileMode.Create” to “FileMode.Open” but didn’t change anything.

so, how do i make it continue on an exsisting file?

David

Sorry it didn’t work out for you. Our files are about 100 MB if uncompressed and this method works wonders. However, for super huge data mining ops, then it sounds like you have a good way of doing it. Congrats.

David

I have got to read a text file(size > 500MB to GBs also) in c#. I have to parse this text file.
the file consists of lines with comma sepearated and some lines not required to parse.

I need to read the log file ( right now doing b stream reader ReadLine() method )
1. then form a data row to build data table.
2. I need to bind this huge data table to grid view.
3. Then Export this grid to Excel Sheet.

Am using stream reader to read the file.
the application hangs or sometimes becomes not responding…

I tried to do the operation in seperate thread then also the behavior didc’t change much (Hanging)

and also Binding to Grid is a good thing to do ?

Thanks,
Ramki

We have a grid that can be as large as 3,000 rows and 1,500 columns (STUPID big)…its a beast, but the key is to use a virtual grid. Most third party grids has some form of virtual grid.

What that does is tells the grid…here is your data source over here (which points to the cache file I talk about in this article), but I only want you to load what the user can see on the screen. To say this dramatically speeds things up is an understatement.

We use the Infragistics grid on our project and it has been really solid.

Hope that helps.

David

Thanks for your reply.

I tried to add to grid row by row after every Readline() of the log.
But this is also consuming time as binding altogether at once after whole reading operation had completed.

Yes having an option to view a bunch of rows is good design. And user can see only a bunch of records at any time. But this is not the requirement.

virtual grid can be done in v 2005 without 3rd party tools?
->infragists: to use these tools we need to buy right?

Ramki

If you want to use infragistics, it does cost money. If you are a freelancer, then it is probably a bit expensive, but if you are working with a company or a good client, it is money well spent…

David

I’m trying to wrap my mind about your whole process here. Tell me if this is correct:

1) Client app communicates to server app requesting data.
2) Server app requests data from database.
3) Server app compresses data and ships off to client app.
4) Client app decompresses data for use.

If I’m way off here, please let me know. I may be working on a project that involves several gigs of data and I’m looking for a solid solution.

David

Good Algorithm, like i read here above, i find it difficult to handle amount large data, like > 500 MB - X GB big from an database.
But the nice thing is we dosent need all that data at once, but i want to have it have “on-the-fly”, is it better to index the database and hope that the sql query will quick be enough to send over data in realtime when the program ask for it or is it better to cache it up and compress it in-memory, or the last thing, maby is better to cache it up to an virtual file, then using filemapping.

clue: this data is for an own Chart program, where i want to show one or more lines, so the problem here is when i want to show data on the chart, is that,it dosent have to show all the data at once, but when you zoom in you want some connection to more detail-data that can be shown on the chart.
But what technic can be quick enough for that?