Home › Forums › English › General Discussions regarding BT747

How to convert large files?

quics's picture
Tue, 06/22/2010 - 00:19 — quics

I am trying to convert large GPX files from our bicycle tour from Alaska to Argentina to a html file to use it on our website (www.radausflug.org).

The GPX file I currently need to convert is 33MB and was created by GPSBabel from Columbus V900 .CSV and  iBlue 747 .bin files.

I already filtered it to include only one point every 50 m.

BT747 encounters the following error during the conversion: java.lang.OutOfMemoryError: Java heap space

I previously used to simply join all my .CSV files and convert the resulting file to HTML which worked fine for filesizes of as much as 700mb

Since I lost my Columbus V900 and am now using a iBlue 747 i have to join a few .csv and a few .bin files in another way.

Tried to convert them all to GPX, filter and join them all with GPSBabel and convert them to HTML with BT747 - however no luck there.

Is there a way to increase the useable memory for Java or change any settings to make it work?

Also tried to convert all files to the "Data logger iBlue 747 csv" format but BT747 doesn't seem to understand that format.

Any help welcome!

thanks, Phil

‹ uploading waypoints to data logger How to convert large files? ›
Tue, 06/22/2010 - 01:01 — mdeweerd
mdeweerd's picture

Hi It is possible to

Hi

  1. It is possible to increase memory for BT747.  I created http://soft.bt747.org/BT747_J2SE_Latest_rxtx2_2p2_big.jnlp for the occasion.  I defined 1GB of heap memory in stead of the regular 192MB.
  2. You can increase that in the startup script too.  For example, you can see 'SET  MEM_HEAP_OPTION=-Xmx192m' in the windows 'run_j2se.bat' file.  That indicates to set the heap limit to 192 MB (instead of the default 64MB).  You can change that number.  The other startup scripts have a similar limit.
  3. Send me a sample of the CSV file you converted to and I'll see what I can do to support it.  I guess it is simple to adjust BT747.  CSV will use less memory than GPX.   Reading GPX relies on the standard functions to read XML files and these tend to read the entire GPX structure in memory which requires a lot of memory.  In the case of CSV, the input file is read line by line.  My email is in the application (about function) or use bt 7 4 7 A T bt 7 4 7  dot  org  (no spaces).

Regarding the HTML: if you want to get rid of the track menu at the bottom of the default output, just add '#footer { display : none; }' in the '<style type="text/css">' section of the file.Further, I guess that you have read the html generation instructions on this site (especially to know how to insert the google map key).

Tue, 06/22/2010 - 01:15 — mdeweerd
mdeweerd's picture

I also had a look at your

I also had a look at your site.

The speed at which this map and tracks is shown is surely enough already amazing.  You might be able to optimize that further by reducing the number of tracks in the html.  The number of tracks is determined by the track splitting choice.  You could increase the time limit to start a new track.  The side effect is that the entire track would be connected, but that is I think not a  problem in your case given that there seems to be a continuity between days.  It is surely something to try to get less than 177 tracks.

Tue, 06/22/2010 - 01:29 — mdeweerd
mdeweerd's picture

I add one other suggestion to

I add one other suggestion to my initial post:

4. Rather than concatenating the files before feeding them in BT747, let BT747 'concatenate them'.  With the GPX files being smaller, the required memory is also less.  You can drag and drop the files to BT747.  They will appear in the 'files to tag' table of tracks.  BT747 will order the tracks and use them to create the output files.

Tue, 06/22/2010 - 07:40 — quics
quics's picture

Hi Mario! Thanks for the fast

Hi Mario!
Thanks for the fast reply! The increased heap space solved my problem.
I sent you one of the CSV files that GPSBabel creates if I use the output type: "Data logger iBlue 747 csv"
which is not recognized by BT747 for some reason.
Thanks for the tip of how to get rid of the track menu too! very useful!
cheers from Panama, phil
www.radausflug.org

Tue, 06/22/2010 - 08:51 — mdeweerd
mdeweerd's picture

Hi Phil I received the CSV

Hi Phil

I received the CSV file.  It did not take long to understand the issue.  BT747 did interpret the CSV, but you have 'Unknown' in the 'VALID' column.  That value resulted in the internal valid field being set to '0'.  As a result, the filter operated (because the value is 'set').  But 'Unknown' does not appear in the list of valid types that you can keep.  Hence no positions was selected.

I made a quick fix: I ignore 'Unknown' in the CSV input.  This is not perfect but it will do.  The perfect solution is to add 'Unknown' everywhere which has quite some impact in the code.

The version containing the update is 2.X.1751 .

 

Now regarding the CSV that Gpsbabel spits out, there are a few imperfections.  'Unknown' should have been 'Unknown mode' according to my code and each line should end with a ',' (because the header does so too).  But that does not hinder BT747 ;-).

Mon, 06/28/2010 - 05:35 — Anonymous

Thanks a lot for the fast

Thanks a lot for the fast fix!

Works fine now even with the GPSBabel CSVs!

I tried to find the "run_j2se.bat" file you mentioned but couldn't find it anywhere on my system. I'm running Win XP and the latest Java version.

Was no problem to use your webstart file for BT747 with large GPX files but I didn't find any other place to change the heap space value.

thanks, phil

Mon, 06/28/2010 - 08:42 — mdeweerd
mdeweerd's picture

Hi I checked the zip

Hi

I checked the zip distribution and the 'run_j2se.bat' file as well as 'run_j2se64.bat' are in there.  These do not come with the webstart.

The zip distribution is on sourceforge.

Kind regards

Mario