View Log Files


Home View Menu Web Menus Data Menu Tools Menu Data Centre Services Alerts Tool

View Log Files is accessed from the View Menu in QVSData

view-log-files-on-menu

The log files screen is a small screen with a list of all the log files in the log files folder which have the extension ‘log’ or ‘txt’

log-files-screen

First, this is taken from my system which is a Super Pro version. Sounds good but just means it is the SQL Server version of Pro with a few personal twiddles and maintenance tasks.

The location of the log files varies according to your Windows operating system. Mine is XP. Each version of QVSData has it’s own folder – ‘Standard@, ‘Pro’, ‘ProPlus’ (Additional mapping bits) and ‘SuperPro’

log-files-screen02

Log files are written for every data import you do whether this is a manual import of by the Windows data import service if you have installed/started it.

There are relatively sparse file for the server and data import controller, but the data imports from providers should have reasonably full information.

So, to continue, follow the instruction below. (Try a USGS file as that is what I shall describe next)

log-files-screen03

The text viewer will open and you will see something like this:

text-screen-view
(Click the Image for a slightly larger/clearer version)

The second line of text is the important part! Below you can move the slider at the bottom of the block to see the whole line.

=====================================================================================
Diagnostics and / or processing errors for run dated 5/14/2013 10:35:59 AM
=====================================================================================
No warnings were issued and no errors were detected.
=====================================================================================
DIAGNOSTICS LINES
[5/14/2013 10:35:55 AM] Diags: (5/14/2013 10:35:54 AM) Getting the US07M1 data file from the server. (S:75CFAED8E3C108A73DD4)
[5/14/2013 10:35:55 AM] Diags: (5/14/2013 10:35:55 AM) Store file is C:\WINDOWS\Temp\usgs10W.json (S:75CFAED8E3C108A73DD4)
[5/14/2013 10:35:55 AM] Starting US07M1 processing (S:75CFAED8E3C108A73DD4)
[5/14/2013 10:35:55 AM] USGS GeoJSON Data import has started. (S:5B051A5E921C9F878A5E)
[5/14/2013 10:35:57 AM] Latest date: 2013-05-14T09:15:28.200Z (S:5B051A5E921C9F878A5E)
[5/14/2013 10:35:57 AM] Earliest date: 2013-05-07T09:39:49.510Z (S:5B051A5E921C9F878A5E)
[5/14/2013 10:35:57 AM] USGS GeoJSON Data import has ended. 868 records processed. (S:5B051A5E921C9F878A5E)
[5/14/2013 10:35:57 AM] SELECT * FROM tbl_USGSDataQVS WHERE (DateTimeAsString >= '2013-05-07T09:39:49.510Z' AND DateTimeAsString <= '2013-05-14T09:15:28.200Z') AND FileSource = 'usgs' ORDER BY DateTimeAsString (S:DE221D311C3577D9A467)
[5/14/2013 10:35:57 AM] Commencing load of existing events for US07M1 (S:DE221D311C3577D9A467)
[5/14/2013 10:35:57 AM] Records for comparison loaded: 868 (S:DE221D311C3577D9A467)
[5/14/2013 10:35:57 AM] US07M1 - Existing quakes: Loaded OK (S:0329C5B0E77178468ACF)
[5/14/2013 10:35:57 AM] US07M1 - Number of events to process: 868 (S:0329C5B0E77178468ACF)
[5/14/2013 10:35:57 AM] Incoming Quake: ID NC71991301 was found in the list and was rejected as a duplicate. (S:0329C5B0E77178468ACF)
[5/14/2013 10:35:57 AM] Incoming Quake: ID MB28881 was found in the list and was rejected as a duplicate. (S:0329C5B0E77178468ACF)
[5/14/2013 10:35:57 AM] Incoming Quake: ID USC000GW9G was found in the list and was rejected as a duplicate. (S:0329C5B0E77178468ACF)
[5/14/2013 10:35:57 AM] Incoming Quake: ID CI15342457 was found in the list and was rejected as a duplicate. (S:0329C5B0E77178468ACF)
[5/14/2013 10:35:57 AM] Incoming Quake: ID CI15342449 was found in the list and was rejected as a duplicate. (S:0329C5B0E77178468ACF)
[5/14/2013 10:35:57 AM] Incoming Quake: ID USC000GW8Y was found in the list and was rejected as a duplicate. (S:0329C5B0E77178468ACF)

So what does it all mean?

Analysing the lines we get:

Diagnostics and / or processing errors for run dated 5/14/2013 10:35:59 AM

The date and time the run started. This could be in European or US format. Above it is in US format as these files were produced by the Windows service.

Manually produced files have the date in European format (below), or I believe in the format for your country – I have not checked that out! Note also that manual runs use the 24 hr clock and system runs use the default windows AM/PM format.

Diagnostics and / or processing errors for run dated 10/05/2013 22:57:12
=====================================================================================
No warnings were issued and no errors were detected.
=====================================================================================
DIAGNOSTICS LINES
[10/05/2013 22:56:57] Diags: (10/05/2013 22:56:55) Getting the US07M1 data file from the server. (P:75CFAED8E3C108A73DD4)

Back to the example.

[5/14/2013 10:35:55 AM] Diags: (5/14/2013 10:35:54 AM) Getting the US07M1 data file from the server. (S:75CFAED8E3C108A73DD4)

Contacting the USGS server for the file specified. This part at the end (S:75CFAED8E3C108A73DD4) tells me which routine performed this in case of errors. Every routine that has an external interface to the program has a unique identifier such as this.

[5/14/2013 10:35:55 AM] Diags: (5/14/2013 10:35:55 AM) Store file is C:\WINDOWS\Temp\usgs10W.json (S:75CFAED8E3C108A73DD4)

What was collected and where it was stored

[5/14/2013 10:35:55 AM] Starting US07M1 processing (S:75CFAED8E3C108A73DD4)
[5/14/2013 10:35:55 AM] USGS GeoJSON Data import has started. (S:5B051A5E921C9F878A5E)
[5/14/2013 10:35:57 AM] Latest date: 2013-05-14T09:15:28.200Z (S:5B051A5E921C9F878A5E)
[5/14/2013 10:35:57 AM] Earliest date: 2013-05-07T09:39:49.510Z (S:5B051A5E921C9F878A5E)
[5/14/2013 10:35:57 AM] USGS GeoJSON Data import has ended. 868 records processed. (S:5B051A5E921C9F878A5E)

The time the processing of the file started, the earliest and latest dates the routine found in the file and the number of records in total.

[5/14/2013 10:35:57 AM] SELECT * FROM tbl_USGSDataQVS WHERE (DateTimeAsString >= '2013-05-07T09:39:49.510Z' AND DateTimeAsString <= '2013-05-14T09:15:28.200Z') AND FileSource = 'usgs' ORDER BY DateTimeAsString (S:DE221D311C3577D9A467)
[5/14/2013 10:35:57 AM] Commencing load of existing events for US07M1 (S:DE221D311C3577D9A467)
[5/14/2013 10:35:57 AM] Records for comparison loaded: 868 (S:DE221D311C3577D9A467)
[5/14/2013 10:35:57 AM] US07M1 - Existing quakes: Loaded OK (S:0329C5B0E77178468ACF)

The program loads the existing earthquake records for the provider based on the dates found. Why? Because if we loaded more than those then many quakes would get deleted when they should not be. The next line tells you how many were loaded. In this instance it was the same number so you can assume that no quakes have been deleted and no new quakes added. If there were more quakes loaded than in the import then there have been deletions and the ones not found in the import will be deleted. Fewer quakes would indicate that new one were present.

[5/14/2013 10:35:57 AM] US07M1 - Number of events to process: 868 (S:0329C5B0E77178468ACF)

From this point on the program records what it found and what it did about it when comparing records between the import and the existing set.

[5/14/2013 10:35:57 AM] Incoming Quake: ID NC71991301 was found in the list and was rejected as a duplicate. (S:0329C5B0E77178468ACF)

So if they numbers are different and new quakes have been added what does it look like?

[10/05/2013 22:58:32] GeoNet (NZ) Events processed: 250 (P:D06BBEE1C184471B3F62)
[10/05/2013 22:58:32] SELECT * FROM tbl_GNTNZDataQVS WHERE (DateTimeAsString >= '2013-05-04T20:49:56.849Z' AND DateTimeAsString <= '2013-05-10T21:41:39.366Z') AND FileSource = 'newz' ORDER BY DateTimeAsString (P:DE221D311C3577D9A467)
[10/05/2013 22:58:32] Commencing load of existing events for NEWZ (P:DE221D311C3577D9A467)
[10/05/2013 22:58:32] Records for comparison loaded: 157 (P:DE221D311C3577D9A467)
[10/05/2013 22:58:32] NEWZ - Existing quakes: Loaded OK (P:0329C5B0E77178468ACF)
[10/05/2013 22:58:32] Incoming Quake: ID WEL2013P351091 was NOT found in the list.
INSERT INTO tbl_GNTNZDataQVS (QuakeID,NetworkSource,FileSource,Version,DateTimeAsString,Latitude,Longitude,Magnitude,MagType,Energy,NumReportingStations,Depth,Location,FeltFlag,Status,Revised) VALUES ('2013p351091','wel','newz','1','2013-05-10T21:41:39.366Z',-40.33922,176.39255,1.86,'ML',38904.514499428,0,53,'East Manawatu-S Hawkes Bay-North Madden Bank',0,'A',0) (P:0329C5B0E77178468ACF)
[10/05/2013 22:58:32] Incoming Quake: ID WEL2013P350837 was NOT found in the list.
INSERT INTO tbl_GNTNZDataQVS (QuakeID,NetworkSource,FileSource,Version,DateTimeAsString,Latitude,Longitude,Magnitude,MagType,Energy,NumReportingStations,Depth,Location,FeltFlag,Status,Revised) VALUES ('2013p350837','wel','newz','1','2013-05-10T19:26:15.135Z',-39.05708,176.693,1.7,'ML',22387.2113856834,0,5.1,'Kaimanawa-Hawkes Bay',0,'A',0) (P:0329C5B0E77178468ACF)

You get the FULL SQL necessary to insert the record into the table which may not be of use to you but tells me exactly where something may have gone wrong if it has.

Log files of the type ‘.txt’ – the providers – are overwritten each time an import is performed thus on a system running the service it will happen every 5 minutes if you are using the default.

The files of type ‘.log’ are added to as the day goes along, and when the time passes midnight a new log file is created. Logs are kept for 7 days.

Something you need to be aware of. The windows service downloads every five minutes and every file it downloads is stored and timed and dated. Once the TYPE of file reaches 1000 in number the system deletes older files so you will after a period of time have about 9 days worth of each provider file if you are running 24/7.

Manual files are NOT deleted and you may need to do so from time to time. If this is a problem let me know and I can automate the removal.

Other similar or related topics

Service Controller
QVS Service Data Imports
QVS Services menu

Comments policy

Advertisements

Please feel free to leave a reply. (All comments are moderated)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s