A small change in the Australian data has caused the program to dump the Australian imports. All other imports should run OK. A new version of the program should be released later today – hopefully before or by 2200 hrs UTC.
Due to problems where data is only imported manually because of some very short data times (mainly greece) I have changed the import control to ignore missing earthquake data.
When an import is performed the program seeks out existing data and compares it to the incoming data to find duplicates and reject them, retaining just the new, and of course compares for changes to existing data as well.
In order to do this there has to be existing data within the time span of the new import thus if the new import is for 48 hours then there has to be existing data in the past 48 hours, even if it is only one event. Once data was not found for the past 48 hours (in this example) the program would cease to collect data from that source permanently.
The reason for this was because if the lack of data was because of a data/connection error then many duplications could potentially occur when the error resolved. Most providers data can be successfully parsed to reduce this problem but in particular those that do not use IDs for events, and for some reason I have not determined the New Zealand data, are very prone to duplications.
There is a de-duplication facility in the Data Centre and this should be run at least once a month to verify the data integrity. This will be even more important now the restriction on finding no existing data has been removed.