We’ve had data from two different phones combined into a single patrol after processing on SMART Desktop. There are two patrol tracks quite far apart from each. Two members are specified as leaders in the patrol summary.
The patrol is problematic for other reasons too in that it appears to cover multiple dates from 20 Jan to 2 Feb. We’ve had a few occasions recently with multi date patrols and I’ve assumed this is because a stop message has failed to arrive or be processed so susbequent new patrol data from the same phone gets tacked onto the end of the last. It seems the arrival of a new Start message is not enough to prompt it to consider the previous patrol to have ended, or the new Start message fails to process cos the previous was not stopped. Finding it hard to figure out. But surely the deviceID would prevent two phones being merged into one patrol. I don’t think we had this sort of error before.
Unfortunately I can’t very easily trace which data packages are involved to check if a relevant packages didn’t process. I can’t find a way to connect deviceID in the packages with a given Patrol in SMART.
We are operating in a challenging newtowrk environment where phone data may sit on phones without uploading between patrols, so it is possible a new patrol will start before the previous patrol has uploaded. So it is possible that a phone can come into network coverage with more than one completed patrol to upload, or even a completed patrol to upload and an active patrol underway. Presumbly if a start or stop message fails to process, or is not processed before new track data are uploaded then that new track data can get added to a former patrol and thereby starts adding new dates on to that patrol. And when the stop/start turns up in the queue they fail to process cos the date/Device ID combination is already occupied? Speculating wildly here! But we need to know what the limits are so we design our protocols to fit the limitations.
Running SMART 7.5.6 and Mobile 1.0.436
Many thanks for any suggestions
This is really odd. There can be challenging scenarios when patrol data gets uploaded or processed out of order, but I haven’t seen it result in this before. If there is an issue with processing a start or stop patrol observation, that would have been indicated when processing rather than combining them in a single patrol. Most commonly, when there isn’t a ‘patrol end’ message for a patrol with a concurrent active patrol from the same device, there are warnings when processing the trackpoints because they could match multiple patrols until that’s resolved.
All that is to say that this shouldn’t have happened, and I have no idea why it did. I’ll escalate this with the SMART development team to get more insight into this. Stay tuned for more question and info.
Would it be possible to get the data queue files from that approximate time range? If you look at the files, they should have the patrol start date/time in any of the observations, which you can match to the patrol.
Data is combined based on the SMART_PatrolID provided in the JSON data, so the only way data from two separate devices should be combined into a single patrol is if these values are the same.
The only other way data can be combined into the same patrol is if the user decides to create a new leg in an existing patrol, instead of creating a new patrol when processing the data. Is there any chance this is happening? Does the patrol you’re referring to have multiple legs?
I’ve asked the user if they could possibly have opted out of the default to create a new patrol but instead added new data to an existing patrol. That requires some deliberate steps away from the default so seems unlikely. We’ve not discussed or ever used this option before. Let’s see what he says.
But the alternative also seems impossible too! Presumably there is no log anywhere that will have recorded whether data were processed to a new patrol or added to an existing patrol?
The combined data were processed on different dates according to the data queue. A single patrol was processed first and then two days later data from another phone had arrived on the queue and were processed, covering a range of dates (4 - some quite old) before and after that single patrol. None of these 4 have processed stop times (all 23:59:59). There are no messages that failed to process. I suppose it’s possible the second phone didn’t complete its uploading before it went out of network coverage again so the stop message never arrived. But I can’t see how the absence of these stop messages is related to this mixing of patrols.
The combined patrol does have multiple legs, but not by our design. Presumably as a result of the processing in SMART assigning different dates of activity to different legs.
And just as a small point, trackpoint data don’t carry the PatrolID number so presumably they are dependent on DeviceID plus date/time to connect them to the correct patrol.
What I would like to be able to do is reset all the relevant items on the queue, delete the patrol from Desktop, and process them again. But I’m not sure I have the stamina to go through each message in turn to reset the status! I don’t think I can rest them in bulk can I?
It does seem that a user accidentally added the later patrol as a leg to the earlier patrol, as there isn’t a way to do that through the automatic processing. It requires the user to select which patrol to add it to at the time it is added.
There isn’t really a log tracking that specific information, but there is a table in the database that links the CTPatrolID and DeviceID to the SMART patrol leg ID. If you export your CA and unzip the file, in the database folder there will be a smart.ct_patrol_link.CtPatrolLink.dat table, but you would need to reference the other tables in order to understand the information.
You are correct that the trackpoints rely on the combination of DeviceID plus date/time to file the data, which is why you may receive warnings when processing data for a device that has two concurrent patrols in SMART (usually the result of a patrol not ending).
Unfortunately, there isn’t a way to change the status of the files in bulk unless you have direct access to the database. Do you know how many files there are for the two patrols? If you could share them, the SMART developers can take a look.
Were you able to find out anything further on this?
Yes. I requeued all the messages (man, that’s laborious!!), deleted the messed up patrols in Desktop, and ran the processing again from my computer. It all processed fine.
We’ve had another instance where two phones ended up on the same patrol. In this second case, we found a new patrol got split so some data went to a new patrol and the rest ended up in a previous patrol that had been created a few days earlier.
We have some other real messes where multiple legs have been created in a single patrol covering multiple dates when in fact it ought to have created separate patrols for each date.
Ive concluded that these have problems have been occurring when the computer that is doing the processing has had poor network connectivity. The processing has gone very slowly or perhaps not completed in some way or been interrupted. And this seems to upset things. I don’t have more careful analysis than that I’m afraid.
Until now I had assumed the problem was with incomplete upload of messages from the phone due to poor connection (or user error, failing to stop patrols). There certainly are still issues in the queue (e.g. duplicated messages) but seems that some of the problems can also come during processing.
It would be a lot easier to resolve problems (e.g. reprocess when connection is better) if there was facility to bulk select messages and bulk edit their status on Connect. One by one editing is just not viable. Also bulk select of messages in the processing queue in Desktop.
Hopefully for future versions!
All my best
I’ve now witnessed myself that Desktop can add data from a different phone to an existing Patrol. I was using a cable connection to the second phone where some data were on board that hadn’t uploaded to Connect. The “Recover data” option in SMART Mobile generated two JSON files on the phone. The Mobile>Import option created 8 new patrols in Desktop and modified one existing patrol. The import process reported some errors (e.g. some points didn’t process) but nonetheless the process completed. The modified patrol was from a different phone in a different place covering different dates. I have kept copies of the 2 JSON files from the second phone, the bug report from the phone, the bug report from the original phone in case that helps, and error log from Dekstop.
Thanks for taking the time to investigate this further. If you could send me the files and bug reports, I can pass them on to the developers to dig into.
FYI, there are plans to add the bulk selection/updating of items in the Data Queue. I’m unsure if will be part of SMART 8 or an interim update before then.