After entering some 20+ Independent incidents, all with photos attached, in SMART Desktop 7.5.6, the attempt to sync with Connect 7.5.5. ended with error message (s), see attachment, with error messages. Upload got stuck on different JPGs.
I resolved this situation once by first downloading all changes from Connect to SMART Desktop, then deleting CA on Connect and then uploading the whole CA from SMART Desktop to Connect. After this being done, the synchronization worked OK. The CA had to be downloaded to every SMART Desktop instance for synchronization to work.
However, two days later and after some more Independent incidents entered in SMART Desktop, the issue came back. I attach log file as well. The incidents happened on 18.01.2023 and again on 20.01.2023.
I tried exporting CA from one PC and importing it to another, replacing older version with the most recent one, but the error came together with the latest version of CA
I repeated described steps to remedy the issue, but I am afraid the problem will happen again. Our CA is 2.6 GB, half of it are attached protos of incidents, another half are offline maps. Cold this be a problem?
JK_SMART_LOG.zip (149.5 KB)
Thank you for sharing all of those details. It looks like it’s failing due to some sort of permissions issue with the JPGs in the filestore. I don’t know why this would be happening, but I’ll see if I can get some help troubleshooting this.
The size of the CA shouldn’t be an issue with just syncing the new independent incidents, though I imagine it makes the workaround of restoring the Connect CA from your desktop take a long time.
I’ll let you know what I find out.
Would you be able to share the server logs as well?
logs.zip (489.4 KB)
Mat, here are server logs.
The server logs indicate several issues that may be contributing to this problem.
- There’s some issue related to Tomcat config files
- There appears to be a race condition when restarting the services. When the Postgres service gets started it locks the database until it’s done its internal checks. When the Tomcat service is started too fast and trying to access the database it runs into problems.
I suggest a manual restart of the system by first shutting down Tomcat and Postgres, then restart Postgres first and wait 1 minute to finish up the internal checks and finally restart Tomcat.
The ‘Access Denied’ errors might be related to some Windows security or virus scanner problems. In the Windows Event log might provide a hint into that problem.
Our Admin Alen adjusted starting of Postgres to be with 15s delay after server restart, and Tomcat with 3m delay. We will see.
However, the nine JSONs (attached) which previously could not be processed, are still stuck on server. I do not know if those are related to the previously untuned server components or something else.
JSONs_stuck_on_server.zip (17.0 KB)
What happens when you try to process these files? Can you send the corresponding error information?
Well, stuck JSONS seems to have been processes and only one remained (attached)
08541286aec74c12a81f38138d54d82e.zip (777 Bytes)
The remaining JSON gives very generic error message, unless there is more detailed log somewhere else?
Is there any more information in the error log?
This file just contains an end patrol observation for a patrol that started on 2023/01/24 at 10:13:06. This observation was taken at 2023-01-24 14:43:37, and observation counter indicates that there is only one observation before this in the patrol (which I believe would be the patrol start). Is there anything in your data that would conflict with this?
If you are unable to process this file, you could manually update the patrol end date/time for that patrol.
I am unable to fix the file. Since this is “just” one patrol end, I am going to delete it on server. If you want to take a look , the latest log file is here.
Thanks on you effort, Matt!
log.zip (855.6 KB)
It seems I cannot escape from this error. Seven new JSON packages, all from the same device/user and unprocesable.
Downloads.zip (171.5 KB)
Was there any sort of warning and error when trying to process these new items? Can you send the error log from you SMART desktop as well?
I do not know what to say! The stuck packages got processed somehow later!?
Will wait for the next occasion and hope it will never happen.
It sounds like what’s happening here is that not all of the files are processing successfully the first time you run through them. This is fairly normal, as the files need to be processed in a very specific order so that the information can be saved to your database (e.g. an observation for a patrol can’t be processed before the ‘Patrol Start’ has been processed). You may have to run the processing a couple of times to get through all of the files, so don’t worry too much if they don’t all immediately get processed in the first attempt. If you get to the point where you try processing and it isn’t able to process any of the data queue files, that is an indication that there’s a problem that needs further investigation.
The random appearance of error when uploading some jpg image still occurs, even when there is only one instance of SMART desktop running. This happens on several different PCs running SMART desktop.
The way around is to download changes from server to Desktop, delete CA on server, and upload whole CA to server. this is annoying, since CA has currently 3GB.
the access denied error on windows system is most of the time related to either permission issues. Please check if the service account Tomcat is using has full access rights on the full directory under c:\temp\connect_datastore.
Another reason could also be another process locking the file, such as a Virus scanner.
You can also check the Windows Eventvwr for Access Denied log messages in the Security log.
It seems your advice helped! I have a bit non-standard settings for Windows Environment variables. I keep “Temp/Tmp” folders and SWAP file on partition different than Windows partition, to reduce the size when doing the image backup of my Windows partition.
I created Temp folder on C drive and gave full access to this folder and to my real “Temp” folder, then I erases all what could be erased in my real Temp folder. The upload to server worked after this! It seems that C:\Temp needs to exist for this to work, even if it is not defined as such in Windows environment.
Great that you finally were able to fix the problem. Sometimes the operating system just gets in the way
I have to go back to this issue. I managed to work-around the issue of uploading media and other files to server, when files had some “special” access rights, SYSTEM, Administrator etc… It works if all rights are removed, except “Everyone → full control”. The earlier solution is OK to me. However, since we are building an network of local coordinators, who are supposed to enter data (media files included) themselves, it is too much to expect that they will do the same every time for every file, or that they will be able to create folder with relaxes inheritable permissions and that this will be routine for everyone.
Would it be possible that SMART Desktop removes all restrictive rights from media files, while those are being attached to observations? I guess this would resolve the issue of restricted uploading to Connect.