Monday April 5, 2021 Monday – Second Morning Update
We are pleased to inform you that the employer portal has been brought back online. You may login as you did previously or directly from this link https://payalitycloud.payrollservers.us/pg/login.aspx. We would kindly ask that those clients without an immediate need to access the system wait until later to do so.
Here is what we know.
Late on April 1st at around 10:50 PM, SwipeClock detected suspicious activity within their cloud environment which impacted the availability to access the timekeeping software. This service interruption also prevented employees from punching in via WebClock and the SmartPhone app.
Out of an abundance of caution, SwipeClock kept the system offline as their internal team and independent forensic experts conducted a thorough review of the data. While that investigation is still ongoing, they have made significant progress and there continues to be no evidence of theft of data from their servers.
I want to reiterate, that the time keeping system and database is stored in a completely separate environment from Payality’s payroll system. There has be no security issue whatsoever with Payality’s payroll system or database and our payroll system has been fully operational.
On April 3rd, SwipeClock restored the use of the smartphone app and web clock, that in most instances, allowed employees to resume clocking in/out. This limited service also enabled hardware clocks to transmit punch data, collected during the outage, to the employees timecards. Processing this backlog of punches from April 1 to April 3rd to the servers and timecards is ongoing. If you don’t see all punch data for those 3 days when you first log into the system, it will be there soon.
Timecard entries for employees that were unable to punch in via WebClock or the smartphone app from the time of service disruption that started at approximately 10:50 PM on April 1st and lasted until approximately 5:00 PM April 3rd, will need to be manually entered.
As a reminder, we have programmed Payality’s custom web clock login url https://payalitycloud.payrollservers.us/webclock/ to redirect to the generic web clock login url http://clock.payrollservers.us / that SwipeClock recommended in a previous update. If your employees see a blank screen when clicking on their current web clock link, it will be necessary to clear cache and browsing history on the computer/device for the web clock login screen to appear. They may also go directly to http://clock.payrollservers.us/ to clock in.
Monday April 5, 2021 Morning Update
We received an update from our time and attendance partner, SwipeClock at 4:30 AM today. We are pleased to announce that all employee data and punches up to the time the system was taken offline late in the evening on April 1 have been restored. Here is how this announcement affects you and your employees.
Employee use of Smartphone App and WebClock – Both are now functional for all employees (if enabled by the client).
Physical clocks – all punches stored in the clocks from the time the system went offline to present have synched to the employee’s timecards.
Time and Attendance Employer Portal – While functionality has been restored and brought back online, SwipeClock is requesting that clients allow more time today for final testing and wait to login until they provide an “all clear” in an update sometime later today.
Payality is extending the deadline to submit payroll for those clients with an April 6 check date until 3:30 PM today. We would kindly request that those clients with a check date of April 7 or later wait to login to the time and attendance employer portal as long as possible. We will extend the payroll submission deadline on Tuesday and Wednesday of this week until 3:00 PM as well.
We will email you as soon as we receive the “all clear” notice from SwipeClock
SwipeClock Update Monday April 5, 4:30 AM
We have now restored all data up until the time the system was taken down on April 1.
As we test the system for bringing it back online, it is possible for users to log in to the employer time management console. We ask that partners and clients alike to refrain from doing so until all testing is complete and we provide all clear here and in an official email update. It is likely the system may see service interruptions as we continue to test and we don’t want users to experience any issues.
Sunday April 4, 2021 Afternoon Update
Dear Payality Client:
We just received an update from our time and attendance partner, SwipeClock. They have restored the syncs between the physical clocks and those punches are being uploaded to the employee’s timecard. As stated in their update below, their “goal” is to have the system up for full processing tomorrow (that includes access to the employer portal to enable viewing/editing of timecards). I will continue to monitor for any updates this evening and email you when the guide they reference is released or other specific timelines/updates are provided.
Having said that, Payality is working on other contingencies should there be any further delay. These include:
- Allow for a later payroll submission time (3:30 PM) for next day check dates
- Provide reporting showing the employee’s average hours worked per pay period over the last 6 months in the event that it becomes necessary to estimate hours for the most recent pay period.
I realize that these are not optimal options and I sincerely hope that they will be unnecessary. I’m sorry for the uncertainty and issues that this matter has caused.
SwipeClock update posted on April 4, 2021 at 4:16 PM
On the evening of April 1, 2021, we detected suspicious activity within our cloud environment. We immediately took actions to safeguard the systems which impacted the availability of clients to access our timekeeping software.
Our development team restored limited service on April 3 which returned timeclock functionality and punch collection to the TimeWorksPlus system.
With this service restored, employees were once again able to use the WebClock and mobile app to clock in and out along with physical clocks. It also meant that punch data collected on physical clocks during the outage could be transmitted from the internal clock memory to our servers.
Next Steps:Our goal is to have the system up for full processing tomorrow. We are finalizing a guide outlining steps to prepare for the system coming back online. Please keep in mind you may temporarily experience performance inconsistencies as we continue to bring everything online. Watch our status page for details.
Timeline:We will provide frequent updates throughout the evening on our Status Page as we work to bring the system back online
Sunday April 4, 2021 Morning Update
Dear Payality Client:
We just received the latest update from SwipeClock this morning and the status appears unchanged from the email update that I sent last night at 7:30 PM. They did however set Monday morning as the ETA for full system access. Another update is due this afternoon after 4 PM.
As I mentioned last evening, employees entered in the time and attendance system before March 16 are able to resume use of the Smartphone app or web clock. Employees will use web clock as they have in the past. As it states in the SwipeClock update below, they continue to work diligently to restore employee data and clock in/out punches from after March 16.
We have programmed Payality’s custom web clock login url https://payalitycloud.payrollservers.us/webclock/ to redirect to the generic web clock login url http://clock.payrollservers.us/ that SwipeClock mentioned in last evenings update. If your employees see a blank screen when clicking on the current web clock link, it will be necessary to clear cache and browsing history on the computer/device for the web clock login screen to appear. They may also go directly to http://clock.payrollservers.us/ to clock in.
Please have employees continue to use the physical clocks as the punches are being stored in the clock memory and will automatically sync to their timecards once full system access is restored.
The client admin site to view/edit punches is still unavailable but is expected to be brought back online Monday morning.
I will email you again this afternoon after I have a chance to review SwipeClock’s scheduled 4 PM update.
SwipeClock Status Update received on April 4, 2021 at 7:15 AM
Mobile and Webclock punch collection continues to function and is performing well.
We have also restored the functionality for Webclocks embedded in third-party portals. If you use this functionality, we recommend you verify and validate.
NOTE that any TimeWorksPlus site or employee that was added to the system after March 16, 2021 is not able to punch at this time. We continue working to restore data entered into the system after March 16 and will keep you updated on these efforts.
All efforts remain focused on restoring full system access. We are working towards a goal of having the system available Monday morning. We will provide an update on this goal later today.
We will issue another email update on 4/4/2021 at 7:00 pm Eastern.
Saturday April 3, 2021 Afternoon Update
Dear Payality Clients.
We received an update from our time and attendance partner, SwipeClock at 5 PM. (Please see below). I’ve been trying to absorb this information and speak directly with SwipeClock for some clarification to be able to provide you with an accurate and honest update.
As stated in the update below, SwipeClock was able to bring the ability for employees to use the smartphone app and webclock to clock in/out, back online However, to do so, they restored data from a March 16 back-up. Because of this, any employees who were added after March 16 will not be able to clock in/out for now. Also, if the employee changed their password after March 16, they will need to reset their password using the Forgot Password? Link as shown in the screen shot below.
For those clients with physical time clocks, those punches will continue to be accepted and stored on the clocks. SwipeClock is still working on restoring the sync with the clocks so the time will be transferred to the client portal for review and editing.
The client portal that allows clients/supervisors to view and edit timecards is still not active. Also, any punches after March 16 are also not currently available. SwipeClock says that they are working hard to get those punches as well as the most current employee data restored but haven’t provided a firm date/time. In my conversations with SwipeClock this evening, they feel confident that it will occur by Monday morning. They did state that they will have an update tomorrow at 7 AM Pacific Time and I will provide an update as well as guidance if client portal access and/or all punch history from March 16 forward (as well as the most recent employee data) is not restored before Monday morning.
SwipeClock Update Received April 3, 2021 at 5:00 PM
Incident Overview:On the evening of April 1, 2021, we detected suspicious activity within our cloud environment. Upon discovery, we immediately disabled access to certain systems, which impacted the availability for clients to access our time and attendance solutions.
Following our recovery plan, we prioritized safely and securely restoring our systems. The next priority was the restoration of punch collection via web and mobile devices. Physical clocks have been collecting punches in offline mode.
Current Status:We are now able to report that smartphone app (TimeWorksPlus Employee) and Webclock punch collection has been restored and is now active. Please direct all employees to the following link to login and punch using the Swipeclock Webclock: http://clock.payrollservers.us/
Data restoration is being executed in a staged approach. In order to restore systems as quickly as possible, we used backups with data through March 16, 2021. We are working to restore data after March 16 and will keep you updated on these efforts.
NOTE that any employees that were added to the system after March 16, 2021 will not be eligible for punching at this time.
With punch collection now restored, all hands are focused on restoring full system access and remaining historical data.
Timeline:We will issue another email update at 4/4/2021 at 10:00 am Eastern.
Saturday April 3, 2021 Saturday Morning Update
Dear Payality Clients:
I just received this update from our Time and Attendance partner, SwipeClock. They are continuing to work to restore service and have promised an additional update at 4 PM today. If I hear anything else before then, I will email you immediately. I want to reiterate that the time and attendance system is a separate database and hosted on a different environment/servers than our payroll system. No payroll data has been compromised and our payroll system is fully functional. Please accept my apologies for this failure to provide the service that you expect and deserve.
Swipeclock continues to work through our recovery plan steps to securely and safely restore service. To date, we have deployed next generation endpoint threat detection and response monitoring software on devices connected to our network, changed all system passwords, and continue to implement additional security controls as we work to restore service.
We are currently validating the data and going through a complete testing process. Currently, we do not have any evidence that data in our system has been compromised. We have hired independent forensic experts to investigate what occurred and what we can do to prevent this from occurring in the future. We have also asked them to confirm that no data was compromised, and we will keep you updated about any key findings.
Upon thorough validation and testing, we will begin the process of running through the backlog of time punches collected on hardware devices and prepare to accept live punches through Webclock and the mobile apps.
We continue to work towards full system access and will continue to provide updates as we have them.
We will issue another update later this afternoon/evening on 4/3/2021 at 7:00 pm Eastern.
Friday April 2, 2021 Afternoon Update
Time and Attendance Update
We were just notified by our Time and Attendance partner, SwipeClock, that the effort to bring the time and attendance system back online is still a work in progress and a firm timeline is not available.
We also learned that the issue causing the system to go down was not caused by the Microsoft Azure outage, but rather suspicious activity detected within their cloud environment late last night. As a preventative measure, SwipeClock shut down all access until a full assessment of the situation could be done and they are now taking steps to ensure there are no potential threats going forward and to bring their system back online as quickly as possible.
We want to assure you that SwipeClock’s time and attendance database and Payality’s payroll databases are completely separate and do not reside in the same environment/servers. There have been no security issues whatsoever with any of the payroll data and our payroll system is fully functional.
Swipeclock is working first to restore the ability for employees to punch in/out. We recommend having affected employees note the hours manually and we will assist in updating the punch activity in the system when it is restored.
Friday April 2, 2021 Morning Update
Microsoft Azure reported a widespread outage last night that affected millions of cloud-based solutions. Our payroll system is fully accessible, however, our time and attendance solution, including the website and smartphone apps are still down. Our time and attendance partner is working diligently to get the service back online and we will provide regular updates with their progress.”
At this time, the site and smartphone apps where employers view/edit timecards and approve time-off is currently down.
In addition, this impacts employee’s clocking in and out in the following ways:
For Web Clock/Smartphone users
Employees will be unable to clock in/out while the system is down.
For physical clock users
Employees can continue to punch in/out using a physical clock. This information will be stored on the clock. Once the outage is resolved, the collected punches will be synced to the server.
We will update you as soon as the time and attendance system is back online.