Here's where you can come to learn about recent changes to Engage technology, including new capabilities and features
Engage is extremely proud of maintaining 99% or better up-time. Interested in seeing the up-time of Engage infrastructure? You can view our public status page at any time:
Engage API OUtAge 9/5/18
On Weds September 5 2018 12:41pm PST the Engage API was experiencing a massive outage. This is impacted all parts of Engage including login, dashboard, and widgets. This issue was corrected with an API restart at September 5 2018 at 1:05pm PT.
Engage DASHBOARD LOGIN ISSUE 8/7/18
On Tues August 7 2018 4:41pm PST the subdomain dashboard.engage.co (the login site) was not accessible. As of 7:09am CST on 8/8/2018 dashboard.engage.co was still inaccessible. Two senior engineers and key architects of Engage are looking into the issue. Under normal circumstances the dashboard page where you can login should load. However, for certain customers the page does not load and is resulting in a page load request timeout. A reset of the web server occurred, but customers plagued with the page load timeout issue were still experiencing the issue. Engage uses a CDN (content delivery network) the sits between our service and customers. The CDN helps provider faster web page load times and more accessibility to Engage services from different markets. It is presumed that there is a CDN configuration or service issue going on and we're researching this directly with our CND vendor. UPDATE: 8/8/18 2:02pm PST. A patch was released for the issue. Dashboard should be accessible and login should be immediately available. Please note that if you're having trouble accessing dashboard.engage.co that it could be a delay in DNS propagation. If so, please attempt accessing the site later and/or performing a shift>refresh of your browser window pointing to dashboard.engage.co. Thank you and again, our most sincere apologies for those impacted by this most unfortunate issue.
Engage DASHBOARD LOGIN ISSUE 7/6/18
On Fri July 6 2018 4:19am PST the login site was not allowing logins to the Engage dashboard. This was reset at 7:04am PST and login is working as expected.
Engage API ISSUE 6/14/18
On Thurs June 14 2018 10:19am PST a critical issue was discovered in the Engage API. The API has been unstable and certain affected customers have experienced problems logging in. They symptom is an "error" message upon logging in. We are researching the problem. For those of you having trouble logging in.
We discovered at 12:41am PST a widening and critical issue impacting subdomain pages as well, including profile and directory pages. Updates will be pushed as soon as we have resolution.
UPDATE: at 5:21pm PST a reset of all Engage infrastructure occurred. We recommend refreshing your browser to avoid any potential cached issues. The issues were caused by memory issues and we're looking into methods to combat/avoid this issue again.
Engage chatlog access error 4/17/18
On Tues April 17 2018 6:13am PST a critical issue was discovered in the Engage dashboard that prevents some users from viewing previous chats after they have logged out and logged back in. This issue has been discovered to have effected some users going back as far as three weeks (to Mar 28 2018). The symptom is that users have chats, log out of the dashboard, then re-login at a later date and recent chats do not display in the dashboard and are not discoverable in chat history search. This is an extremely complex issue that also appears to be impacting chat forwarding (wherein you forward your chat logs to your email address of record). This issue is currently in "review" status, but has been noted as an unusually difficult issue as it is affecting some users, but not others. We are also interacting with a major vendor to better understand recourse measures. We will detail more information as it comes available. In the interim we ask that, if you believe you are being impacted by this problem, to consider manually copying your log for redundancy purposes. Our most sincere apology for any incovenience this is causing any of you impacted. We're very transparent about our flaws and this is one rare issue that we're finding extremely vexing. UPDATE: the problem was worked on and we believe a fix is working across all user accounts. The week of 5/21/18 we also re-checked this and it appears that the fix is holding for all chat logging and conversation history access through the dashboard.
Engage api ACCESS 4/3/18
On Tues April 3 2018 7:14am PST the Engage API was experiencing connectivity problems. The cause of this issue is related to a full cache. The primary symptom is that those who are not currently logged in experience an "error" message when attempting to login to the Engage dashboard. Our engineering group was been notified of the issue. A restart of the cache was implemented and we believe logging in is working. If you continue to receive an error, please attempt a browser refresh. We apologize for any inconvenience. Thank you.
Engage api & DISK SPACE 1/2/18
On Tues January 2 2018 6:14am PST the Engage API was experiencing connectivity problems. In addition, a storm of new year user logins resulted in a disk space limit being reached on our XMPP server. We cleared logs and restarted all infrastructure and thus as of 9:54am PST it appears all systems are working as expected. We have learned from some users that a browser or computer restart have helped with logging in, which could be related to browser caching. We want to apologize for any inconvenience. Wishing you a happy and successful 2018!
Engage api down 12/22/17
On Thurs December 22 2017 4:14am PST that Engage API was experiencing connectivity problems. One of the problems caused by this issue is an inability to login from the Engage dashboard; upon login attempt either a red "error" or "authentication failed" message shows on the dashboard login screen and does not allow you to login. Because of this problem users who are not able to login also can not connect to our presence (status) system and therefore will not show as "online". Our CTO and a Senior Engineer were alerted to the problem, error logs were cleared, and the API was reset at 6:57am PST. Engage should be fully functional for you as of the reset.
Potential Login issues 12/14/17
On Thurs December 14 2017 6:02am PST we were alerted that there have been regional challenges for some users logging in or staying connected to the Engage dashboard. The symptom is that upon login either a red "error" or "authentication failed" message shows on the dashboard login screen and does not allow you to login, or the login takes a long time and when you do see the dashboard load the content is slow to display. Our CTO and a Senior Engineer reviewed the issue and successfully corrected the issue at 7:17am PST.
Login issues 11/7/17
There have been regional challenges for some users logging in or staying connected to the Engage dashboard. The symptom is that upon login either a red "error" or "authentication failed" message shows on the dashboard login screen and does not allow you to login, or the login takes a long time and when you do see the dashboard load the content is NULL. These issues appear to be connected to a major internet outages. The connection problems, which affected several major ISPs, including Comcast, Verizon, and AT&T, come just after the one-year anniversary of a DDoS attack on internet-infrastructure company, Dyn, that crippled the net for a day in 2016. However, this week's outages are thought to have had a slightly more unremarkable cause: a misconfiguration at Level 3, an internet backbone company and enterprise ISP, that underpins other big networks.
Infrastructure Maintenance Notice for 11/3/17
One or more of instances of Engage infrastructure in the Eastern U.S. Region are scheduled to be rebooted between Fri, 3 Nov 2017 07:15:00 GMT and Fri, 3 Nov 2017 10:00:00 GMT for required system maintenance in order to deploy important updates. Security and operational excellence are our top priorities, and therefore we occasionally need to do host maintenance. Each instance will experience a clean reboot and will be unavailable while the updates are applied. This generally takes no more than a few minutes to complete. Each instance will return to normal operation after the reboot, and all instance configuration and data will be retained.
Engage performs maintenance regularly to ensure that the service continues uninterrupted for our customers. In most cases, maintenance can be performed without service interruption. When maintenance cannot be performed without service interruption, we work hard to keep any impact as brief as possible. Thank you in advance for your patience and understanding.
POTENTIAL IMPACT notice from Engage 7/27/17 & 7/28/17
The Engage RESTful API service was experiencing downtime as of 7:53am PST, representing our first significant service interruption since November of 2016. During this period you may be unable to login to the Engage dashboard, and/or logging in may be slow. Another symptom is that the chat history that automatically populate may not display at all, or may be extremely slow to load. Underlying hardware hosting our application in the us-east-1 region is experiencing degradation, and our server has been unreachable. Our engineering team is researching the issue and we will post updates here as they become available. Thank you and apologies for any inconvenience.
UPDATE 8:22am PST 7/27/17. Engage relies on Amazon AWS/EC2 services in the delivery of Engage. Amazon has a major server/hardware instance failure. We are looking to create and move to a new instance.
UPDATE 10:21am PST 7/27/17. It appears that this server/hardware instance failure is catastrophic. We are copying the instance, and are rebuilding the instance on completely new hardware. This process may take up to 4 hours. In the interim period you may have delays in logging in, you may not be able to login at all, certain data and analytics may not be available, directory services may not update promptly, and some or all site widgets (chat call-to-action elements) may not load on your primary or third-party websites. Again, we apologize for this unforeseen issue experienced by our vendor Amazon AWS.
UPDATE 12:32am PST 7/27/17. Multiple engineers at Engage are working at breakneck speed to create new instances of the failed servers. There were several challenging dependencies and inter-dependencies in moving to a new instance and this work is in-progress. We hope to relaunch the parts of Engage impacted by the failed servers in the next 1-2 hours. It's all-hands-on-deck and thank you for your patience as we work through this extraordinary issue.
UPDATE 3:47pm PST 7/27/17. Well, that's been fun. We did get the work done. Chat widgets are loading, company directories are working, the chat profile pages are loading, chat is working. However, we're still noticing some minor impact issues, especially in the chat dashboard.
UPDATE: 7:05pm PST 7/27/17. Everything is up and running, but we're still seeing some "connection issues" in the chat dashboard relating to "recent conversations," "chat history," and "stats." The recent conversations and any prior chat searchers are showing as empty (null) in some instances, and the stats reporting will sit and spin if you try to search a date range or keyword search. We'll continue working through this issues. Thank you again for your patience as we work through this major hardware failure at Amazon.
UPDATE: 6:03am PST 7/28/17. While it appears the vast majority of users are able to successfully log in to the chat dashboard, a few users continue to be impacted. The following issues continue to be reported: 1) unable to login - it just 'sits and spins,' 2) the dashboard does not populate with my recent conversations, 3) the dashboard loads like normal but if I click on a recent conversation it sits and spins, 4) If I try to look up chat history or stats it does not populate with data and/or sits and spins. We are still working to establish better service connectivity throughout the network. We have rebooted all Engage services and taken a number of steps. Engage also lives behind a global CDN (content deliver network) and we've confirmed that what's working is loading fast. For those of you who were impacted yesterday we apologize. For those who continue to experience access issues, please know that we'll continue working to make sure we deliver the incredible service and uptime that we're known for. Thank you!
UPDATE: 9:55am PST 7/28/18. We have completely reconfigured our servers on the east coast node where Amazon AWS experienced the major hardware failure. To the best of our knowledge, all chat, dashboard, widgets, chat history and stats are functional. The only issue we are still seeing is the connection to browser preview. Our heroic engineering team who have been working for 24 hours to rebuild instances are working on the browser preview rebuild, then they plan to get some well-deserved sleep. If you continue to have issues, please first try shift>refresh in your browser. If things are still not working for you, please email us at support. Thank you!
POTENTIAL IMPACT notice from Engage 11/1/16
Level 3 is one of the largest networks in the U.S. and they are having major outage issues today. As a backbone to U.S. Internet infrastructure many ISPs rely on data transmission on Level 3. The current issues they are experiencing could impact your use of Engage. The status of your users may not be transmitted, which could result in "error" issues in logging in as well as status not updating in widgets. Engage infrastructure is up, but parts of the Internet are being impacted which would cause unusual behavior that could be experienced by some Engage clients. Here is a map of the Level 3 outage area. Engage was not impacted by the last major outage on 10/21/16 and we are not seeing any impacts today, but if your ISP uses Level 3 as a data carrier, that could explain the issues. Also, we do not know for certain if this is another DoS attack similar to what occurred on 10/21/16, or if this is merely isolated to just Level 3.
A special notice from Engage 10/21/16
What is believeD to be a series of unprecedented DoS attacks has caused major impacts across the Internet today. Major websites have been impacted at different points, and reports from the East Coast U.S. said at times half the Internet went dark. It has been reported that the U.S. Government is looking at this as a criminal act.
We wanted to update you about Engage. Our website engage.co has been impacted at several points and some visitors, especially those on the east coast, have not been able to access our site, or parts of our site.
However, none of our underlying services have been impacted. The chat dashboard has been up continuously and all widget and chat API services remain at 100% uptime. You can always view publicly the status of Engage infrastructure by visiting: http://stats.pingdom.com/7axuogji186z
Engage employs an extremely high level of security around our platform, and we support the uptime of our platform with distributed infrastructure across a global content delivery network. While we fully expect our service to remain accessible to customers, we are continuing to monitor this situation closely.
Engage Error 7.3.16
- On Sunday, July 3rd we were alerted by a client to an issue where they manually logged out of Engage, shut their computer down, and received incoming chats over the evening. The symptoms were that when logging in the next day being surprised to see that incoming chats had occurred over the evening, that the timed-out response did not work, and that the customer contact info and customer information panel were not populating with customer data. We had not seen this issue previously and all initial tests on our end were unable to reproduce the issue (though we could observe the behavior in that user's account). The issue seemed to expand to several other users on 7.5.16 (for unknown reasons). As a result, we went all-hands-on-deck with all available engineering resources being applied towards addressing the issue. What we suspect right now is that this issue is being caused by our dashboard infrastructure correctly seeing the user has changed their status to offline, that information is being reported to our status server, but the status server is not updating the status to "release" the users status to "offline". A number of new test functions have been written to observe the behavior and a solution to the problem is being worked through. This problem had not been previously observed and so its occurrence and persistence have been quite perplexing, but we're optimistic we'll address the issue promptly. If you believe you have been affected with this issue (most users have not been), please email us at firstname.lastname@example.org so we can monitor your account. Thank you. On 7.7.16 this problem continued to be "in-process" with a fix. The engineers developed an alternative method to detect and confirm status between the dashboard and the status servers. A solution is currently in QA testing. An unscheduled maintenance window will occur after hours to reboot one or more servers and a software patch will be applied. To those affected, thank you for your patience. Many of you have taken time to help provide us more information and a fix wouldn't be possible without your cooperation, so thank you! On 7.8.16 a server reboot occurred, and a software patch was applied. We applied a new status check methodology which allows Engage to check both the dashboard AND status servers for availability. For those who are NOT double-confirmed as available for chat, you will be default timed-out in 90-120 seconds. We believe with this fix the issue has been resolved. If you continue to remain in an "online" status and don't believe you should be, please login to Engage, then logout. This will reset your status and going forward we will double-check your true availability.
Engage Error 5.30.16
- On Thursday May 30th we were alerted by a client about broken chat sessions occurring on an Android device. Engineering found the issue with Android. Chrome on android was recently updated, and the code that creates the desktop notifications was recognizing the Notification API on mobile, but when it tried to create the notification it failed and blocked incoming messages from being processed. As a result, it was possible for an agent in dashboard to not see messages coming in from a customer, or the ability for the customer to see incoming messages. The fix was tested and deployed on 6.1.16. In addition, engineering has added additional checks designed to keep this from happening in the future.
Engage Error 5.23.16
- At approximately 10:45am PDT on May 23rd, 2016 an updated version of Engage was pushed out that included new code that deployed to a single chat server and not the entire infrastructure, as intended. As a result, it was possible that for some users incoming chats were immediately disconnected and that actual and visible agent status could not be uniform. A key symptom of this issue was that a chat agent would be logged out of the dashboard, requiring them to login again. A fix was released at approximately 11:57am PDT. Steps have been taken to prevent this in the future and we apologize for any inconvenience this may have caused. Rest assured this was not infrastructure downtime, but rather a software release issue.
Engage Dashboard Changes pushed 4.26.16
- Administrators can now control (limit) the number of agents who appear in the widget. In some instances (especially where you have more than 30 agents "online" simultaneously) the number of available agents could be overwhelming to visitors. With this new setting you can limit the visible agents to a number (limit) you determine, such as 15 agents.
Engage Dashboard Changes pushed 4.4.16
- Administrators can now disable the widget on your website with a single click. This feature was designed so that when installing Engage and getting team members (users) onboarded the widget won't load. Once you're ready for showtime, you can click it and the widget will be live. Now you can install the widget code right away, but activate the widget when it makes sense for you.
Engage Dashboard Changes pushed 3.25.16
- Administrators can now change your team members status between "active" and "inactive" and also delete users. The inactive status means that the user is effectively hidden from displaying in widgets. This is a useful tool if an employee is gone on vacation or leave, or if they use Engage but are not part of the customer-facing set of team members you wish to display (such as IT people, marketing people, or admins). To change a user status: 1) login to the admin dashboard, 2) click on "edit account" then click on "team" and then choose the user and click on "status" where you'll be able to toggle between "active" and "inactive".
Engage Dashboard Changes pushed 3.21.16
- Administrators can now change your company directory domain name. For example: http://directory.engage.co/ppr When the domain has been changed, it is updated at the sub-directory structure in the URL. In addition, this change is automatically synced with the Engage API and updates all widgets and pointers without any required modifications.
- New reset password function.
Engage Dashboard Changes pushed 3.8.16
- From the Engage dashboard you can now edit your profile
- The Engage dashboard design and layout received modest changes
- The message dialogue window has messages of alternating colors
- Account admins can now edit user profiles
- Account admins can now change user passwords
- Account admins can now edit their company profile
- Account admins can now build unlimited widgets
- Users and admins can now add photos and logos
Features, Capabilities & Improvements in Development
- Syndication code Browser Preview Tracking which allows the dashboard to display any third-party site where you are syndicated to as a referrer source for incoming Engagements
- Color palette for widgets
- Make "end chat" and "email transcript" more obvious in the dashboard
- Ability for admins to add categories
- Ability for admins to delete/suspend
- Advanced statistics / analytics
- WebRTC updates
- Real-time B2B/B2C intelligence (Extensions Panel)
- Metadata and Schema
- Chat bots and chat automation
- Pre-canned messages (aka operator scripting)
- In-dialogue preview of shared URLs
- Design updates to HTML5 Profile page, including new "End Chat" button
- Adding "online" menu option to HTML5 Company Directory pages
- Flag to proactively show the same agent previously interacted with based on a tracking cookie
- Android / Google Play app, with push notifications
- Apple / iPhone app, with push notifications
- In-dialogue file/asset sharing
- More CRM integrations planned, including SugarCRM, Highrise, Insightly, Solve, PipelineDeals, SuiteCRM, vTiger, Zoho, baseCRM, Netsuite, Nimble, SAP and Infusionsoft
- Simplifying access to existing CRM integrations including Salesforce and Microsoft Dynamics
- Adding an "SMS" option for sending asynchronous client communications from the dashboard directly to the customers mobile phone