Investigating a few reports of an inability to WebConnect to any device. Initial indications suggest this is related to IPv6.
Subscribe
Investigating - We are getting some reports that on certain ISP IPV6 on the router does not allow webconnect to be used on specific OvrC locations. Disabling IPV6 on the router appears to be a workaround. Our teams are investigating the issue.
Mar 31, 2023 - 13:32 EDT
Mar 31, 2023 - 13:32 EDT
Update - We continue to investigate issues where Intercom Anywhere calls fail to provide video, primarily over cellular connections. Intercom Anywhere is an amazingly complex feature, providing real-time two-way audio and video communication through dozens of nodes, across multiple forms of transmission (Wi-Fi, hardwired ethernet, and cellular 3G/4G/LTE/5G) and hosted by multiple third-parties (mobile carriers, internet service providers, manufacturer of the networking devices on the LAN). The various technologies, platforms, and protocols used during each call have made it incredibly challenging for our engineering teams to locate a single, all-encompassing cause for our dropped video transmissions. Our investigations and analysis of the node-to-node traffic, research into the underlying technologies, communications with mobile carriers, and reviews of countless call logs have not yet identified a single, all-encompassing problem as the root cause for the dropped video streams. However, our teams have identified a number of ways to better our technology and optimize call transmissions.
Through a series of updates over the past month, we have increased our video transmission success rate by over 15%; total incident failures are now in the low single-digit percentages. Considering the myriad of networking and technical variables that can cause an intercom failure (most of which are outside our control), low single-digit failures rates represent a healthy service. While we are encouraged by this positive impact for your customers, we still have work to do. We are still observing higher-than-desired customer impact rates, over half of which occur on AT&T. We are working with AT&T and other carriers to improve performance in these scenarios.
Along with the updates as mentioned earlier, we have implemented new infrastructure technologies that will provide us with more data and resources to investigate and resolve this issue. Over the coming weeks, we will continue to roll out updates and fixes that will improve the performance of our intercom service. We appreciate your patience and thank you again for your trust in us as we continue to resolve this issue for you.
Feb 20, 2023 - 16:25 EST
Through a series of updates over the past month, we have increased our video transmission success rate by over 15%; total incident failures are now in the low single-digit percentages. Considering the myriad of networking and technical variables that can cause an intercom failure (most of which are outside our control), low single-digit failures rates represent a healthy service. While we are encouraged by this positive impact for your customers, we still have work to do. We are still observing higher-than-desired customer impact rates, over half of which occur on AT&T. We are working with AT&T and other carriers to improve performance in these scenarios.
Along with the updates as mentioned earlier, we have implemented new infrastructure technologies that will provide us with more data and resources to investigate and resolve this issue. Over the coming weeks, we will continue to roll out updates and fixes that will improve the performance of our intercom service. We appreciate your patience and thank you again for your trust in us as we continue to resolve this issue for you.
Feb 20, 2023 - 16:25 EST
Identified - Identified – We have been investigating an issue in which Intercom Anywhere calls fail to provide video and deliver an audio-only call while displaying an error message: “Poor Connection. Continuing with audio. Video will return when the connection improves.” After noting that about 6% of all Intercom Anywhere calls revert to audio-only, we prioritized and escalated this issue.
While our engineering teams have not found a root cause yet, our investigations helped us better understand the scenarios where intercom calls are successful and where they are more likely to fail. This allows us to provide some guidance and recommendations while we continue working to resolve this issue.
REVIEWING HEALTHY CALL DATA
Through our investigation, we found that the grand majority of Intercom Anywhere calls transmit audio and video successfully, maintaining a healthy connection throughout the call. Looking more closely at those healthy calls, it’s clear that there is a healthy spread across all hardware models (DS2 POE, DS2 WIFI, DS2 MINI, and WIFI, Chime WIFI and POE) as well as both C4 Mobile OS platforms (iOS and Android). Because success was uniformly present across all devices and platforms, the issue is likely NOT due to any specific hardware failure. More importantly, any resolution will likely NOT involve replacing a device.
A hardware issue would have been made obvious if one model or connection type (WIFI vs PoE) had been absent or barely represented when looking at healthy calls.
REVIEWING FAILED-VIDEO DATA
Similar to investigating healthy call data, our teams were able to recognize important patterns that are now guiding our initiatives to resolve this issue. Perhaps most importantly, of the 6% of calls that fail, the calls that connect over a cellular network have a 3x higher chance of failing when compared to calls completed over WIFI. Because WIFI calls exhibited a statistically higher number of healthy calls, the data implies that either a weak WIFI signal or an issue with the local network resulted in the dropped video. Moreover, the mobile phone model, OS type, mobile app version, and Control4 controller version did not demonstrate any outlying behavior. The data suggests that failures are predominately from how carriers are handling our intercom calls. While cellular intercom calls typically have a 3x higher failure rate, 2 carriers demonstrated a much higher failure rate: AT&T, especially on 5G, and Verizon.
NEXT STEPS AND TROUBLESHOOTING GUIDE
With this data in hand, we have approached AT&T and Verizon about these anomalies and continue to work to identify the ultimate root cause for these failed video connections. We are also working hard to provide a backend fix on our side (addressing how our cloud talks to the carrier’s cloud), which means a resolution would be delivered “over the air” that would involve minimal action, if any, on the side of partner / end-customers; most likely through a software update.
In the meantime, we have compiled an “Intercom + Troubleshooting Best Practices Guide” for our Control4 Partners, which is a step-by-step guide that provides:
Hardware Verification: checklist of installation best practices and recommended settings and configurations for the Chime/DS2 and their drivers.
Local Network Validation: checklist of Intercom Anywhere system requirements, including driver configurations, healthy ISP upload speeds, and (if needed) strong WIFI signal strength.
ISP/Cellular Provider: best practices for optimal performance, even when proper network conditions cannot be met.
The most critical and helpful steps in that guide include:
Upload Speed from ISP: Perform a speed test to verify that the average range in the ISP upload speed does not fall below 10Mbps. If the average range over time consistently drops below 10Mpbs, the Partner should recommend upgrading the Customer’s service plan with their ISP.
Connection Strength and Driver Config: Within Composer Pro, verify the connection strength for the Chime or DS2 (recommended to no less than –70dBm), and if necessary, configure the Chime with the following settings to ensure maximum performance even in poor network conditions:
Frame Size: 1024x768
Max FPS: 15
Bit Rate Control: Constrained
Quality: Excellent
Max Bit Rate: 1 Mbps
Policy: Frame Rate Priority
POSSIBLE WORKAROUNDS
If all best practices and configurations have been employed and the customer still experiences Intercom Anywhere calls with a failed video connection, the issue is likely between the Control4 cloud and the cellular provider. As we prepare to roll out our first back-end measures to address this issue, the customer can try the following workarounds to improve the video connection:
Connect Over WIFI: WIFI calls are 3x more likely to connect video than cellular calls. If possible, be sure to connect the mobile device to WIFI before calling or answering a call. NOTE: Free WIFI hotspots generally limit users to a maximum of 5Mbps upload speed which is not enough for a video connection.
Connect Over 2.4GHz/LTE: The data suggests issues with video connections when using the 5GHz band or the 5G cellular network. Try connecting the mobile device to a 2.4GHz WIFI SSID or turning off 5G to ensure an LTE/4G cellular connection.
Mass market VPNs can introduce network issues and further restrict bandwidth. If the customer must use a VPN, ensure it is highly rated and properly configured.
We recognize the frustration that this issue has caused and apologize to our Partners and customers. Our team is committed to the quality of our products and services, ensuring that they meet the needs of our Partners and customers. As such, we are dedicated to resolving these issues as quickly as possible. We will continue working to improve the customer experience with Intercom Anywhere, even when proper network conditions cannot be met.
Dec 20, 2022 - 11:38 EST
While our engineering teams have not found a root cause yet, our investigations helped us better understand the scenarios where intercom calls are successful and where they are more likely to fail. This allows us to provide some guidance and recommendations while we continue working to resolve this issue.
REVIEWING HEALTHY CALL DATA
Through our investigation, we found that the grand majority of Intercom Anywhere calls transmit audio and video successfully, maintaining a healthy connection throughout the call. Looking more closely at those healthy calls, it’s clear that there is a healthy spread across all hardware models (DS2 POE, DS2 WIFI, DS2 MINI, and WIFI, Chime WIFI and POE) as well as both C4 Mobile OS platforms (iOS and Android). Because success was uniformly present across all devices and platforms, the issue is likely NOT due to any specific hardware failure. More importantly, any resolution will likely NOT involve replacing a device.
A hardware issue would have been made obvious if one model or connection type (WIFI vs PoE) had been absent or barely represented when looking at healthy calls.
REVIEWING FAILED-VIDEO DATA
Similar to investigating healthy call data, our teams were able to recognize important patterns that are now guiding our initiatives to resolve this issue. Perhaps most importantly, of the 6% of calls that fail, the calls that connect over a cellular network have a 3x higher chance of failing when compared to calls completed over WIFI. Because WIFI calls exhibited a statistically higher number of healthy calls, the data implies that either a weak WIFI signal or an issue with the local network resulted in the dropped video. Moreover, the mobile phone model, OS type, mobile app version, and Control4 controller version did not demonstrate any outlying behavior. The data suggests that failures are predominately from how carriers are handling our intercom calls. While cellular intercom calls typically have a 3x higher failure rate, 2 carriers demonstrated a much higher failure rate: AT&T, especially on 5G, and Verizon.
NEXT STEPS AND TROUBLESHOOTING GUIDE
With this data in hand, we have approached AT&T and Verizon about these anomalies and continue to work to identify the ultimate root cause for these failed video connections. We are also working hard to provide a backend fix on our side (addressing how our cloud talks to the carrier’s cloud), which means a resolution would be delivered “over the air” that would involve minimal action, if any, on the side of partner / end-customers; most likely through a software update.
In the meantime, we have compiled an “Intercom + Troubleshooting Best Practices Guide” for our Control4 Partners, which is a step-by-step guide that provides:
Hardware Verification: checklist of installation best practices and recommended settings and configurations for the Chime/DS2 and their drivers.
Local Network Validation: checklist of Intercom Anywhere system requirements, including driver configurations, healthy ISP upload speeds, and (if needed) strong WIFI signal strength.
ISP/Cellular Provider: best practices for optimal performance, even when proper network conditions cannot be met.
The most critical and helpful steps in that guide include:
Upload Speed from ISP: Perform a speed test to verify that the average range in the ISP upload speed does not fall below 10Mbps. If the average range over time consistently drops below 10Mpbs, the Partner should recommend upgrading the Customer’s service plan with their ISP.
Connection Strength and Driver Config: Within Composer Pro, verify the connection strength for the Chime or DS2 (recommended to no less than –70dBm), and if necessary, configure the Chime with the following settings to ensure maximum performance even in poor network conditions:
Frame Size: 1024x768
Max FPS: 15
Bit Rate Control: Constrained
Quality: Excellent
Max Bit Rate: 1 Mbps
Policy: Frame Rate Priority
POSSIBLE WORKAROUNDS
If all best practices and configurations have been employed and the customer still experiences Intercom Anywhere calls with a failed video connection, the issue is likely between the Control4 cloud and the cellular provider. As we prepare to roll out our first back-end measures to address this issue, the customer can try the following workarounds to improve the video connection:
Connect Over WIFI: WIFI calls are 3x more likely to connect video than cellular calls. If possible, be sure to connect the mobile device to WIFI before calling or answering a call. NOTE: Free WIFI hotspots generally limit users to a maximum of 5Mbps upload speed which is not enough for a video connection.
Connect Over 2.4GHz/LTE: The data suggests issues with video connections when using the 5GHz band or the 5G cellular network. Try connecting the mobile device to a 2.4GHz WIFI SSID or turning off 5G to ensure an LTE/4G cellular connection.
Mass market VPNs can introduce network issues and further restrict bandwidth. If the customer must use a VPN, ensure it is highly rated and properly configured.
We recognize the frustration that this issue has caused and apologize to our Partners and customers. Our team is committed to the quality of our products and services, ensuring that they meet the needs of our Partners and customers. As such, we are dedicated to resolving these issues as quickly as possible. We will continue working to improve the customer experience with Intercom Anywhere, even when proper network conditions cannot be met.
Dec 20, 2022 - 11:38 EST
About This Site
Status of Snap One Product Platform Cloud Sites and Services.
End-Customer Services Health
?
Operational
Mobile App: Control4 Intercom Anywhere Android
Operational
Mobile App: Control4 Intercom Anywhere iOS
Operational
Mobile App: Control4 OS 2.x Android
Operational
Mobile App: Control4 OS 3.x Android
?
Operational
Mobile App: Control4 OS 2.x iOS
?
Operational
Mobile App: Control4 OS 3.x iOS
?
Operational
Mobile App: Luma View
?
Operational
Mobile App: OvrC Connect
?
Operational
Service: Control4 Access Anywhere
?
Operational
Service: Control4 Alexa
?
Operational
Service: Control4 APIs
?
Operational
Service: Control4 Google Assistant
?
Operational
Service: Control4 Intercom Anywhere
Operational
Service: Control4 Storage
?
Operational
Service: Control4 Updates
?
Operational
Service: Control4 Weather
?
Operational
Service: Control4 When >> Then
?
Operational
Service: Snap One DDNS [Luma, Wattbox, and other products]
?
Operational
Website: www.control4.com
Operational
Website: customer.control4.com
Operational
Website: triadspeakers.com
?
Operational
Partner Services Health
?
Degraded Performance
Service: Control4 Composer Express
?
Operational
Service: Control4 Composer Pro
Operational
Service: Control4 Drivers
?
Operational
Service: Ihiji Invision
Operational
Service: OvrC Device Communication & Configuration
?
Operational
Service: OvrC Firmware Updates
?
Operational
Service: OvrC Login
?
Operational
Service: OvrC Notifications
?
Operational
Service: OvrC Public APIs
?
Operational
Service: OvrC Web Connect
?
Degraded Performance
Service: Pakedge BakPak
?
Operational
Website: dealer.control4.com
?
Operational
Website: dealer-emea.control4.com
?
Operational
Website: dealer.pakedge.com
?
Operational
Website: dealer.triadspeakers.com
?
Operational
Website: forums.control4.com
?
Operational
Website: my.control4.com
?
Operational
Service: OvrC Speed Test
?
Operational
Website: www.ovrc.com
?
Operational
Third Party Integrations Health (e.g., Spotify, Jandy, Tidal, etc.)
Operational
Amazon Alexa
?
Operational
Amazon Music
?
Operational
Deezer
?
Operational
ecobee
?
Operational
Google Nest
?
Operational
Google Voice
?
Operational
iHeartradio
?
Operational
Jandy
?
Operational
Napster
?
Operational
Pandora
?
Operational
Qobuz
?
Operational
SiriusXM
Operational
Sonos
?
Operational
Soundmachine
?
Operational
Spotify
?
Operational
Tidal
?
Operational
TuneIn
?
Operational
Third Party Dependencies Health (e.g., Amazon, Akamai, The Weather Channel, etc.)
?
Operational
Akamai
?
Operational
Amazon Web Services
?
Operational
Apple APNS
?
Operational
AT&T
?
Operational
LaunchDarkly Client-side streaming API
Operational
LaunchDarkly Edge API
Operational
LaunchDarkly Server-side streaming API
Operational
LaunchDarkly Polling API
Operational
MongoDB
?
Operational
Ookla
?
Operational
Paytrace
?
Operational
The Weather Channel
?
Operational
T-Mobile
Operational
Verizon
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.