Identified - Identified – We have been investigating an issue in which Intercom Anywhere calls fail to provide video and deliver an audio-only call while displaying an error message: “Poor Connection. Continuing with audio. Video will return when the connection improves.” After noting that about 6% of all Intercom Anywhere calls revert to audio-only, we prioritized and escalated this issue.
While our engineering teams have not found a root cause yet, our investigations helped us better understand the scenarios where intercom calls are successful and where they are more likely to fail. This allows us to provide some guidance and recommendations while we continue working to resolve this issue.
REVIEWING HEALTHY CALL DATA
Through our investigation, we found that the grand majority of Intercom Anywhere calls transmit audio and video successfully, maintaining a healthy connection throughout the call. Looking more closely at those healthy calls, it’s clear that there is a healthy spread across all hardware models (DS2 POE, DS2 WIFI, DS2 MINI, and WIFI, Chime WIFI and POE) as well as both C4 Mobile OS platforms (iOS and Android). Because success was uniformly present across all devices and platforms, the issue is likely NOT due to any specific hardware failure. More importantly, any resolution will likely NOT involve replacing a device.
A hardware issue would have been made obvious if one model or connection type (WIFI vs PoE) had been absent or barely represented when looking at healthy calls.
REVIEWING FAILED-VIDEO DATA
Similar to investigating healthy call data, our teams were able to recognize important patterns that are now guiding our initiatives to resolve this issue. Perhaps most importantly, of the 6% of calls that fail, the calls that connect over a cellular network have a 3x higher chance of failing when compared to calls completed over WIFI. Because WIFI calls exhibited a statistically higher number of healthy calls, the data implies that either a weak WIFI signal or an issue with the local network resulted in the dropped video. Moreover, the mobile phone model, OS type, mobile app version, and Control4 controller version did not demonstrate any outlying behavior. The data suggests that failures are predominately from how carriers are handling our intercom calls. While cellular intercom calls typically have a 3x higher failure rate, 2 carriers demonstrated a much higher failure rate: AT&T, especially on 5G, and Verizon.
NEXT STEPS AND TROUBLESHOOTING GUIDE
With this data in hand, we have approached AT&T and Verizon about these anomalies and continue to work to identify the ultimate root cause for these failed video connections. We are also working hard to provide a backend fix on our side (addressing how our cloud talks to the carrier’s cloud), which means a resolution would be delivered “over the air” that would involve minimal action, if any, on the side of partner / end-customers; most likely through a software update.
In the meantime, we have compiled an “Intercom + Troubleshooting Best Practices Guide” for our Control4 Partners, which is a step-by-step guide that provides:
Hardware Verification: checklist of installation best practices and recommended settings and configurations for the Chime/DS2 and their drivers.
Local Network Validation: checklist of Intercom Anywhere system requirements, including driver configurations, healthy ISP upload speeds, and (if needed) strong WIFI signal strength.
ISP/Cellular Provider: best practices for optimal performance, even when proper network conditions cannot be met.
The most critical and helpful steps in that guide include:
Upload Speed from ISP: Perform a speed test to verify that the average range in the ISP upload speed does not fall below 10Mbps. If the average range over time consistently drops below 10Mpbs, the Partner should recommend upgrading the Customer’s service plan with their ISP.
Connection Strength and Driver Config: Within Composer Pro, verify the connection strength for the Chime or DS2 (recommended to no less than –70dBm), and if necessary, configure the Chime with the following settings to ensure maximum performance even in poor network conditions:
Frame Size: 1024x768
Max FPS: 15
Bit Rate Control: Constrained
Quality: Excellent
Max Bit Rate: 1 Mbps
Policy: Frame Rate Priority
POSSIBLE WORKAROUNDS
If all best practices and configurations have been employed and the customer still experiences Intercom Anywhere calls with a failed video connection, the issue is likely between the Control4 cloud and the cellular provider. As we prepare to roll out our first back-end measures to address this issue, the customer can try the following workarounds to improve the video connection:
Connect Over WIFI: WIFI calls are 3x more likely to connect video than cellular calls. If possible, be sure to connect the mobile device to WIFI before calling or answering a call. NOTE: Free WIFI hotspots generally limit users to a maximum of 5Mbps upload speed which is not enough for a video connection.
Connect Over 2.4GHz/LTE: The data suggests issues with video connections when using the 5GHz band or the 5G cellular network. Try connecting the mobile device to a 2.4GHz WIFI SSID or turning off 5G to ensure an LTE/4G cellular connection.
Mass market VPNs can introduce network issues and further restrict bandwidth. If the customer must use a VPN, ensure it is highly rated and properly configured.
We recognize the frustration that this issue has caused and apologize to our Partners and customers. Our team is committed to the quality of our products and services, ensuring that they meet the needs of our Partners and customers. As such, we are dedicated to resolving these issues as quickly as possible. We will continue working to improve the customer experience with Intercom Anywhere, even when proper network conditions cannot be met.
Dec 20, 2022 - 11:38 EST
While our engineering teams have not found a root cause yet, our investigations helped us better understand the scenarios where intercom calls are successful and where they are more likely to fail. This allows us to provide some guidance and recommendations while we continue working to resolve this issue.
REVIEWING HEALTHY CALL DATA
Through our investigation, we found that the grand majority of Intercom Anywhere calls transmit audio and video successfully, maintaining a healthy connection throughout the call. Looking more closely at those healthy calls, it’s clear that there is a healthy spread across all hardware models (DS2 POE, DS2 WIFI, DS2 MINI, and WIFI, Chime WIFI and POE) as well as both C4 Mobile OS platforms (iOS and Android). Because success was uniformly present across all devices and platforms, the issue is likely NOT due to any specific hardware failure. More importantly, any resolution will likely NOT involve replacing a device.
A hardware issue would have been made obvious if one model or connection type (WIFI vs PoE) had been absent or barely represented when looking at healthy calls.
REVIEWING FAILED-VIDEO DATA
Similar to investigating healthy call data, our teams were able to recognize important patterns that are now guiding our initiatives to resolve this issue. Perhaps most importantly, of the 6% of calls that fail, the calls that connect over a cellular network have a 3x higher chance of failing when compared to calls completed over WIFI. Because WIFI calls exhibited a statistically higher number of healthy calls, the data implies that either a weak WIFI signal or an issue with the local network resulted in the dropped video. Moreover, the mobile phone model, OS type, mobile app version, and Control4 controller version did not demonstrate any outlying behavior. The data suggests that failures are predominately from how carriers are handling our intercom calls. While cellular intercom calls typically have a 3x higher failure rate, 2 carriers demonstrated a much higher failure rate: AT&T, especially on 5G, and Verizon.
NEXT STEPS AND TROUBLESHOOTING GUIDE
With this data in hand, we have approached AT&T and Verizon about these anomalies and continue to work to identify the ultimate root cause for these failed video connections. We are also working hard to provide a backend fix on our side (addressing how our cloud talks to the carrier’s cloud), which means a resolution would be delivered “over the air” that would involve minimal action, if any, on the side of partner / end-customers; most likely through a software update.
In the meantime, we have compiled an “Intercom + Troubleshooting Best Practices Guide” for our Control4 Partners, which is a step-by-step guide that provides:
Hardware Verification: checklist of installation best practices and recommended settings and configurations for the Chime/DS2 and their drivers.
Local Network Validation: checklist of Intercom Anywhere system requirements, including driver configurations, healthy ISP upload speeds, and (if needed) strong WIFI signal strength.
ISP/Cellular Provider: best practices for optimal performance, even when proper network conditions cannot be met.
The most critical and helpful steps in that guide include:
Upload Speed from ISP: Perform a speed test to verify that the average range in the ISP upload speed does not fall below 10Mbps. If the average range over time consistently drops below 10Mpbs, the Partner should recommend upgrading the Customer’s service plan with their ISP.
Connection Strength and Driver Config: Within Composer Pro, verify the connection strength for the Chime or DS2 (recommended to no less than –70dBm), and if necessary, configure the Chime with the following settings to ensure maximum performance even in poor network conditions:
Frame Size: 1024x768
Max FPS: 15
Bit Rate Control: Constrained
Quality: Excellent
Max Bit Rate: 1 Mbps
Policy: Frame Rate Priority
POSSIBLE WORKAROUNDS
If all best practices and configurations have been employed and the customer still experiences Intercom Anywhere calls with a failed video connection, the issue is likely between the Control4 cloud and the cellular provider. As we prepare to roll out our first back-end measures to address this issue, the customer can try the following workarounds to improve the video connection:
Connect Over WIFI: WIFI calls are 3x more likely to connect video than cellular calls. If possible, be sure to connect the mobile device to WIFI before calling or answering a call. NOTE: Free WIFI hotspots generally limit users to a maximum of 5Mbps upload speed which is not enough for a video connection.
Connect Over 2.4GHz/LTE: The data suggests issues with video connections when using the 5GHz band or the 5G cellular network. Try connecting the mobile device to a 2.4GHz WIFI SSID or turning off 5G to ensure an LTE/4G cellular connection.
Mass market VPNs can introduce network issues and further restrict bandwidth. If the customer must use a VPN, ensure it is highly rated and properly configured.
We recognize the frustration that this issue has caused and apologize to our Partners and customers. Our team is committed to the quality of our products and services, ensuring that they meet the needs of our Partners and customers. As such, we are dedicated to resolving these issues as quickly as possible. We will continue working to improve the customer experience with Intercom Anywhere, even when proper network conditions cannot be met.
Dec 20, 2022 - 11:38 EST
About This Site
Status of Snap One Product Platform Cloud Sites and Services.
End-Customer Services Health
?
Operational
Mobile App: Control4 Intercom Anywhere Android
?
Operational
Mobile App: Control4 Intercom Anywhere iOS
?
Operational
Mobile App: Control4 OS 2.x Android
?
Operational
Mobile App: Control4 OS 3.x Android
?
Operational
Mobile App: Control4 OS 2.x iOS
?
Operational
Mobile App: Control4 OS 3.x iOS
?
Operational
Mobile App: LumaLink
?
Operational
Mobile App: OvrC Connect
?
Operational
Service: Control4 Access Anywhere
?
Operational
Service: Control4 Alexa
?
Operational
Service: Control4 APIs
?
Operational
Service: Control4 Google Assistant
?
Operational
Service: Control4 Intercom Anywhere
Operational
Service: Control4 Storage
?
Operational
Service: Control4 Updates
?
Operational
Service: Control4 Weather
?
Operational
Service: Control4 When >> Then
?
Operational
Service: Snap One DDNS [Luma, Wattbox, and other products]
?
Operational
Website: www.control4.com
Operational
Website: customer.control4.com
Operational
Website: triadspeakers.com
?
Operational
Partner Services Health
?
Operational
Service: Control4 Composer Express
?
Operational
Service: Control4 Composer Pro
Operational
Service: Control4 Drivers
?
Operational
Service: Ihiji Invision
Operational
Service: OvrC Device Communication & Configuration
?
Operational
Service: OvrC Firmware Updates
?
Operational
Service: OvrC Login
?
Operational
Service: OvrC Notifications
?
Operational
Service: OvrC Public APIs
?
Operational
Service: OvrC Web Connect
?
Operational
Service: Pakedge BakPak
?
Operational
Website: dealer.control4.com
?
Operational
Website: dealer-emea.control4.com
?
Operational
Website: dealer.pakedge.com
?
Operational
Website: dealer.triadspeakers.com
?
Operational
Website: forums.control4.com
?
Operational
Website: my.control4.com
?
Operational
Service: OvrC Speed Test
?
Operational
Website: www.ovrc.com
?
Operational
Third Party Integrations Health (e.g., Spotify, Jandy, Tidal, etc.)
Operational
Alexa
?
Operational
Amazon Music
?
Operational
Deezer
?
Operational
ecobee
?
Operational
Google Nest
?
Operational
Google Voice
?
Operational
iHeartradio
?
Operational
Jandy
?
Operational
Napster
?
Operational
Pandora
?
Operational
Qobuz
?
Operational
SiriusXM
Operational
Sonos
?
Operational
Soundmachine
?
Operational
Spotify
?
Operational
Tidal
?
Operational
TuneIn
?
Operational
Third Party Dependencies Health (e.g., Amazon, Akamai, The Weather Channel, etc.)
?
Operational
Akamai
?
Operational
Apple APNS
?
Operational
Amazon Web Services
?
Operational
MongoDB
?
Operational
Ookla
?
Operational
Paytrace
?
Operational
The Weather Channel
?
Operational
Paytrace
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.