How do you troubleshoot IoT devices?

Submitted by fredrik.nyman on Fri, 02/15/2019 - 13:00

Continuing on the subject of troubleshooting the network. Troubleshooting MPEG video has the benefit of a user that can tell you if it doesn't work and you can simply ask that user if the problem persists once you have fixed it. But what if there isn't any obvious way to determine if things are working, for example is that trashcan really signalling that its' full or does the temperature device really update the building climate control properly?

Image removed.

With more and more IoT devices entering the networks, troubleshooting becomes increasingly difficult. Headless (no screen or keyboard) devices depend on their preprogramed start-up sequence and their ability to connect to central resources to work properly. If it is a simple matter of broken cable, the problem might be easy to track down and resolve. But if the problem is more complicated, then so is the troubleshooting. You can't see if the device gets it's IP, you cant test ping and traceroute from it. Its just a black box and you have no way of knowing if it is properly configured.

So what if the device is connected, link is up, the routing in the network should work and yet there is no data received? That waste bin's status report does not reach to the waste management server.

At this point the seasoned network engineers usually pulls out the heavy artillery - Wireshark. Let's figure out if the bin gets its IP and even attempt to send the report and to the right destination, and let's see if there is an answer to it by looking at the packets on the wire.

By running a packet analysis tool such as Wireshark you will be able to pick up the packets and debug them one by one as it happens in the network. While the IoT device might be headless, you can determine from the packet flow what actually happens. Does the device send a DHCP packet, is the server responding, are the configuration parameters correct, does the device attempt to connect to its cloud controller.

But hold on you say, I have to bring half the office equipment with me to go out and get a Wireshark session going on the link connecting to that waste bin. It's not worth the time or effort to do that! Cables, computers, getting access to the facility… bah!

This is precisely why a feature such as the remote mirroring in Waystream's MS and ASR switches are so useful, particularly with the increasing number of IoT devices in the networks.

Image removed.

With the remote mirroring feature you can actually make that remote port very local. Here's how it works; Configure the switch to mirror the traffic on the port where the IoT device is connected over a tunnel to your desktop computer. This allows you to see the packets in and out on the port without leaving your comfortable chair. You can even apply an access-list to capture only the traffic that you are interested in.

You do not have to leave the office and go on-site with computer, cables and stuff, you do not have to bother getting access to the facilities with keys and passcodes, you can avoid having that janitor look over your shoulder stressing you out with all sorts of questions while you try to figure out what is happening.

Instead, you get going with the troubleshooting in minutes and you can debug everything from the comfort of your own desk with access to all resources you need including the ability to see both ends of the communication at the same time, usually very difficult when in the field. You can verify the DHCP, try the ping and traceroute between the server and the client to verify connectivity and see the actual report packet being sent by the client but not reaching the server.

The remote mirroring feature is a valuable tool to troubleshoot more advanced problems than simple cable issues. With more and more headless IoT devices this is precisely the kind of features needed to make troubleshooting easier.

Blog posts

Detecting service quality issues

Submitted by fredrik.nyman on Tue, 03/24/2020 - 15:31

While networks handles the increased load from Corona with ease, not all services can say the same. Detecting quality issues in service delivery is much more difficult than checking if the network itself can handle the load. There is probably a million reasons why service quality can degrade even if the network seems to be working fine. The trick to detecting and troubleshooting these situations is use of telemetry and service assurance.

European networks unvoluntary casulties of US-China trade-war

Submitted by fredrik.nyman on Mon, 09/02/2019 - 13:19

I usually don't comment on our competition but recent events such as new legislation proposal from the government in Sweden has sparked a well-needed debate. I commented on the situation for Swedish city networks last week in this IDG article https://computersweden.idg.se/2.2683/1.722350/konkurrent-varnar-huawei-stadsnat and also wrote a debat

Turn on automation of your FTTH network

Submitted by fredrik.nyman on Mon, 04/01/2019 - 09:08

The distributed nature of a fiber to the home network means that you will have equipment spread out and you might not always do the on-site installation yourself. If every switch has to pass your desk for pre-configuration port before getting deployed into the field you will need to deal with the logistics of getting the units from your warehouse via your desk, packing and unpacking, and clearly marking them so that the right unit goes into the right location.

SDN and NFV in FTTH

Submitted by fredrik.nyman on Thu, 03/21/2019 - 09:51

I love acronyms. You got three of them in the title of this post.

In recent years we got Software Defined Networking (SDN) and Network Function Virtualization (NFV). Many of the large telcos have invested millions into research of these subjects and are pushing the industry in this direction. Telefonica has expressed high ambitions to move to a completely SDN/NFV enabled network in record time. All the big ones are involved.

Keeping product lines around

Submitted by fredrik.nyman on Fri, 03/15/2019 - 09:50

Building fibre to the home networks are different from any traditional enterprise or telecommunications network. One of the main differences is the time it takes to complete the network. You make a plan, design a an architecture with VLANs and redundancy and imagine how this will scale as the number of connected customers increase. But then the years go by, because building a fibre network to connect every home in the community can take decades.

Save the planet - work from home

Submitted by fredrik.nyman on Thu, 03/07/2019 - 10:30

In my last post i revealed how dirty a fiber network can be depending on the source of electricity powering the network. I showed how a typcial 24-port access switch might contribute anything between 23kg to 485kg of carbon dioxide per year to the atmosphere depending on the electricity mix and how that can be reduced with lowpower optical modules.