XGS4600 - SFP+ Transceivers stop transmitting light

Bash

We are having an issue where both 10Gb and 1Gbtransceivers stop transmitting light at seemingly random times. This can only be fixed by unplugging and replugging in the transceiver. When looking at the statistics we get the following;

stack# show interfaces transceiver 1/30

  Transceiver Information

Port : 1/30 (SFP)

Vendor : Fibrestore

Part Number : DWDM-SFP10G-80

Serial Number : J0609190156

Revision : A

Date Code : 2016-09-07

Transceiver : 10GBASE-LR

Calibration : Internal

++ : high alarm, + : high warn, - : low warn, -- : low alarm.

neg inf : negative infinity(0mW).

               Current  High Alarm   High Warn    Low Warn   Low Alarm

                         Threshold   Threshold   Threshold   Threshold

--------------- -------------- ----------- ----------- ----------- -----------

Temperature(C) ++       25.00        0.00        0.00        0.00        0.00

Voltage(V) --        3.00        6.00        6.00        6.00        6.00

TX Bias(mA) --       0.00      131.00      131.00      131.00      131.00

TX Power(dbm) --        neg inf        8.16        8.16        8.16        8.16

RX Power(dbm) --      -11.80        8.16        8.16        8.16        8.16

Flapping the port by disabling it and reenabling it has no effect.
This issue has only started happening since an upgrade to version 4.5(ABBI.1). To fix this we downgraded to 4.4(ABBI.2) but that brings its own issues.

When looking at interface statistics on 4.4(ABBI.2) we get corrupted information;

stack# show interfaces transceiver 1/30

  Transceiver Information

Port : 1/30 (SFP)

Vendor : ��������

Part Number : ��

Serial Number : ��������

Revision : ��

Date Code : 20�-�-�

Transceiver : Others

Calibration : Internal

++ : high alarm, + : high warn, - : low warn, -- : low alarm.

neg inf : negative infinity(0mW).

               Current  High Alarm   High Warn    Low Warn   Low Alarm

                         Threshold   Threshold   Threshold   Threshold

--------------- -------------- ----------- ----------- ----------- -----------

Temperature(C) ++       30.00        0.00        0.00        0.00        0.00

Voltage(V) --        3.00        6.00        6.00        6.00        6.00

TX Bias(mA) --       64.00      131.00      131.00      131.00      131.00

TX Power(dbm) --        2.27        8.16        8.16        8.16        8.16

RX Power(dbm) --      -11.80        8.16        8.16        8.16        8.16

This is happening with a variety of transceiver manufacturers.

Zyxel小編 Lucious

Hi @Bash

We've been recently reported a similar case and now under clarification as well, please help to clarify below info,

1.
Did the port dropping (stop transmitting light) issue only happen in firmware 4.5(ABBI.1)?
Which version of firmware do you confirm with no such issue? 4.40(ABBI.2) or any other?
2. 
Given the fact that what we see in this page is all "���", we'd like to confirm if it is some kind of encoding problem between terminal software and browser(HTML). In 4.40(ABBI.2), could you provide a screenshot of corrupted DDMI information on the specific transceiver?
3.
For the corrupted info problem, could you try repeating "show interfaces transceiver" command for times to see if any change/difference? Moreover, try unplugging and replugging in the transceiver to see if any improvement.

Your cooperation will be highly appreciated!
Zyxel_Lucious

Bash

Hi @Zyxel_Lucious

Thanks again for the quick response. In answer to your questions;

1.
Yes, the port dropping only occurred since we upgraded to firmware 4.5(ABBI.1), we since downgraded to  4.40(ABBI.2) and have not seen the issue again. We have not trialled any other version of firmware to look for this issue.

2. Attached is a screenshot, we see the "���" in the terminal also.

3. I have tried repeating the command but the result is the same, unfortunately this switch is in a remote location and I cannot unpug and replug any time soon. I will try and replicate the issue on a more local switch.


Thanks,
Ben

Zyxel小編 Lucious

Hi @Bash

Regarding the 3rd answer, maybe you can try the command "inactive / no inactive" to bounce the port instead of actual unplugging/replugging.

Zyxel_Lucious

Bash

Hi @Zyxel_Lucious

As mentioned in my original post 'Flapping the port by disabling it and reenabling it has no effect.' We attempted this first of all but found it took no effect and the only solution was to physcially unplug and replug in the optic.

Thanks,
Ben

Zyxel小編 Lucious

Hi @Bash

We're now preparing a debug code (based on 4.50) to analyse the port dropping issue.
Would it be ok for you to test the code for us? It may also involve some debug command input / log capture.

We'll PM you for the further info.

Zyxel_Lucious

Bash

Hi @Zyxel_Lucious

Yes that would be absolutely fine, I can certainly test that for you.

Thanks,
Ben

B4RN_Mercer

Hi @Zyxel_Lucious,

I would like to follow this thread up. We have deployed a pre release firmware to some of our stacks which mitigates a high CPU bug but regrettably the optic issue still persists and If anything its gotten worse.

Could you please advise on the best plan of action. We're still having to manually visit site in order to pull and re seat the optic.

Thanks,

Matthew

Zyxel小編 Lucious

Hi @B4RN_Mercer

We've been analyzing and co-working with transceiver vendor regarding the SFP link dropping issue for months, it turns out that root cause is seemingly signal instability of the transceivers, depends their hardware design and IC vendor.

After all, Zyxel transceivers are recommended on the switches because they're well-verified and the interoperability will be guaranteed.

Sincerely,

Zyxel_Lucious

B4RN_Mercer

Hi @Zyxel_Lucious

We've been collecting some data since the last post to try and correlate the faults to some sort of pattern.

What we're seeing is that the xgs4600-32f has issues but not the 52f...

We have approximately 100 32f switches deployed and we're seeing the optic dropping issue almost daily.

The data we have across multiple different sites is that when the xgs4600-32f is stacked the issues occur on switch 3 pretty much exclusively. There has been times where optics have dropped on other stack members but that is incredibly rare.

Please can you continue diagnostics as we still believe that there's more to it than just our optic vendor.

Kindest Regards

Matthew

Zyxel小編 Lucious

@B4RN_Mercer

Your concern is understandable, but I would like to share that we've had similar experience from other cases and this type of IoP issue usually comes from the quality of different transceiver vendors. 

Zyxel has fully tested our transceivers to make sure that the system works stable with our own-brand transceivers.

Local Zyxel sales will contact you shortly to further support your business.

Zyxel_Lucious