I’d tell you a UDP joke, but you might not get it!
Nothing is more apt with the problem I have just finished troubleshooting: The classic VoIP RTP traffic one way audio issue. One day our Cisco IP Telphony system was purring like a kitten, the next day not so much.
I was recently logged a myriad of tickets about one way audio issues. Calls coming in from the PSTN, being transfered and then the calling party not hearing the called party. I also had calls transfered to our DIDs experience the same one way audio issues. Unless there has been a change, this type of problem usually doesn’t appear out of nothing. I had to crack out the standard Wireshark, Cisco RTMT and CUBE to work out where the fault lied.
We had two issues:
1st Problem = Internal transfers were showing CUCM IP in SIP signaling as the media endpoint. Not the actual IP address of the IP Phone (which it should be). After all CUCM does not do media unless your using MTP.
2nd Problem = Transfer from PSTN IVR to a DID at our sites was experiencing one way audio. Internal extension could hear PSTN calling but not other direction.
1st Problem
I used debug ccsip messages on the voice gateway (CUBE Router) and could see that media IP was being set in SIP signaling to CUCM IP address. As it only occured during internal transfer, call pickup etc, I was able to workaround this issue with the following setting on the SIP Trunk Security Profile within Cisco Call Manager: Send send-receive SDP in mid-call INVITE
Tick this box and apply / reset the SIP Trunk on CM.
I have never had to use midcall early offer – which is what this option does. Cisco support forums is where I found the suggestion. It helps to fix when call goes on hold with some voice routers and CM sets ip to 0.0.0.0 in the media signaling.
I knew at this point, this was mearley a workaround to buy time so users could still internal transfer.3.10
2nd Problem
For this issue as the external IVR was hosted by the ITSP I could reproduce the problem at any time. I spanned the traffic from one site’s CUBE voice router and placed a test call. Using Wireshark and the Voip Analyser option I could see the following:
RTP Media from endpoint to CUBE is being sent to the wrong port.
Below are SIP traces that detail the correct RTP port signaling is sent to CUCM but the wrong port is sent back to CUBE:
** Re-invite sent from CUBE with port 18970 ***
Aug 8 13:57:04.747 AWST: //12242/BB3E4A7EB7FB/SIP/Msg/ccsipDisplayMsg:
Sent:
INVITE sip:XXX5@X.X.X.60:5060 SIP/2.0
Via: SIP/2.0/UDP X.X.1.250:5060;branch=z9hG4bKE9474F
Remote-Party-ID:
From:
To:
Date: Mon, 08 Aug 2022 13:57:04 AWST
Call-ID: BB3F34BA-161511ED-B801E050-3032D1FE@X.X.X.250
Supported: 100rel,timer,resource-priority,replaces,sdp-anat
Min-SE: 1800
Cisco-Guid: 3141421694-0370479597-3086737488-0808636926
User-Agent: Cisco-SIPGateway/IOS-15.5.3.S6b
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
CSeq: 102 INVITE
Max-Forwards: 70
Timestamp: 1659938224
Contact:
Expires: 300
Allow-Events: telephone-event
Content-Type: application/sdp
Content-Length: 312
v=0
o=CiscoSystemsSIP-GW-UserAgent 4625 8101 IN IP4 X.X.X.250
s=SIP Call
c=IN IP4 X.X.X.250
t=0 0
m=audio 18970 RTP/AVP 8 0 18 101
c=IN IP4 X.X.X.250
a=rtpmap:8 PCMA/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ptime:20
** Reinvite received on CUCM with port 62194 ***
INVITE sip:XXX:5060 SIP/2.0
Via: SIP/2.0/UDP X.X.X.250:5060;branch=z9hG4bKE9474F
Remote-Party-ID:
From:
To:
Date: Mon, 08 Aug 2022 13:57:04 AWST
Call-ID: BB3F34BA-161511ED-B801E050-3032D1FE@X.X.X.250
Supported: 100rel,timer,resource-priority,replaces,sdp-anat
Min-SE: 1800
Cisco-Guid: 3141421694-0370479597-3086737488-0808636926
User-Agent: Cisco-SIPGateway/IOS-15.5.3.S6b
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER, SUBSCRIBE, NOTIFY, INFO, REGISTER
CSeq: 102 INVITE
Max-Forwards: 70
Timestamp: 1659938224
Contact:
Expires: 300
Allow-Events: telephone-event
Content-Type: application/sdp
Content-Length: 312
v=0
o=CiscoSystemsSIP-GW-UserAgent 4625 8101 IN IP4 X.X.X.250
s=SIP Call
c=IN IP4 X.X.X.250
t=0 0
m=audio 62194 RTP/AVP 8 0 18 101
c=IN IP4 X.X.X.250
a=rtpmap:8 PCMA/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=ptime:20
Call leg from ITSP to CUBE was working fine. Call lege from CUBE to endpoint via CM signaling was showing a mismatch in the RTP ports flowing from endpoint -> CUBE. At first this stumped me. Why was the Call Manger telling the endpoint to send RTP media to a random port outside of CUBES range and what CUBE actually signaled in the first place?
Lets lay out network flow for Call manager SIP signaling (something in path is the culprit):
ITSP -> CUBE -> WAN ACCEL -> WAN ROUTER/FIREWALL -> WAN mesh -> WAN ROUTER/FIREWALL -> WAN ACCEL -> CUCM
For RTP media:
ITSP -> CUBE -> ENDPOINT – this is flow through mode
I tested with our WAN accel in bypass mode – no change. Above logs and captures sent to Cisco TAC for further analysis, they suggest to check there is no firewall in path and confirmed that CUBE and CM were not the culprits due to the timestamps in the SIP messaging above.
I then asked the service provider about the firewalls – which are Fortigate appliances. These were recently upgraded to a new code level. As per the following link SIP / RTP inspection was disabled and both problems were resolved.
For Problem 1 I was able to disable SDP in Mid Call invite and the signaling was no longer setting the media IP to the Call Manager IP Address.
For Problem 2 the return RTP port from CUCM was negotiated correctly (and not changed by the Firewalls in path) as depicted in the below successful call flow:
In hindsight there were many articles on Cisco support forums referencing firewalls and NAT with respect to RTP traffic having issues. In this case both SIP and RTP packets were being changed by the firewall. Once turned off both problems are now resolved.
Comments