Travis Hegner | Ramblings of an I.T. guy.

Running Multiple Pass Repositories

Mon, 14 Sep 2015 00:00:00 +0000

I’ve been using Pass: The standard unix password manager to store and manage all of my passwords for quite a while now. It is essentially a shell script wrapper around many other standard *nix tools, but it makes the task of password management a lot easier to do. It basically stores all of your passwords (and additional info, if you so choose) as flat text files encrypted with GPG. It has tasks for viewing, editing, and generating new passwords very easily. One of it’s best features is that it can treat your password store as a git repository, so that historical passwords are saved, and it’s easier to sync your whole password database between multiple machines.

I’ve ~~finally~~ recently convinced my co-workers that we should use the pass system to share all of our work passwords. This would allow us to stop using the same password for everything, while simultaneously allowing us to easily keep each other comprised of the current password for each device. I quickly found myself in a dilemma though, because I wanted to keep my personal pass repository, as well as my work one, on the same machine.

Pass defaults to using ~/.password-store for it’s storage directory. When executing any of the pass commands, it uses that directory to look for your encrypted files, and do tab completion. I really wanted to keep both repositories available, so what could I do?

Digging through man page, I discovered that I can set an environment variable to control the context of the password store directory. Helpful, yes, but kind of a pain to have to switch the environment variable back and forth in order to get the info from the correct repository. So I decided to set up some custom functions to allow me to query either password repository on a whim.

tpass() {
  PASSWORD_STORE_DIR=/home/thegner/git_repos/passdb pass "$@"
}

#for working autocomplete:
compdef \_tpass tpass
\_tpass() {
  PASSWORD_STORE_DIR=/home/thegner/git_repos/passdb \_pass
}

With those two functions added to your .zshrc, you can easily query a second password repository as you see fit. When I want to query my second password repository, I simply use the tpass command in place of the pass command. The second function even provides a working auto-complete for commands and filenames on the second repository. You could add as many password repositories as you desire by adding these two functions for each one with unique names.

Cloudera Manager: Too Many Open Files

Fri, 04 Sep 2015 00:00:00 +0000

Working with cloudera manager 5.4.6, I recently discovered a bug in the way they handle limits on the cloudera-scm-server service. The symptoms start with the web interface of the cloudera manager being completely unresponsive when trying to access it. SSH into the host running the service, and it’s java process is consuming a lot of CPU.

Executing:

sudo service cloudera-scm-server restart

Will fix the issue, but only temporarily.

As you dig into the issue, you realize that the service is still logging to the .log.1 file, rather than the normal .log file. Tailing the file will yield the error Too Many Open Files.

First I tried increasing the overall fs.max-files for the whole system with sysctl, but the issue still occurred. After some research, I found that the process itself was hitting it’s own limits, via the /etc/security/limits.conf system. The system also includes a /etc/security/limits.d/ directory for custom settings. In the directory is even a file for cloudera-scm.conf. In that file, there are settings for the cloudera-scm user which up the nofile soft and hard limits.

After discovering this, and analyzing /proc/<pid>/limits for the cloudera-scm-server service, I realized that the settings for that user were not being applied to that process. More research showed that the limits system is only applied at login through pam, and this service was being started by an init script.

I added the following ulimit commands in between the lines as shown:

echo -n "Starting $prog: "
ulimit -n 32768
ulimit -Hn 1048576
$CMF_SUDO_CMD /bin/bash -c "nohup $SERVER_SCRIPT $CMF_SERVER_ARGS" > $SERVER_OUT 2>&1 </dev/null &

And cloudera manager has been running well since.

You did WHAT with iptables? (Part 4: The "Revert to old behavior" Solution)

Thu, 30 Apr 2015 00:00:00 +0000

I was conferring with my newest colleague, and explaining to him the hack job I had to do to make music on hold work. Apparently, his google-fu is better than mine, as he found this gem:

https://supportforums.cisco.com/discussion/11890741/unicast-ucm-music-hold-fails-through-cube-asr

Turns out, they did put in a “revert to old behavior” option. It’s called “Duplex Streaming Enabled”.

Needless to say, that works much better than what I was doing.

We Got Reddited

Thu, 12 Mar 2015 00:00:00 +0000

So, someone on reddit thought it would be rather funny to post a link to this image on our server at work:

It was posted under the heading NSFW (Not Safe For Work). If you aren’t aware, that typically denotes a post to something inappropriate in nature. It’s work safety gear, so it’s pretty funny. One of the comments on the thread suggested that: “Their sysadmin is going to be very confused.” Indeed, we were:

The original image was rather large in nature, so we were moving a lot of outbound traffic (a lot for us anyway). So, to mitigate, I resized the image, and did an http 301 redirect back to a custom (and very slim) html page:

We’ve never had that much traffic before, and we appreciate the opportunity to see what it’s like, even for a silly reason.

Raising Boys

Tue, 22 Apr 2014 00:00:00 +0000

You did WHAT with iptables? (Part 3: The Actual Solution)

Fri, 17 Jan 2014 00:00:00 +0000

If you didn’t guess by the title, this is Part 3 in a whole series about abusing iptables. It’d be wise to read Part 1, then Part 2, then come back here.

If you are looking for a copy/paste answer, you won’t find it. The actual solution is spattered throughout all three parts to tell a story of triumph over data packets. A story of solving a Layer 7 problem with Layer 4 technologies. Some of the code you’ll find is not part of the end result, but shows the thought process I used to make it to a final solution. This is intentional to make you think about what you are actually doing before you copy and paste my highly specialized, for my environment only, commands and code into your environment and cause you to completely ruin your whole network, and eventually lose your job. Your Welcome.

Nevertheless, the information contained in all three parts of this article, as well as in every other article on this blog, is purely for entertainment and informational purposes only. If you use this information and break something, then I’m not responsible. Just go fix it and leave me alone.

Good, now that we have that settled. Let’s get on with it. Now, where did we leave off? Oh, right, starting over.

I contemplated putting a media proxy between my UCM and the phone provider, but there was no guarantee that the media proxy would actually re-originate the RTP stream since each leg of the call would be using the same codec. There was also no guarantee that the (open source, of course) media proxy would play nice with the whole broken source port thing anyway. Plus that would have been a lot of work since I had no way to know before injecting a media proxy that the call was being held or resumed. And I’ve worked really, really hard to prevent my users voice packets to come back through the VPN, into my data center’s internet connection, and right back out again to the provider. I’m not going to start now. Besides, that would destroy the voice quality that my users have come to expect.

I contemplated standing up another instance of UCM in my cluster. A colossal waste of compute resources in my virtual environment just to have a dedicated MOH source. That way I wouldn’t have to do any detection, and I could blindly SNAT the source port and go on with life. But then I remembered that this would have the same problem with connection tracking as the current iptables mess that I had created.

So my best options were my own custom kernel module to add a stateless SNAT sort of target, or to look into this QUEUE target that I had seen some references to. At this point all I knew was that the QUEUE target made the data packets available in user-land. Alright, this looks promising. I quickly learned that NFQUEUE is basically a more robust version of QUEUE, so I change gears and headed in that direction. With everything learned so far, I simply needed to figure out how to get the packet out of the queue, modify the source address and port, then send it back to the kernel. Sounds easy right? Sort of, The standard way to do this is to write a pure C program against libnetfilter_queue. Great, that is almost as hard as a kernel module. Possible, but not something that I can do overnight. Then I keep reading down that link and I find nfqueue-bindings. Awesome! Someone has written perl libraries which will allow me to connect to the queue, grab packets, issue verdicts, and even send a modified packet back through. PERFECT!

So I change my MOH-BLOCK chain to send packets destined for a host in the MOH set to -j NFQUEUE --queue-num 0 instead of -j ACCEPT.

Then I copy the rewrite.pl example from the nfqueue-bindings project on github.com, and hack it up to do what I need. I fire up the script as root, and after a couple of snags (make sure you load the nfnetlink_queue kernel module), it was off and running.

The outbound MOH packets from my UCM are marked with NOTRACK, then queued for my perl script to statelessly flip the source ip and port. Music on hold is heard in every call scenario I can think to test, no matter how many times the caller is placed on hold. It is a sloppy and beautiful thing.

I know what you are thinking (not really), so I’ll try and answer those questions now.

“What happens to the packets coming from the caller while they are on hold?”
Since the outbound packets which match that “connection” aren’t tracked as a connection at all, they hit the default rule of the forward table and are dropped. I value our customers privacy, so I won’t do it, but wouldn’t it be neat to capture those packets and store them somewhere. Then we could go in and decode those RTP streams to hear what people are actually saying while they are on hold. Sounds like the beginning of what could be a successful, and hilarious, website.

“How well will this solution scale?”
I don’t know. How much CPU does your firewall have available? In my tests, I’m seeing about 1% CPU per MOH stream on a 2.4GHz processor. I have only ever seen 7 simultaneous MOH streams in my environment, so I don’t have anything to worry about currently.

“That’s nothing. I need it to scale higher than that.” Great. I’m happy for you. Write a pure C daemon against libnetfilter_queue, and that should do better.

“Pffft. Child’s play, I need more!”
OK, then. Write a custom kernel module to do stateless nat/pat/pat-sat-on-(hat|cat|bat). This will prevent the data from having to be copied from kernel space to user-land and back. This should prove to be a significant performance increase. Please let me know if you do, and if it’s open source, because I could really simplify my configuration with something like that.

“Wait a minute. Didn’t you say that your inbound local calls worked fine? If this is the solution, then how did those work in the first place?”
Oh, you were paying attention. Good Job, I like that. That is a funny story really. In all of my testing and analyzing and ruling out certain possibilities based on the fact that particular scenario worked, I discovered a large security hole in my providers network. Ironically, what worked only did so because of a broken provider configuration. Wrap your head around that. Basically, it seems, that calls initiated from their network outbound, don’t seem to have any firewall protection on the media gateways terminating the RTP stream. What that means is that given a known ip/port which is currently receiving an RTP stream for a currently on-going voice call, I can forcibly inject my own RTP stream regardless of what the source ip address and port are. Depending on the packet timing, you can hear the injected audio clearly, otherwise it comes across as a jumbling of the two streams mixed, or as just garbage. In any case, it means that a random internet stranger has the power to interrupt their customer’s (or their customer’s customer’s rather) phone calls. I notified my provider in one of my trouble tickets with a detailed account of what I believed was happening, and I was told that “they’ll look into it”. As of this writing (one week later), they have yet to fix it. Figures.

Well there you have it. One man’s account of a seemingly endless battle with disparate SIP implementations, which ultimately ended up being ~~solved~~ worked around with perl. Isn’t everything done in perl a workaround anyway?

No, I won’t sync your outlook with your new phone.

You did WHAT with iptables? (Part 2: The Almost Solution)

Fri, 17 Jan 2014 00:00:00 +0000

If you haven’t read Part 1: The Problem, I’d suggest you go do that first.

After discovering the actual problem, I started pouring over UCM documentation and options. Different combinations of options cause the SIP trunk to behave in different ways, but there doesn’t seem to be anything related to this mystical port 4000. Finally I discover some documentation covering UCM port usage which lists TCP/4000 as a “phantom” port that UCM uses when it’s not expecting to receive any return data. Well, it appears that UDP also uses this so-called “phantom” port because that is exactly the nature of it’s use in this scenario. In most cases, this wouldn’t be an issue, but when you try to run SIP over the internet, it becomes a blatent issue because of everything described in Part 1. Even in the advanced options, there is no “revert to old behavior”.

Why do they do this? I can only speculate, but I would guess it’s to shave a handful of milliseconds off of the SDP body creation process. Since it isn’t expecting to actually receive any data they can simply shove a hard coded “phantom” port into the SDP, without actually having to consult the networking stack to find an available port to be used later. It’s a little silly, because once we actually send the RTP stream, it has to consult the network stack for an available port anyway, they are simply putting the delay later into the dialogue, and then sending the RTP with some unknown random high-port. This simply proves that Cisco’s view of SIP is a little box on my local network which does it’s own magic with a provider. Well, my environment doesn’t work that way. I run a pure software phone system all the way to the provider, all over the internet.

Why? Because a typical branch was taken from an $800/mo phone bill, to $50/mo. This equates to well over $100,000 per year in telephone savings across the organization. I have hundreds of endpoints across the country all sharing from a pool of dozens virtual phone lines, with hundreds of “local” phone numbers across the country. Right now, it costs a branch about $10-$11/mo per desk phone. This is with all taxes, usage, and long distance included. And it scales magically. The more phones I add to this system, the cheaper it gets per phone.

Alright, so I realized that can’t solve this problem by changing the source of the problem, I’m going to have to get a little creative. Let’s do a quick re-cap of what I needed to accomplish. I needed to:

Force the remote end to send the RTP stream even while the caller is on hold.
From the firewall, identify RTP packets that are part of an MOH stream.
Do an SNAT on those identified packets to change the source port.

So, number one is easy, I’ve done it a couple of times already. Write a function in my SIP proxy which flips the “a=sendonly” attribute over to an “a=sendrecv” attribute. I’m already mangling all of the SIP packets anyway to assist in my remote branch NAT traversal. Done.

Number two seams simple at first, but then you remember that the UCM does more than MOH. And there is no way to distinguish whether the packet matching this rule is part of a voicemail, a conference, or an MOH stream. If I change the source port of the wrong packet, I’ll break things worse than they already are. The only way to identify the type of packet is by looking at the traffic coming in from this packet’s destination. So we have to look at the connection level of the firewall. Should be easy right? On our fancy statefull firewall, with our fancy connection tracking? Not so fast.

In theory, I can identify a music on hold stream if the provider is sending it to the destination port of 4000 right? Sure! Then I can mark the connection and manipulate the return traffic however I please! Yeah!

Wait, that didn’t work, Why? Because the conntrack module doesn’t see the two streams as the same connection. Why? Because the outbound source port doesn’t match the inbound destination port. DOH! A Catch 22. Try Again.

Well, I still needed to identify destinations (address/port combos) that were currently part of an MOH session, since that was the only way to distinguish outbound MOH packets. I had just learned about port knocking, so I started looking into those techniques with the recent module. I quickly discovered that the recent module only stores remote ip addresses and not ip/port combos, for good reason based on what it’s designed to do. Also, I can only match against the source of a packet. In this scenario, I will at some point need to match against a packet’s destination.

Then I discovered IPSet. With IPSet I can create dynamic lists of ip/port combos, based on certain criteria of a packet. I can also match against ip/port combos within a set. Source or destination. Sweet.

sudo ipset create MOH hash:ip,port timeout 1 sudo ipset create TWO hash:ip,port timeout 1

So these commands create two lists to hold ip/port combos. I have to have two lists so that I can block all outbound RTP streams from UCM, until the stream has been properly identified as either music on hold (MOH) or two-way (TWO). Since the UCM is so much closer to my firewall than the remote end, I will almost always receive those un-identified RTP packets first.

You wouldn’t expect this to be an issue, but I learned (the hard way, of course) that the outbound packets establish an entry in the conntrack module, and only the first packet of a connection is matched against the nat table. So if I receive an outbound packet first, it’s un-identified, and therefore not in the MOH list. Then it goes out with the default SNAT rule for my UCM. All future packets in this stream are matched and NATed the same way, even though we have since added the destination to the MOH list (upon arrival of the first return packet). I could find no way to clear the connection state of a matched packet from an iptables rule, forcing a re-evaluation of the nat table rules. Perhaps this could be a future enhancement. Maybe I’ll get around to a feature request one day. It would simplify this particular configuration by a lot, as you’ll soon discover.

Continuing, let’s Identify the streams, and add them to the correct set using the SET target:

sudo iptables -t raw -N MOH-LIST sudo iptables -t raw -A PREROUTING -d 1.1.1.1/32 -i eth1 -p udp -j MOH-LIST sudo iptables -t raw -A MOH-LIST -d 1.1.1.1/32 -i eth1 -p udp -m udp ! --dport 4000 -j SET --del-set MOH src,src sudo iptables -t raw -A MOH-LIST -d 1.1.1.1/32 -i eth1 -p udp -m udp --dport 4000 -j SET --del-set TWO src,src sudo iptables -t raw -A MOH-LIST -d 1.1.1.1/32 -i eth1 -p udp -m udp ! --dport 4000 -j SET --add-set TWO src,src sudo iptables -t raw -A MOH-LIST -d 1.1.1.1/32 -i eth1 -p udp -m udp --dport 4000 -j SET --add-set MOH src,src

In (mostly) plain English, this set of commands takes the source address and port of any UDP traffic destined for port 4000 and removes that combo from the TWO list, and also stores that combo to the MOH list. For any UDP packet NOT (!) destined for port 4000, it does the opposite. The removals are necessary because in the case of a transfer to voicemail, an RTP stream is quickly changed from MOH to two way, and a lingering entry in the list (even with a one second timeout) could cause us to rewrite valid packets. Also take notice that we are doing this in the raw table. This is because the packets we are using for identity are ultimately dropped by the forward table, as there isn’t a connection established yet.

When I originally had this identification process set up at the beginning of the forward table, the ip/port combos were never stored to the list. I will assume it’s because they were dropped before the chain was finished being processed. Now, let’s not forget to block the un-identified outbound RTP packets, and allow the identified ones through using the “set” match:

sudo iptables -N MOH-BLOCK sudo iptables -I FORWARD -s 10.10.10.10/32 -i eth0 -o eth1 -p udp -j MOH-BLOCK sudo iptables -A MOH-BLOCK -s 10.10.10.10/32 -i eth0 -o eth1 -p udp -m set --match-set TWO dst,dst -j ACCEPT sudo iptables -A MOH-BLOCK -s 10.10.10.10/32 -i eth0 -o eth1 -p udp -m set --match-set MOH dst,dst -j ACCEPT sudo iptables -A MOH-BLOCK -s 10.10.10.10/32 -i eth0 -o eth1 -p udp -j DROP

First we direct all outbound UDP packets to our new custom chain “MOH-BLOCK”. Here we can test whether the current packet has been identified as an MOH packet, or a two-way packet. If the packet is identified, the firewall allows the outbound packets through. Now we just have to do an SNAT for our public address and port 4000 on only our MOH packets (obviously before our default SNAT for UCM)!

sudo iptables -I POSTROUTING -s 10.10.10.10/32 -o eth1 -m set --match-set MOH dst,dst -j SNAT --to-source 1.1.1.1:4000

Once the outbound packets are through, a connection is established and the inbound packets are let in as well. And this works! BRAVO!

“What’s that,?” “Oh, the second time,?” “Ah, I see, I’ll look into it.”

As luck should have it, our master plan has a flaw. If you put the caller on hold, they hear hold music. If you take them off of hold, and then put them back on within 180 seconds, they don’t. Interesting.

So I begin digging again. During the second time the user is on hold, I can see the RTP stream coming from the provider, and I can issue a sudo ipset list and verify that the remote destination is correctly placed in the MOH set. I can see the outbound RTP packets entering the firewall, but never leaving. What is it this time!?!

I happened to notice something interesting in all of this analysis. My provider, no matter how many times I re-establish the media stream to a different endpoint, no matter the reason, they always re-establish the media stream with the same remote address/port combination. My UCM does not. Each time the caller is placed on hold, UCM begins using a new random high-port.

Now here is where it get’s interesting (you mean you aren’t interested already? This stuff is amazingly entertaining!). A UDP “connection” as it’s seen by the state machine is simply a set of ip/port combos that identifies traffic that has come through the firewall within the last X seconds. With an UNREPLIED connection (one way traffic, no return traffic) that connection lasts 30 seconds by default. With an ASSURED connection (two way traffic from same ip/ports in both directions) lasting 180 seconds. The timer is re-set with each packet that traverses the connection.

Since we are now SNATing the port, the new set of ip/port addresses matches an existing connection (Really conntrack? Now you want to match my connection!?). But since the original source port from my UCM is not the same as what the connection was established with, the firewall thinks it’s a port address collision and drops the outbound RTP packets (Or, I assume that is why anyway).

Not to worry though. We use the most powerful and flexible firewall in the world! We use iptables! Just jump into the raw table again and drop a NOTRACK target onto our outbound MOH RTP stream.

sudo iptables -t raw -A PREROUTING -s 10.10.10.10/32 -i eth0 -p udp -m set --match-set MOH dst,dst -j NOTRACK

Alright! Just a quick packet capture to make sure everything is wor, ewww, Yeah, shoving private IP addresses out to the internet is definitely not going to work.

Not to fear! We can just simply do,. a,, stateless SNAT? No? Maybe the raw table has a target to,? No. What about the mangle table, that would be the perfect, Not there either? Huh.

I guess the nat table requires connection tracking. It’s for performance reasons I suppose. Since it doesn’t evaluate the nat table for every packet, it uses the information stored as part of the connection to correctly re-write the later packets as they traverse the firewall. I’m a little surprised though that there really isn’t a way to just manually flip the address and port to what I want it to be. Understandably, how could kernel developers foresee a need to do stateless nat when we have a very nice, full featured statefull nat system available. Of course, a target could be written for the mangle table to do what we need I’m sure. It is, after all, in the open source world. But that would take me a long time, because I’ve never really written a kernel module before.

So is it back to the white board on this one? Find out in Part 3!

You did WHAT with iptables? (Part 1: The Problem)

Fri, 17 Jan 2014 00:00:00 +0000

Technically speaking: stateless source address and port translation. Only on relevant packets, which are indistinguishable at the packet level from irrelevant packets. Why? Well, Music On Hold, of course.

The article that follows is dedicated to all of my friends and family who encompass my entire career as: “He fixes computers”. Here is a brief glimpse at a small part of what I actually do. In reality, I’m not even good at fixing computers anymore, and I don’t like doing it. Especially when the computer is running windows. Double especially when the computer is running windows 8. What an atrocity.

WARNING: In order to understand anything written here, it is recommended that you have moderate to expert knowledge of the following technologies: Linux, NetFilter/IPTables Firewalls, UDP/IP Packets, SIP/SDP, RTP, NAT, and Cisco UCM. I think that should do it, for this article at least.

Oh, you’re still here? This article is more so meant as a brain dump of documentation while it’s still fresh in my mind, I can’t believe you’re still planning to read it. Well then, you asked for it.

It all started when we were rolling our branches over to the new version of UCM. I was testing the fancy new call queuing features when I noticed that callers from the PSTN over SIP were not able to hear the initial announcement when entering the queue, or the music on hold while waiting patiently in the queue. The caller would simply hear dead air until either they hung up, or the call was transferred to an available rep. I worked on it for a few weeks, and then got bored and gave up, never really discovering the true nature of the problem. Well, eventually, after ignoring it for as long as I could, I dove back into the problem head first. By this point, we were fully operational on our new UCM Cluster, and the old UCM Publisher had been happily retired to the basement.

I began troubleshooting and discovered that the MOH treatment was failing on inbound toll-free calls, and outbound calls. Oddly enough, it worked on inbound local calls with an identical configuration. Surely this has to be a provider issue right? I have a working and non-working setup with identical configurations, the only difference is at the provider, well, yes and no, but I’ll get into that later.

I put in a couple of support tickets with my provider, which I always regret because I typically get the “we can’t support late offer” dance (I am always tempted to scream “shibboleet”). Even though my old UCM cluster did nothing but delayed offer, and it worked perfectly. Thanks. So I kept digging. An oddity I noticed early on is that in the new version of UCM (actually, in every version since 8.5 or so), they started sending an “m=audio 4000,” line in the MOH SDP answer. In normal SIP, that 4000 would be the UDP port to which the remote destination is supposed to send it’s RTP stream. It’s typically a random high port. But, that shouldn’t matter in this case, because the other new thing in this SDP is “a=sendonly”, which means this is a send-only stream, and that we aren’t expecting to receive any RTP data back from the remote party anyway. Also, remember, that this can’t be the problem, because it works fine with inbound local calls, and my UCM is doing the exact same thing for those. Just to be safe, I tried removing the “a=sendonly” line, but it didn’t help.

I poured over packet capture after packet capture for hours analyzing SIP packet after SIP packet and SDP body after SDP body looking for something, anything, different between the working and non working examples. Alas, there was none. I was finally defeated enough to bring my old, decrepit, UCM Publisher up from the basement, restore my old SIP proxy from backups, and re-create that whole old environment so that I could get a packet capture of a working example from that. By this point I was capturing the full SIP dialog, and the RTP stream post firewall, just to make sure there wasn’t an issue with NAT traversal (that can be a bane when dealing with SIP, you know). So I captured only what I needed (all UDP traffic on the public interface of my production firewall), and the first thing I notice is that I am receiving an RTP stream from the remote end even when the call is on hold. Interesting. Come to think of it, that makes sense because the old UCM is not sending an “a=sendonly” audio attribute. But I was still stumped because I had tried removing that in my new environment and it did not help. Not to mention, the local inbound calls work fine even with a true send-only stream.

Intrigued by this new discovery (read: not overlooking this behavior anymore), I flipped my “a=sendonly” to “a=sendrecv” in my new environment (removing it is harder, and the SDP RFC says it’s the default when missing anyway). Sure enough, a new packet capture showed that I was receiving an RTP stream from the remote device during an MOH session. But still no music on hold. Comparing the old environment SDP to the new environment SDP, I went through every difference, and modified my new environment to match exactly the old one with one exception “m=audio 4000,”.

When you are looking at typical networking, a client will send data to a destination server at a particular destination port. Well the client must also provide a source port for that data on which it expects to receive possible responses. In a similar way, nearly all SIP devices will utilize the remote end’s destination port (learned during the SDP exchange), as the source port of it’s RTP stream. This is smart because it helps with the traversal of statefull firewalls, when a connection is initiated from a trusted network, typically the firewall dynamically allows the return traffic (traffic with the opposite source and destination ip/port) through without question. This is because typically once the connection is initiated, traffic is expected to return. This makes a firewall administrator’s life a lot easier. With iptables, the command which does this looks something like the following:

sudo iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

Now I already knew that this was how most SIP devices behaved, so as a test at one point in all of this I flipped any UDP traffic destined for my provider from my UCM to be from source port 4000. This still didn’t work, so I quickly flipped it back to normal. The UCM box not only does RTP for music on hold, but also does call conferencing, and media relay for our voicemail system (apologies for anyone that was trying to leave a voicemail or run a conference call during that test). The odd thing to remember is that all two-way audio works perfectly fine. What in the world!?!

So I did like Winnie The Pooh, and went back to my old UCM packet capture. Sure enough, the source port of it’s MOH RTP stream was in fact the same as the destination port which it had advised the remote end to send to in the SDP (even though it’s music on hold). Then a light-bulb went off. I’m probably not the only sysadmin in the world that runs a dynamic firewall for established connections. So the old system worked because of the combination of a proper source port, and the remote destination actually sending the RTP stream to the supposed destination port. That RTP stream is required to establish a connection through the remote firewall, even though it’s never actually used. OK, now I had a solid theory and I needed to test it. I don’t really like disrupting production traffic, so I had to do something different this time. That’s when I stumbled onto rtptools. Sweet, so I can use my test SIP proxy to completely obliterate my UCM’s SDP, and make the remote end send RTP to what will be my desktop. This creates an established connection on the remote firewall, expecting return traffic from the correct address/port (my desktop, remember). Then, with my live capture running I can discover the current RTP destination address and port of my cell phone, and use rtpplay to send a sample RTP stream with the proper source and destination parameters. OK, here goes:

./rtpplay -f ../pcap/g711u.rtp -s 4000 2.2.2.2/12345

Wow, the most beautiful 5 second slice of hold music I’ve ever heard played from a cell phone speaker. I played it over and over again because I couldn’t believe what I was hearing.

Now I had positively identified the problem. How would I go about solving it? Let’s walk through a few things I have tried at Part 2: The Almost Solution.

Every Parent's Worst Fear

Wed, 30 Oct 2013 00:00:00 +0000

Any man that goes to a real barber knows that a barber is more than just a guy that cuts your hair. He is a listener, a counselor, a therapist, and most importantly, a friend. My friend, who is also a man of faith, and a loving and devoted father, just learned that his only child – a 6 year old son – has cancer.

As the husband of a cancer survivor, I can relate to the gut-wrenching fear of not knowing what is going to happen. I can sympathize with the willingness to do anything to make it go away, and the feeling of helplessness when there is nothing in your power to do that.

As the father of three beautiful young sons myself, I know that this kind of news is every parent’s worst fear. I literally wept for my friend when I found out.

My friend is self employed, and now has to balance between time at work to make sure medical bills are paid, and this precious most important time with his son as he goes through 51 weeks of chemotherapy. If each of my friends and connections were to take 5 minutes of time to donate the price of a cheeseburger, and then re-share this post to your own friends and connections, I have no doubt that my friend could spend a lot more time with his son – where he should be. I’ve seen scams that look just like this fly across social networks being shared by thousands of people. Please rest assured this is no scam, and these are very real friends of mine, who are in very real pain.

You can read more about the situation on my friends website: http://pennfieldbarbershop.com/

You can easily donate on-line at the following link: ~~https://www.giveforward.com/fundraiser/c8c3/help-maddux-in-his-battle-against-cancer~~
UPDATE: A new donation URL is available: http://www.gofundme.com/helpmaddux

Let’s come together and give this family one less burden to deal with, and allow them to fight this disease, and heal, together.

Thank you,

Travis

Online Shopping

Thu, 07 Feb 2013 00:00:00 +0000

Dear Online Retailers,

Please do not force me to create an account on your site for what is most likely a one time purchase. I understand that you want my email to spam me, but there is no need for you to keep an account and probably store my password and credit card information indefinitely.

Chances are, if I do happen to shop at your site again, I won’t remember that I have an account, nor the password for that account, and it will simply make it harder for me to check out with my potential future purchase. That is all.

Sincerely,

Annoyed Customer