Capture Challenges

How we built our 2018 Holiday capture challenge

16 min read

We’ve done several capture challenges over the years, and two of our most popular were our Trick-or-Treat and Grinchident challenges of 2018. We had several packet analysis experts take on the latter, and we put together a panel discussion with them on how they go about tackling network troubleshooting and security issues when armed with just packet captures and some notes.

The 2018 Holiday challenge

Though we've taken down some of the resources to complete the original challenge, you can read through it. It's a tough one! One of our panelists, Tony Efantis, put together a great write-up of his solution that you can explore as well.

The 2018 Holiday challenge

Though we've taken down some of the resources to complete the original challenge, you can read through it. It's a tough one! One of our panelists, Tony Efantis, put together a great write-up of his solution that you can explore as well.

We love building challenges like these, and in the process we learn a lot about networking, packets, and the tools that exist to manipulate and generate them. Settle in as we get into the details of how we create it all.

Challenge goals

When we started making this challenge, we had a couple of goals in mind. Part of the challenge was inspired by the SANS Holiday Hack Challenge and the way in which they guide you through the challenge with many steps and clues that end up teaching you as you go. We wanted to do that while exposing people to CloudShark and revisiting some of our packet analysis articles that explored things like memcached and BitTorrent.

The end goal of the challenge was for the participants to recover pieces of our company’s holiday greeting card (a jpg) and reassemble it. This gave us the idea of using a kind of “pcap-ception” (if you’ve never seen the movie Inception, watch it) where we hid a pcap inside of a pcap inside of a pcap. This plus adding encryption made for a pretty difficult challenge!

Each of the original capture chunks had a “flag” packet added to the start to indicate you were on the right track. Hidden inside the flag text of “HAPPYHOLIDAYS” were the characters for the URL of the confession pcap in CloudShark!

The chunks were hidden in various creative ways so that they related back to blog posts we had done over the course of the year. We had individual pcaps of each chunk being hidden somewhere, and separate captures of traffic showing someone browsing to each of the blog posts as hints.

Finally, we wanted individual pieces of fun bits of “noise”, like browsing random anti-holiday blogs, or RTP streams of holiday music, or fake emails between previous “Grinch” actors.

All in all, it took around seventy-five individual pcaps to deliver our Holiday challenge. Here’s how we made them.

Infrastructure

We used a few separate systems to generate the traffic for these capture files. When capturing network traces, it’s important to not rely on captures from any of the systems that are terminating packets in the network - you might get different results than if you are sitting in-line. We use the Profishark network TAP, a high-speed portable TAP that’s super easy to use and partnered with us to upload captures to a CS Enterprise system.

Profitshark Network Tap

TAP vs. SPAN?

The terms TAP vs. SPAN refer to the kind of interface one uses when gathering network captures. Building a SPAN port on an existing switch, or capturing directly from one of the systems in the network is possible, but will often lead to dropped packets. Using a network TAP is a much better way to get the data you need. Profitap has a good article on the differences.

TAP vs. SPAN?

The terms TAP vs. SPAN refer to the kind of interface one uses when gathering network captures. Building a SPAN port on an existing switch, or capturing directly from one of the systems in the network is possible, but will often lead to dropped packets. Using a network TAP is a much better way to get the data you need. Profitap has a good article on the differences.

To build the captures, we used the Profishark 1G TAP to capture all of the packets sent by one of the machines in the system. The Profishark captures all of the data, but we only wanted to look at a single stream - the one we knew was carrying our manufactured data for the challenge. After the capture was uploaded to CloudShark, we logged into our CS Enterprise system, found the capture waiting for us, and applied a display filter to show us only the stream we were looking for, in this case, a series of these:

ip.addr == <source ip> && ip.addr == <dest ip> && tcp.port == <application>

This gives us a view of what we want, so we used CloudShark’s capture export feature to split off just this data into a new capture. We did this for each of the captures we needed for each piece of the puzzle, and worried about putting them together later.

Taking a bottom-up approach

The challenge PCAPs were created from the bottom up. We started with a single capture that contained the file transfer of our holiday card JPEG file. We took this TCP stream, and split it into different chunks. Since it’s encrypted, we hid the SSL Keylog to decrypt the stream in another PCAP, which also contained a confession from the Grinch. The whole thing was then uploaded to CloudShark for you to find.

Chunk by chunk

Photo Chunks

Instead of using a real server, we generated a self-signed certificate for photos.cloudshark.org and simulated the site by running ‘openssl s_server‘ to listen for and accept a TLS connection on port 443:

openssl s_server -accept 443 -cert photos.cloudshark.org/server.crt -key photos.cloudshark.org/server.key

On a second machine, we added an entry to /etc/hosts so photos.cloudshark.org would point to our simulated HTTP server. Because we also needed to be able to decrypt the stream later on, we used a version of curl that we compiled to enable the SSL keylog feature.

Finally, we uploaded the photo and saved the SSL keylog file to use later on to decrypt the stream. Since we used a self-signed certificate we also had to add the ‘-k’ or –insecure’ flag to curl so that it would ignore the certificate:

SSLKEYLOGFILE=~/git/challenge-tools/keylog ./curl -k --upload-file ~/git/challenge-tools/happy-holidays-2018.jpg --header "Content-Type: image/jpeg" https://photos.cloudshark.org/

We captured the traffic of this upload, exported this to a new session and used editcap to break the capture file into “chunks”:

editcap holiday_picture_upload_full.pcapng holiday_picture_chunk_<#>.pcapng <packets>

Here is how the packets from the ‘holiday_picture_upload_full.pcapng’ file can be selected to be saved in the chunk (from the editcap man page):

An optional list of packet numbers can be specified on the command tail; individual packet numbers separated by whitespace and/or ranges of packet numbers can be specified as start-end, referring to all packets from start to end. By default the selected packets with those numbers will not be written to the capture file. If the -r flag is specified, the whole packet selection is reversed; in that case only the selected packets will be written to the capture file.

To make the chunks, we selected the first and last three packets from the full holiday picture upload to store in chunk 0 and split the rest of the packets as evenly as we could across 4 additional pcaps.

Armed with chuck 0 as part of the initial challenge material, you’ll discover that it was the beginning and end of a TCP stream. Knowing those sequence numbers, you can figure out when you had found all of the chunks we had hidden.

Confession/SSL Keylog

Next we created a pcap to hide the SSL keylog we saved when uploading the file so that this could be used to decrypt the upload and view the reassembled chunks once they were all found. We created a confession message from the Grinch and used ROT 47 to encode it (after all, we didn’t want to make it too obvious):

They came despite memcache! They came despite seeding!

They came despite all of my attacks succeeding!

I tried to stop packets and this holiday fun

But I hope you’ll use CloudShark before the damage is done

To hide the SSL keylog within this message, we created a python script using Scapy. If you’ve never used Scapy before, it’s a great python library that uses libpcap for a number of packet generation and packet decoding functions.

The keylog is hidden by sending the first line of the confession in the initial TCP SYN packet of the 3-way handshake. According to the TCP specification, data isn’t sent in this packet, but with Scapy we can do it anyway.

Next, we sent a packet containing the SSL keylog, but sent it with a sequence number 1 byte higher than it should have been. This means there was 1 byte that is missing from this stream based on the packets sent. Finally, we sent the rest of the message with a sequence number 1 less than the packet containing the SSL keylog essentially overwriting the packet containing the keylog. Try running Follow Stream on the following TCP stream and notice that the keylog is not displayed:

https://www.cloudshark.org/captures/8b77a8be8971

Here is the script we wrote using Scapy to accomplish this.

The Grinch’s confession

Merging and splitting captures

We used some command line tools when building these captures to "split" and "merge" captures. These features are also easily done in CloudShark using the merge and export tools, right in your browser, working directly with your CloudShark repository.

Merging and splitting captures

We used some command line tools when building these captures to "split" and "merge" captures. These features are also easily done in CloudShark using the merge and export tools, right in your browser, working directly with your CloudShark repository.

To get to the confession pcap, and to help people know they were on the right track, we added a single UDP packet to the beginning of each pcap using ‘mergecap’. Here is an example of one of the UDP packets:

https://www.cloudshark.org/captures/b5851b0eed1e

The HAPPYHOLIDAYS text matches what we saved in holiday_chunk_0.pcap so we hoped this would help as a hint. Once all of the chunks are assembled there is also a secret message hidden across all of the UDP packets:

https://www.cloudshark.org/captures/3e16813f3701

This hidden link leads to the confession PCAP we created above.

Hiding the chunks

We hid each chunk, with it’s UDP “flag”, within different protocols or on different servers for people to find. We stored one in memcache, one in a BitTorrent stream, one on a webserver with questionable security, and finally linked as a reference in a Suricata IDS rule!

Memcached

The memcached protocol is a generic mechanism intended to help speed up web applications by reducing the number of requests made to a database. It does this by serving as a cache for data returned by the database or other API calls.

There are some ways to take advantage of memcached for DDoS attacks, which is why we featured it in this article.

This chunk was encoded with base64 using the following website:

https://www.base64encode.org/

Then we used the following bash command to upload the chunk to a memcached server and captured this traffic:

echo -e "set file 0 60 \
$(ls -l ${file} | cut -d ' ' -f 5)\r\n \
$(cat ${file})\r" | nc 172.16.0.22 11211

The set memcache command requires a name (file), set of flags (0), expiration (60) and the size of the file (ls -l ${file} | cut -d ' ' -f 5). We then saved this capture file:

https://www.cloudshark.org/captures/b0eb2b9a0d11

BitTorrent

For this chunk we wanted to share this file using BitTorrent, which was featured in one of our analyses of a malware-traffic-analysis.net challenge.

To do this, we used a machine running CentOS 7 and installed the opentracker-ipv4 package from the epel repository. Then we created a torrent for this file using uTorrent and used our new server as the tracker. Port 6969 needed to be allowed for the BitTorrent tracker software and the sha1 hash of the pcap file was added to a whitelist configured in /var/opentracker/whitelist so that only this file could be served by our BitTorrent tracker.

From there we started a few BitTorrent clients to seed the capture file and saved the BitTorrent file as hp-c1.torrent. Then the traffic was captured while uploading this torrent file over HTTP so that the file could be downloaded using the HTTP objects analysis tool.

Web Server

To hide this chunk we deployed a web server and served a fake “G.E.S.” (Grinch Exfiltration Service) application. Viewing the source for this application revealed a hidden URL:

// Don't forget to remove this line
// since we don't want anybody to find the real download!
window.location = "./stolen_files/7f5505174c8cb8dcbe513a51f2c70c9e.pcap";

That chunk was hosted and could be downloaded by going directly to the url.

VPN Leak

We needed a way to point people to the web server we deployed above. To do this we configured a host with an VPN incorrectly setup to leak DNS data. Then we used a browser to visit the “G.E.S.” application and captured all of the traffic. Decrypting and viewing the traffic sent browsing to our application isn’t possible since it is protected by the VPN. Before the browser can open our web application though it must perform a DNS query to find the IP address of the web server. Our “leaky” VPN performs this query in the clear revealing the host name of the web server where another hidden chunk could be found. Another hint was given by showing the Grinch browsing to our blog post about misconfigured VPNs leaking sensitive information.

Threat Assessment

This chunk we hosted on CS Personal and made this public. To lead people to this chunk we started by sending an ascii art image of the Grinch:

https://www.cloudshark.org/captures/7bee98cd8520

Then we created a custom Suricata rule that we used with CS Threat Assessment to alert on this ascii art:

alert tcp 172.16.1.130 35450 -> 172.16.0.22 3 (msg: "NO GRINCH XMAS Xfiltration"; content:"|20 20 20 20 5f 20 2e 2f 20 20 20 20 20 2c 2e 27|"; sid:99999; rev:1; reference:url,www.cloudshark.org/captures/18645097ca4e; classtype:grinch-activity;)

This triggers on the pattern that is found in the Grinch ascii art:

    _ ./     ,.'

Some rules in CS Threat Assessment contain a reference with a link to more information about the rule or what is being alerted on. For our holiday capture challenge we used this to link to the final chunk of the photo.

Music

We also created a series of RTP streams as an easter egg for people to listen to while searching for our holiday photo. These were created with gstreamer and we captured the traffic sent:

gst-launch-1.0 filesrc location="<file>" ! wavparse ! audioconvert
! audioresample ! alawenc !  rtppcmapay  ! udpsink host=<ip address> port=<port>

Here is an example of one of the RTP streams:

https://www.cloudshark.org/captures/ba75250e018a

Some of these required configuring a Decode As rule to decode the UDP stream as RTP so that it could be played back using the ‘RTP Streams’ analysis tool:

https://www.cloudshark.org/analysis/ba75250e018a/rtp_streams

Hints

To help people find these chunks we created traffic to serve as hints.

We’ll see about them spreading their holiday cheer!

I’ll break up their well wishes, their attempts to endear!

These digital images are easy to split,

I’ll spread pieces around and see where they fit!

They’ve done all this work on recent network hacks;

We’ll see if they can follow my nefarious tracks!

They’ll never remember, where this image is stashed,

in an old protocol where short messages are cached.

Their peer-obsessed world, to me it seems foreign!

I’ll plant seeds for the next piece - their joy is abhorrent!

They are onto us Mads, their spies are around.

We’ll have to make sure this VPN is configured and sound!

Mads responded with :klajno;hgaouhp0'jqv3-0[j2.4jk24fg90hj. (likely due to a lack of thumbs):

https://www.cloudshark.org/analysis/1539760d8221/follow?stream=15&proto=tcp

  • To help point people to our RTP stream easter eggs we simulated sending e-mails and used ‘smtp-sink’ to simulate an e-mail server:
 smtp-sink -h email.qaggle.net -u nobody 172.16.0.22:25 1000

We then wrote text files containing the body of the e-mail and sent them using curl.

curl --url 'smtp://email.qaggle.net' --mail-from <from address> \
  --mail-rcpt '<to address>' --upload-file <text file>

This created realistic looking traffic simulating an e-mail:

https://www.cloudshark.org/analysis/70754693809f/follow?stream=0&proto=tcp

Noise

With so many hints we also added some “noise” to sift through. Using the same method used to create the hints by browsing our blog we captured traffic including a Bing search for “How to ruin Christmas” and a few other blog posts we found that we thought the Grinch would like including a post titled “10 Reasons I Hate Christmas”:

https://www.cloudshark.org/captures/575ed19ccd3c

Putting it all together

This left us with 28 separate captures we needed to merge together to create a capture of the Grinch’s network traffic take while he stole our photo.

Simply merging these captures wouldn’t work. When merging, the timestamps of the packets are considered and every packet from every file is placed in order based on the timestamp of each individual packet. It took us multiple days to generate these captures and the result of merging would have looked something like:

  1. All of the GRC traffic
  2. Storing the chunk in memcached
  3. Ascii art we wrote a rule for
  4. VPN traffic from the Grinch
  5. Upload of BitTorrent file
  6. All of the browser traffic hints and noise
  7. All of the e-mail hints
  8. All of the RTP streams

That would put everything in order, and make the challenge too easy! Thus began the “great holiday merge of 2018”. It started with a hand drawn diagram of what we wanted the capture to ultimately look like. This helped us get a feel for what parts we needed in the capture and a general order of where to put the parts.

Next Zach wrote this script to configure a zero time for the capture (12/24/2018 at midnight), and for each capture, we added a skew time (number of seconds) for when the capture should start based off of the zero time. This let us start with one of the RTP streams, one of the e-mail hints 1 second later, then one of the browser hints 12 seconds later, and so on until each capture had it’s own start time X seconds after 12/24/2018 at midnight.

Finally, all of these captures were uploaded to CloudShark and merged together to complete creating the challenge!

This collection contains all of the capture files that we used to create this capture challenge.

Final thoughts

Overall, it sounds like a lot of work, but we love making challenges like this and seeing our customers and the great packet capture community use their collective skills to solve problems that, while fabricated, exercise the thinking needed to approach series network or security problems. We look forward to bringing you the next one!

Get articles like this in your inbox: