* TIME: 2021-08-22 22:55:58 * URL: ----- Skip to content [ ](https://github.com/) [ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser- name%3E%2F%3Crepo-name%3E&source=header-repo) * Why GitHub? [Features →](/features) * [Mobile →](/mobile) * [Actions →](/features/actions) * [Codespaces →](/features/codespaces) * [Packages →](/features/packages) * [Security →](/features/security) * [Code review →](/features/code-review/) * [Issues →](/features/issues/) * [Integrations →](/features/integrations) * [GitHub Sponsors →](/sponsors) * [Customer stories→](/customer-stories) * [Team](/team) * [Enterprise](/enterprise) * Explore * [Explore GitHub →](/explore) #### Learn and contribute * [Topics →](/topics) * [Collections →](/collections) * [Trending →](/trending) * [Learning Lab →](https://lab.github.com/) * [Open source guides →](https://opensource.guide) #### Connect with others * [The ReadME Project →](https://github.com/readme) * [Events →](https://github.com/events) * [Community forum →](https://github.community) * [GitHub Education →](https://education.github.com) * [GitHub Stars program →](https://stars.github.com) * [Marketplace](/marketplace) * Pricing [Plans →](/pricing) * [Compare plans →](/pricing#feature-comparison) * [Contact Sales →](https://enterprise.github.com/contact) * [Education →](https://education.github.com) * [ ![]() In this repository All GitHub ↵ Jump to ↵ ]() * No suggested jump to results * [ ![]() In this repository All GitHub ↵ Jump to ↵ ]() * [ ![]() In this user All GitHub ↵ Jump to ↵ ]() * [ ![]() In this repository All GitHub ↵ Jump to ↵ ]() [ Sign in ](/login?return_to=https%3A%2F%2Fgithub.com%2FZSShen%2FMeltingPot) [ Sign up ](/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser- name%3E%2F%3Crepo-name%3E&source=header-repo&source_repo=ZSShen%2FMeltingPot) # [ZSShen](/ZSShen) / **[MeltingPot](/ZSShen/MeltingPot) ** * [ Notifications ](/login?return_to=%2FZSShen%2FMeltingPot) * [ Star ](/login?return_to=%2FZSShen%2FMeltingPot) [ 22 ](/ZSShen/MeltingPot/stargazers) * [ Fork ](/login?return_to=%2FZSShen%2FMeltingPot) [ 6 ](/ZSShen/MeltingPot/network/members) A tool to cluster similar executables (PEs, DEXs, and etc), extract common signature, and generate Yara patterns for malware detection. [ MIT License ](/ZSShen/MeltingPot/blob/master/COPYING) [ 22 stars ](/ZSShen/MeltingPot/stargazers) [ 6 forks ](/ZSShen/MeltingPot/network/members) [ Star ](/login?return_to=%2FZSShen%2FMeltingPot) [ Notifications ](/login?return_to=%2FZSShen%2FMeltingPot) * [ Code ](/ZSShen/MeltingPot) * [ Issues 1 ](/ZSShen/MeltingPot/issues) * [ Pull requests 0 ](/ZSShen/MeltingPot/pulls) * [ Actions ](/ZSShen/MeltingPot/actions) * [ Projects 0 ](/ZSShen/MeltingPot/projects) * [ Wiki ](/ZSShen/MeltingPot/wiki) * [ Security ](/ZSShen/MeltingPot/security) * [ Insights ](/ZSShen/MeltingPot/pulse) More * [ Code ](/ZSShen/MeltingPot) * [ Issues ](/ZSShen/MeltingPot/issues) * [ Pull requests ](/ZSShen/MeltingPot/pulls) * [ Actions ](/ZSShen/MeltingPot/actions) * [ Projects ](/ZSShen/MeltingPot/projects) * [ Wiki ](/ZSShen/MeltingPot/wiki) * [ Security ](/ZSShen/MeltingPot/security) * [ Insights ](/ZSShen/MeltingPot/pulse) master Switch branches/tags Branches Tags Could not load branches Nothing to show [ default ](https://github.com/ZSShen/MeltingPot/tree/) [View all branches](/ZSShen/MeltingPot/branches) Could not load tags Nothing to show [ default ](https://github.com/ZSShen/MeltingPot/tree/) [View all tags](/ZSShen/MeltingPot/tags) [ **1** branch ](/ZSShen/MeltingPot/branches) [ **0** tags ](/ZSShen/MeltingPot/tags) [ Go to file ](/ZSShen/MeltingPot/find/master) Code * [ ](https://docs.github.com/articles/which-remote-url-should-i-use) Clone HTTPS GitHub CLI Use Git or checkout with SVN using the web URL. Work fast with our official CLI. [Learn more](https://cli.github.com). * [ Open with GitHub Desktop ](https://desktop.github.com) * [ Download ZIP ](/ZSShen/MeltingPot/archive/refs/heads/master.zip) #### Launching GitHub Desktop If nothing happens, [download GitHub Desktop](https://desktop.github.com/) and try again. Go back #### Launching GitHub Desktop If nothing happens, [download GitHub Desktop](https://desktop.github.com/) and try again. Go back #### Launching Xcode If nothing happens, [download Xcode](https://developer.apple.com/xcode/) and try again. Go back #### Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. ## Latest commit Andy Shen [Fix: Silence the subtle bug of Travis CI.](/ZSShen/MeltingPot/commit/a15d83536a74de8c17c836888bfb354a024793d6 "Fix: Silence the subtle bug of Travis CI.") … Loading status checks… [ a15d835 ](/ZSShen/MeltingPot/commit/a15d83536a74de8c17c836888bfb354a024793d6) [ on 5 Jan 2019 ](/ZSShen/MeltingPot/commit/a15d83536a74de8c17c836888bfb354a024793d6) [Fix: Silence the subtle bug of Travis CI.](/ZSShen/MeltingPot/commit/a15d83536a74de8c17c836888bfb354a024793d6) `a15d835` ## Git stats * [ **208** commits ](/ZSShen/MeltingPot/commits/master) ## Files [Permalink](/ZSShen/MeltingPot/tree/a15d83536a74de8c17c836888bfb354a024793d6) Failed to load latest commit information. Type Name Latest commit message Commit time [engine](/ZSShen/MeltingPot/tree/master/engine "engine") [Add: Patch the license comment in source files.](/ZSShen/MeltingPot/commit/efff0b7c01b508f4c474cfb58db174d75dd465fe "Add: Patch the license comment in source files.") 4 years ago [golden](/ZSShen/MeltingPot/tree/master/golden "golden") [AutoRun: Add the new rule for automated testing.](/ZSShen/MeltingPot/commit/dbc518940a101f87024f5dcab34bb4aa8b3b0a28 "AutoRun: Add the new rule for automated testing.") 7 years ago [plugin](/ZSShen/MeltingPot/tree/master/plugin "plugin") [Add: Patch the license comment in source files.](/ZSShen/MeltingPot/commit/efff0b7c01b508f4c474cfb58db174d75dd465fe "Add: Patch the license comment in source files.") 4 years ago [res](/ZSShen/MeltingPot/tree/master/res "res") [Update the new picture for example pattern.](/ZSShen/MeltingPot/commit/e7b7402ff042121fd178f8ae9474923020f2db21 "Update the new picture for example pattern.") 5 years ago [util](/ZSShen/MeltingPot/tree/master/util "util") [Add: Patch the license comment in source files.](/ZSShen/MeltingPot/commit/efff0b7c01b508f4c474cfb58db174d75dd465fe "Add: Patch the license comment in source files.") 4 years ago [.travis.yml](/ZSShen/MeltingPot/blob/master/.travis.yml ".travis.yml") [Slice the error produced by --use-mirror option.](/ZSShen/MeltingPot/commit/ba77e1eaeb854483695178c27e3234eeaf0e3c1f "Slice the error produced by --use-mirror option.") 5 years ago [CMakeLists.txt](/ZSShen/MeltingPot/blob/master/CMakeLists.txt "CMakeLists.txt") [Refine: Add the hinting environment variables to guide valgrind for G…](/ZSShen/MeltingPot/commit/0b408ea135fbb402dca921e006793193910f5514 "Refine: Add the hinting environment variables to guide valgrind for GLIB.") 7 years ago [COPYING](/ZSShen/MeltingPot/blob/master/COPYING "COPYING") [Fix: Silence the subtle bug of Travis CI.](/ZSShen/MeltingPot/commit/a15d83536a74de8c17c836888bfb354a024793d6 "Fix: Silence the subtle bug of Travis CI.") 3 years ago [README.md](/ZSShen/MeltingPot/blob/master/README.md "README.md") [Modify: Update project introduction.](/ZSShen/MeltingPot/commit/f6107d92bdd0a79e2b124cda9c6c438a2256a96d "Modify: Update project introduction.") 3 years ago [clean.py](/ZSShen/MeltingPot/blob/master/clean.py "clean.py") [Try to fix the CI error.](/ZSShen/MeltingPot/commit/9a4cfb0d7b5789a99d9c680116687638c1b08c07 "Try to fix the CI error.") 5 years ago View code MeltingPot Introduction Installation Basic Advanced Usage Contact ## README.md [![Build Status](https://camo.githubusercontent.com/280072181dac5b59331ef56d3af8f01f9061ce9776a61fcf07a706667865b446/68747470733a2f2f7472617669732d63692e6f72672f5a535368656e2f4d656c74696e67506f742e7376673f6272616e63683d6d6173746572)](https://travis- ci.org/ZSShen/MeltingPot) # **MeltingPot** [![](https://raw.githubusercontent.com/ZSShen/BinaryCluster- YaraGenerator/master/res/picture/Engine Intro.png)](https://raw.githubusercontent.com/ZSShen/BinaryCluster- YaraGenerator/master/res/picture/Engine Intro.png) --- MeltingPot is an _**automated common binary signature extractor and pattern generator**_. For the given sample set with the _**same file format**_ , it slices each file into small pieces of binary sequences and correlates the files sharing the similar sequences. To show the result, MeltingPot generates a set of _**[YARA](http://plusvic.github.io/yara/) formatted patterns**_ each of which represents the common signature of a certain file cluster. Such patterns can be directly applied by YARA scan engine. ## **Introduction** **MeltingPot is composed of the main engine and the supporting plugins. The relation is briefly illustrated here:** * The engine first loads the user specified configuration. * It applies the **`file slicing plugin`** to slice the input files. * It correlates the slices by examining their similarity with the help of **`similarity comparison plugin`**. * Now the engine acquires the file slice clusters. It then extracts the common binary signature from each cluster. * Finally, the engine outputs the signatures with **`pattern formation plugin`**. [![](https://raw.githubusercontent.com/ZSShen/BinaryCluster- YaraGenerator/master/res/picture/Pattern.png)](https://raw.githubusercontent.com/ZSShen/BinaryCluster- YaraGenerator/master/res/picture/Pattern.png) --- The example pattern for the Windows Notepad and its packed version using several kinds of software protectors. **As mentioned above, we have three kinds of plugins:** * **`File slicing`** \- Slicing an input file by format parsing. (E.g. Windows PE, Android DEX) * **`Similarity comparison`** \- Measuring the similarity for a pair of slices. (E.g. ssdeep, ngram) * **`Pattern formation`** \- Producing YARA pattern. If analysts intend for custom research purpose, they can craft the custom plugins in the plugin source directory and use the MeltingPot build script to create the libraries. ## **Installation** #### **Basic** First of all, we need to prepare the following utilities: * [CMake](http://www.cmake.org/) \- A cross platform build system. * [Valgrind](http://valgrind.org/) \- An instrumentation framework help for memory debug. * [SSDeep](http://ssdeep.sourceforge.net/) \- A fuzzy hash generation and comparison library. * [GLib](https://developer.gnome.org/glib/) \- A large set of libraries to handle common data structures. * [libconfig](http://www.hyperrealm.com/libconfig/) \- A library to process structured configuration file. For Ubuntu 12.04 and above, it should be easy: $ sudo apt-get install -qq cmake $ sudo apt-get install -qq valgrind $ sudo apt-get install -qq libfuzzy-dev $ sudo apt-get install -qq libglib2.0-dev $ sudo apt-get install -qq libconfig-dev Now we can build the entire source tree under the project root folder: $ ./clean.py --rebuild $ cd build $ cmake .. $ make Then the engine should be under: * `./engine/bin/release/cluster` And the relevant plugins should be under: * `./plugin/slice/lib/release/libslc_*.so` * `./plugin/similarity/lib/release/libsim_*.so` * `./plugin/format/lib/release/libfmt_*.so` #### **Advanced** If we modify the main engine or the plugins, we can move to the corresponding subdirectory to rebuild the binary. To build the engine independently: $ cd engine $ ./clean.py --rebuild $ cd build $ cmake .. -DCMAKE_BUILD_TYPE=Debug|Release $ make Note that we have two build types. For debug build, the compiler debug flags are turned on, and the binary locates at `./engine/bin/debug/cluster`. For optimized build, the binary locates at `./engine/bin/release/cluster`. To build the plugin independently (using File Slicing plugin as example): $ cd plugin/slice $ ./clean.py --rebuild $ cd build $ cmake .. --DCMAKE_BUILD_TYPE=Debug|Release $ make Again, we must specify the build type for compiliation. For debug build, the binary locates at `./plugin/slice/debug/libslc_*.so`. For optimized build, the binary locates at `./plugin/slice/release/libslc_*.so`. For the other two kinds of plugins, the build rule is the same. ## **Usage** To run the engine, we should specify some relevant configurations. The example is shown in `./engine/cluster.conf`. We discuss these parameters below: Parameter | Description ---|--- `SIZE_SLICE` | The size of the sliced file binary `SIZE_HEX_BLOCK` | The length of the signature extracted from a slice cluster `COUNT_HEX_BLOCK` | The number of to be extracted signatures from a cluster `THRESHOLD_SIMILARITY` | The threshold to group similar slices `RATIO_NOISE` | The ratio of dummy bytes (00 or ff) in a signature `RATIO_WILDCARD` | The ratio of wildcard characters in a signature `TRUNCATE_GROUP_SIZE_LESS_THAN` | The threshold to ignore trivial clusters `FLAG_COMMENT` | The knob for pattern comments `PATH_ROOT_INPUT` | The pathname of input sample set `PATH_ROOT_OUTPUT` | The pathname of output pattern folder `PATH_PLUGIN_SLICE` | The pathname of the file slicing plugin `PATH_PLUGIN_SIMILARITY` | The pathname of similarity comparison plugin `PATH_PLUGIN_FORMAT` | The pathname of pattern formation plugin In addition, we have the following advanced parameters: Parameter | Description ---|--- `COUNT_THREAD` | The number of running threads `IO_BANDWIDTH` | The maximum number of files a thread can simultaneously open With the configuration file prepared, we can launch the MeltPot engine: * For normal task, run: ./engine/bin/release/cluster --conf ./engine/cluster.conf * For memory debug, use debug build and run: valgrind ./engine/bin/debug/cluster --conf ./engine/cluster.conf Note that if we apply valgrind for memory debugging, valgrind will produce a "still-reachable" alert in the summary report. This is due to the side effect produced by GLib. MeltingPot should be memory safe :-). ## **Contact** Please contact me via the mail _**[andy.zsshen@gmail.com](mailto:andy.zsshen@gmail.com)**_. ## About A tool to cluster similar executables (PEs, DEXs, and etc), extract common signature, and generate Yara patterns for malware detection. ### Topics [ c ](/topics/c "Topic: c") [ malware-research ](/topics/malware-research "Topic: malware-research") [ malware-samples ](/topics/malware-samples "Topic: malware-samples") [ yara ](/topics/yara "Topic: yara") [ ssdeep ](/topics/ssdeep "Topic: ssdeep") [ malware-detection ](/topics/malware- detection "Topic: malware-detection") ### Resources Readme ### License [ MIT License ](/ZSShen/MeltingPot/blob/master/COPYING) ## [ Releases ](/ZSShen/MeltingPot/releases) No releases published ## [ Packages 0 ](/users/ZSShen/packages?repo_name=MeltingPot) No packages published ## Languages * [ C 80.2% ](/ZSShen/MeltingPot/search?l=c) * [ Python 9.6% ](/ZSShen/MeltingPot/search?l=python) * [ CMake 8.2% ](/ZSShen/MeltingPot/search?l=cmake) * [ C++ 2.0% ](/ZSShen/MeltingPot/search?l=c%2B%2B) * © 2021 GitHub, Inc. * [Terms](https://docs.github.com/en/github/site-policy/github-terms-of-service) * [Privacy](https://docs.github.com/en/github/site-policy/github-privacy-statement) * [Security](https://github.com/security) * [Status](https://www.githubstatus.com/) * [Docs](https://docs.github.com) [ ](https://github.com "GitHub") * [Contact GitHub](https://support.github.com) * [Pricing](https://github.com/pricing) * [API](https://docs.github.com) * [Training](https://services.github.com) * [Blog](https://github.blog) * [About](https://github.com/about) You can’t perform that action at this time. You signed in with another tab or window. [Reload]() to refresh your session. You signed out in another tab or window. [Reload]() to refresh your session.