Microsoft Malware Detection

Problem Statement

In the past few years, the malware industry has grown very rapidly that, the syndicates invest heavily in technologies to evade traditional protection, forcing the anti-malware groups/communities to build more robust softwares to detect and terminate these attacks. The major part of protecting a computer system from a malware attack is to identify whether a given piece of file/software is a malware.

Source/Useful Links

Microsoft has been very active in building anti-malware products over the years and it runs it’s anti- malware utilities over 150 million computers around the world. This generates tens of millions of daily data points to be analyzed as potential malware. In order to be effective in analyzing and classifying such large amounts of data, we need to be able to group them into groups and identify their respective families. This dataset provided by Microsoft contains about 9 classes of malware. Source of the dataset : https://www.kaggle.com/c/malware-classification

Data Overview

Every malware have two files 1.Bytes file 2.Asm file. Total train dataset consist of 200GB data out of which 50GB of data is .bytes files and 150GB of data is .asm files. Lots of Data for a single-box/computer. There are total 10,868 .bytes files and 10,868 asm files total 21,736 files. There are 9 types of malwares (9 classes) in our give data Types of Malware:

Ramnit
Lollipop
Kelihos_ver3
Vundo
Simda
Tracur
Kelihos_ver1
Obfuscator.ACY
Gatak

Splitting of data

I have used the training data only and i divided it randomly in to three parts train,cv,test in 64%,16%,20% ratio.

Contents

Here i will brefily describe what i have done in the notebook.

Step-1

Creating uni gram and bi gram from the bytes files and done some feature engineering.(Like length of the bytes files)

Step-2

Extracting important features from the .asm files and done some feature engineering on asm files.

Step-3

Applying KNN,logisitc regression,Random Forest,XGBOOST model on uni gram and bigram features of bytes files only.Found the best logloss as 0.07 on test data.

Step-4

Applying KNN,logisitc regression,Random Forest,XGBOOST model on asm files only.Found the best logloss as 0.04 on test data.

Step-5

Then merging the both uni gram,bi gram and asm files. and applying logisitc regression,Random Forest,XGBOOST model on top of it.Found the best losloss as 0.03 in test data.

Step-6

Genreated a image features from the asm files.(Ref:Idea after viewing the approach of first place solution in youtube)

step-7

Then combined all the feautures and again applied all the above algorithms but logloss increased.Tried Trauncated SVD to lower the dimension but not improvement on logloss.

step-8

With all the features i applied RF on top of it.Get only the important features.And with that important features only i applied XGBoost on top of it getting log loss as 0.005 on test data.

Credits

KaustuvDash/MicrosoftMalwareDetection

ErrorLooks like something went wrong!

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Error
Looks like something went wrong!

Packages