⬛️ Console Activity Analysis

Remote vs local commands?

Windows

CMD

PowerShell

Windows PowerShell Event logs Operational Monitoring.

%SystemRoot%\System32\winevt\Logs\Microsoft-Windows-Powershell%Operational.evtx, ID 4103, 4104 (if Windows considers suspicious).

%SystemRoot%\System32\winevt\Logs\Microsoft-Windows-Powershell.evtx. ID 400, 800.

%userprofile%\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadline\ConsoleHost - poweshell history.

Powershell has short commands. For example, Get-Command cmdlet can be shortened to gcm. Used along with wildcards, one can obfuscate a script a little: gcm ‘ie-e’ gets the list of commands satisfying this expression, which is only one: Invoke-Expression.

WSL

macOS

By default, all console data is not being logged. Only staff that was typed with sudo is. Below is the screenshot of the Console app (streaming on) when sudo ls command is typed in the Terminal.

And here is what’s being logged when one types sudo su and then ls.

What about using ls or cat without elevating privileges? Nothing is logged at all (trying helloworld, cat metadata.db or visudo). So, how one keeps logging their Terminal history? It would be also great to add PID of the process, effective and real usernames. Here is how I’ve managed to do it on my local machine (to further integrate the logs with Elastic).

There were several problems that I’ve encountered given it’s a macOS. First of all, macOS default interpreter is zsh, which doesn’t have PROMT_COMMAND variable. This variable in bash allows executing some script after each command. I’ve found a soulution in the internet to overcome this pecularity: add bash precmd() { eval "$PROMPT_COMMAND" } at the beginning of the ~/.zshrc file, then add export PROMPT_COMMAND="" right after it. This would simulate the desired behaviour. Don’t forget to source ~/.zshrc or relaunching the Terminal for the changes to be applied.

Now, we can log most of the commands typed here. However, in this case commands that were typed after sudo su won’t be logged. That happens because when using sudo su, most of environment variables are not loaded for safety reasons. However, if you type sudo su - or sudo -l instead, you’ll get the data logged. Shoking!

No matter what interpreter is in use, when you type sudo su, it’s not loading your env variables by default. To overcome this, add sudo visudo. Defaults env_keep += "PROMPT_COMMAND".

Commands can be retrieved by history command on UNIX machines. Depending on the user, the history will return different values. We need both the main user’s and the root’s history.

❓Does it retireve info from zsh_history/bash_history or from elsewhere?

zsh_history/bash_history contain UNIX timestamp as well. history returns only command id and the command itself. Both list the same commands (check the first two and the last two lines).

🔎 Since each command has its own line, I’ve decided to compare these two files. cat ~/.zsh_history | wc -l shows 5503 in my case and history | wc -l - 5325. It’s pretty bad difference. That means they are not correlated too well. Why is that? I’ve redirected the output to a csv file and imported into Excel (for both history and zsh_history). The first thing that caught the eye was the amount of blank lines for the latter. The second - some commands were split into several lines for zsh_history. While the blank lines could be eliminated with cat ~/.zsh_history| sed '/^$/d' | wc -l, the second problem is trickier. But do I need to cope with it? What do I need from this file? Timestamp. So, it’s easier to run and log date before or after each command instead. I will only be referring to the history output then.

Now, to the log file itself. Since I will be integrating it to Elastic (Custom Logging), I’d better make it a JSON (ndjson) format. Below is the beginning of my ~/.zshrc file (other contents is irrelevant for this matter).

{
  "timestamp": $(date), 
  "real_username": "$(id -p | head -n1 | cut -f2)",
  "effective_username": "$(whoami)",
  "pid": $$,
  "commandid": "$(history | tail -n 1 | cut -d" " -f2),
  "command": '$(history | tail -n 1 | xargs | cut -d" " -f2-)'
}

Now, to the effective vs real user. You can read more here. In short, when ordinary user issues commands, both are the same, but whenever the privileges are elevated (for example, with sudo su) or a command is issued from behalf of another user (like sudo -l username) these two are going to be different. This allows us to log the responsible user. Effective user is the one, who the system thinks it is. Someone, whose rights and privileges (if any) the currently logged user is using. Real user is pretty self-explanatory, the logged in user, the one who’s been doing staff. The next problem to solve, is to

There are different shell interpreters. In order to find the appropriate history file, you’ll need to know which one is used on the machine. Run env command and look at the following variables:

SHELL=/bin/zsh
ZSH=/Users/username/.oh-my-zsh
PYENV_SHELL=zsh

Bash History

/Users/%username%/.bash_sessions
/Users/%username%/.bash_history

zsh

Linux

Shell

Bash

/home/%username%/.bash_history

On most Linux machines bash is the default shell interpereter. Each user has a separate bash history. However, if the user has escalated to root, everything goes to the root’s history file (~/.bash_history). ❗️ Not only root’s history is interesting!

📝 Set history time env variable to record timestamps for the q file.

It’s not always easy to determine if this activity is a legitimate admin activity or if it’s something adversary (given that they have managed to escalate).

⚠️ Bash hostory can be tampered with or turned of.

By default, the bash_history is stored in memory when you’re running commands on a system. This means that nothing is actually written to disk until you run the “exit” command and exit the shell. [1]

kill -9 $$ to kill the bash shell. This way whatever was in memory for bash_history won’t be written to disk.

If you simply delete bash_history one can recover it using photorec or other carving utility, for example (unless the sectors for this files were overwritten).

Lateral movement

ssh, netcat and telnet are frequently used for lateral movement. ssh and telnet will give a target IP as well. Agregate and make a frequency analysis. Note all the IPs that the system has connected to.

💡 I would also check with the baseline (which machines this machine usually connects to).

Malicious modifications

vim, nano. Which files were changed? something in /etc/. Look for archiving activity: tar, 7zip etc. What’s archived and why?

Recon

whoami, cat, ifconfig etc. Amount of these commands. Short span. First is usually to run whoami, then might be something else.