This document is devoted to AWS logs. Several examples are analysed and useful fields are marked.
CloudTrail
This is an example log from a AWS CTF challenge. I am going to use this one in order to get an idea of what is being logged and how this can be useful for investigations.
{
"Records": [
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AWSService",
"invokedBy": "ecs-tasks.amazonaws.com"
},
"eventTime": "2018-11-28T22:31:59Z",
"eventSource": "sts.amazonaws.com",
"eventName": "AssumeRole",
"awsRegion": "us-east-1",
"sourceIPAddress": "ecs-tasks.amazonaws.com",
"userAgent": "ecs-tasks.amazonaws.com",
"requestParameters": {
"roleSessionName": "d190d14a-2404-45d6-9113-4eda22d7f2c7",
"roleArn": "arn:aws:iam::653711331788:role/level3"
},
"responseElements": {
"credentials": {
"sessionToken": "FQoGZXIvYXdzEFAaDEbnJXLefTT+kjlmKSKSBNgEUj8tJVL+szjaH5q2npYc2FIPgrLmfkRjK9KqtSW7+lo4WxteBTd77aeAcmIip4GceNBbU86zxGgS1IdNBzEOLnDw6biAzijG0Du/Qazx136qjy+kahHxPlR36C4y/0QrCUZpTFmP3uELsRIKkvhGvuBr6S10pTOZ+GjtUXN3iFV8Ea0KOo/fSP0d4LbZGwI957aJxs2I7N8ji/lKTfwPdq+sxXvSWnaOseinUxZUDS0zdI69CKb6C+qwhR5YTifqyuOvC9OoSlfcBN2FyHpRZf5Bd+Z+mPYTldbAvD/HcdbQo7U4jqlR2WGuXoBfwvypt/Kb6HtPp4g9O0HlTCc7Sb4uiJY81WMbaNFmmYXyj62gi+BN5QaA90YhWn9cU1x9gqt0uEgvSk/RdrwtulTtNyJcuuFvhlD1gaJHGc4eCoWApr+J9nrbPTvSo00sc8IYIVvwOi3NRsmP+ZA9aQOV/qg2L1cYxScQrQ/pKVOnYWJ4XuB0WL8gRYdo1bGI6LWGAtOV+fzVoXU0SfWH7UmPcvttqkWsv1Rr50pspmJneXeY6Ge1szNsvyHqmPFJj7GAXnEKl9xthHK3IaDc6HEprFqYw8SyQqYWGdpeism2S+V4XpIMbQJlC06BxyOg94H0Fiffzs8wwnCUpBU0s69X2tN8sxb4U3FqKl28mwCOuuaweZeGWq5MWxU7Fop2dmjaKN+u/N8F",
"accessKeyId": "ASIAZQNB3KHGNXWXBSJS",
"expiration": "Nov 29, 2018 4:31:59 AM"
}
},
"requestID": "6b7d6c60-f35d-11e8-becc-39e7d43d4afe",
"eventID": "6177ca7e-860e-482c-bde9-50c735af58d6",
"resources": [
{
"ARN": "arn:aws:iam::653711331788:role/level3",
"accountId": "653711331788",
"type": "AWS::IAM::Role"
}
],
"eventType": "AwsApiCall",
"recipientAccountId": "653711331788",
"sharedEventID": "1d18bf74-8392-4496-9dc4-a45cb799b8b4"
},
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AWSService",
"invokedBy": "ecs-tasks.amazonaws.com"
},
"eventTime": "2018-11-28T22:31:59Z",
"eventSource": "sts.amazonaws.com",
"eventName": "AssumeRole",
"awsRegion": "us-east-1",
"sourceIPAddress": "ecs-tasks.amazonaws.com",
"userAgent": "ecs-tasks.amazonaws.com",
"requestParameters": {
"roleSessionName": "d190d14a-2404-45d6-9113-4eda22d7f2c7",
"roleArn": "arn:aws:iam::653711331788:role/ecsTaskExecutionRole"
},
"responseElements": {
"credentials": {
"sessionToken": "FQoGZXIvYXdzEFAaDAhihig4wSQxSSiMjSKSBOW1B0tXzW8SIP7MtRZEikuZamVqb7hEQtmgX5LDeDzpOxpP4G9U8r3cZK74HNErY8W+Scri3Y8NVr/VO7MyzIXnpXiEDo10JfNYnA+urhxB+rYtPF6o8uzm40w/lqdq1/DyOkYYuFRtqWfS4Jy1t83KYAFTuX5IkTGekMidOkIcdnHjsZKQSqKs4U3MQ4doUG4nCyEERDv9yckTPq/R4fKEPTF1BN0jycyiSiR3OTPAWaLGMGxHthAKSA5IXosrPBAq0yD2HhSKZc0kbskCZyGOQCcmVAQK4IdEjyk4Lytc+PEDasvWiGoCQPEqlwwqFJm7EPm838MjCTi4ojN9nVYRP9hFYkXdvnVG6ScwoBfbg135vN1bqYgEKDncW780xUBRYwElM4Q2/6zv7DMj0UegbRJmAwtys4phLrItQqNLWPmBbW/pNgMYoT1IKfzDmuc27AyTHtL8t25hkYOLWZG19EYKm+XeHU9gbs0aDTPssjBFPZp7ssR35IHhgcw5m+etXSoXMxMuHNbVR46ZJ6uieR8roEZ8QfiIijyB8J7i2sH2JOY0HIOonbVYmqdtv++0D+1idTO+6ZbXJIKhEmDmZZDeenF2ZXQGH4PgA4udIdBXnhVLzVFkEKc1MWG3aAhuOhZLvnXkOpkn1XjwfDx3N0UyenOUR4x/gkY9+MEXMZAyO8Va2Y1vNZvnSCvTJtHDKN+u/N8F",
"accessKeyId": "ASIAZQNB3KHGMG7YRUEW",
"expiration": "Nov 29, 2018 4:31:59 AM"
}
},
"requestID": "6b80a0b1-f35d-11e8-becc-39e7d43d4afe",
"eventID": "457af3a9-0b1b-44ca-91e1-8f4a0f873149",
"resources": [
{
"ARN": "arn:aws:iam::653711331788:role/ecsTaskExecutionRole",
"accountId": "653711331788",
"type": "AWS::IAM::Role"
}
],
"eventType": "AwsApiCall",
"recipientAccountId": "653711331788",
"sharedEventID": "5397e1a9-82c7-4a00-9b1c-e44cbd688aa1"
}
]
}
To access logs for this region? you need to login and go to CloudTrail. If SSM is enabled, captures this activity as well. Similarly, session activity audit logs can be captured in S3 or CloudWatch logs.
โ ๏ธ Not any CloudTrail log entry will have a
resources
part. For example,LookupEvents
doesn’t.
๐ก By the way, using CLI to lookup the events gets logged by CloudTrail.
CloudTrail events that are relevant from security perspective - https://www.chrisfarris.com/post/aws-ir/.
๐ BTFM
Here is my bash to retrieve CT logs for a all keys in the account, filtered by API call Decrypt
. I also have a Python version and I am working on the Jupyter book for this.
loggroupname="aws-controltower/CloudTrailLogs"
for keyname in $(aws kms list-aliases --profile ProfileName | jq -r '.Aliases[].AliasName')
do
echo "keyname is \e[92m$keyname\e[0m"
key_arn=$(aws kms describe-key --key-id $keyname --profile ProfileName | jq -r '.KeyMetadata.Arn')
echo "Key ARN is $key_arn"
query_id=$(aws logs --profile ProfileName start-query --log-group-name $loggroupname --start-time 1655588640 --end-time 1659109342 --query-string "FIELD eventTime, userIdentity.userName, sourceIPAddress | SORT eventTime DESC | FILTER eventName = 'Decrypt' AND resources.0.ARN = '$key_arn'" | jq -r '.queryId')
sleep 60
echo "\e[96m$(aws logs --profile ProfileName get-query-results --query-id $query_id)\e[0m"
done
If you receive something like the following snippet for all the keys and you suspect some of them were used at least once during the timespan, try another log group. In my case, the log group that produced such results was disabled (checked in AWS GUI Management Console).
{
"results": [],
"statistics": {
"recordsMatched": 0.0,
"recordsScanned": 0.0,
"bytesScanned": 0.0
},
"status": "Complete"
}
Another useful command to see view these events with fzf
timestamps:
aws logs get-query-results --profile ProfileName --query-id "$query_id" | jq -c '.results[]' | fzf --multi --cycle --reverse --preview-window=right:80%:wrap --preview 'echo {} | jq .'
The following examples are limited to three events. You’ll need to remove --max-results 3
after testing it.
ย aws cloudtrail lookup-events --profile ProfileName --max-results 3 | jq -c '.Events[]' | fzf --multi --cycle --reverse --preview-window=right:80%:wrap --preview 'echo {} | jq .' # --profile ProfileName is optional and is used in case of SSO auth only.
# limit by UNIX timestamps
aws cloudtrail lookup-events --profile ProfileName --start-time 1655535200 --end-time 1655542800 | jq -c '.Events[]' | fzf --multi --cycle --reverse --preview-window=right:80%:wrap --preview 'echo {} | jq .'
# useful command to covert to UNIX timestamps
date -ju -f "%F %T" "2022-06-18 08:00:00" "+%s"
VPC Flow Logs
If these are enabled, you will be able to tell, who connected to what resource and when. Flow Logs are not enabled by default. It’s not the same as pcap file! It doesn’t provide full packet info. Just statistics, addresses, ports, result (accept/reject).
Logs are sent then either to the CloudWatch or S3 (depending on the configuration). An IAM role needs to be created prior to that. Here are the minimum requirements for the CloudWatch role: CreateLogGroup, CreateLogStream, PutLogEvents, DescribeLogGroups, DescribeLogStreams need to beallowed for all resources. To be able to assume this role, VPC Flow Logs service needs to have its own policy where "service":"vpc-flow-logs.amazonaws.com"
is set to allow AssumeRole
action. See here for more details.
Log breakthrough. Looks like a simple string with a space as a delimiter. Each field has its place within the string.
Traffic logs between VPCs withing the same org/acc.
If these are enabled, you will be able to tell, who connected to what resource and when. Flow Logs are not enabled by default. AWS states that loggin rejected traffic is sufficient for bad actor detection. The recommended log retention period is 1 year. However, these logs can be archived and stored elsewhere where it’s cheaper.
It’s not the same as pcap file! It doesn’t provide full packet info. Just statistics, addresses, ports, result (accept/reject).
Logs are sent then either to the CloudWatch or S3 (depending on the configuration). Min delay between the capture and its copying to the above mentioned services is 1-6 minutes.
๐ BTFM
Retrieve the logs:
aws ec2 describe-flow-logs # append --profile for SSO or make sure you have the correct creds in ~/.aws/config
aws logs filter-log-events --log-group-name <groupname> # append --profile for SSO or make sure you have the correct creds in ~/.aws/config
Filter the logs - docs.
Resource Timeline
In AWS go to Config โ Resources โ <resource name>
โ Resource Timeline (in the upper right corner). Shows all the changed made to the resource (created, configuration changed, resource deleted).
IAM Activity
Two places to look at it.
- Credentials report
iam/home#/credential_report
Contains the following fields:
- user
- arn
- user_creation_time
- password_enabled
- password_last_used
- password_last_changed
- password_next_rotation
- mfa_active
- access_key_1_active
- access_key_1_last_rotated
- access_key_1_last_used_date
- access_key_1_last_used_region
- access_key_1_last_used_service
- access_key_2_active
- access_key_2_last_rotated
- access_key_2_last_used_date
- access_key_2_last_used_region
- access_key_2_last_used_service
- cert_1_active
- cert_1_last_rotated
- cert_2_active
- cert_2_last_rotated
Access Advisor
Go to IAM -> username -> Access Advisor. Not every user might (and should) have access to this one. Access Advisor shows the services that this user can access and when those services were last accessed.
VPC Traffic Mirroring
A great idea to use. Deploying an EC2 with some IDS, mirror traffic there for detection. Filter for the most relevant stuff to pass it to SIEM?
VPN
If the organization uses a VPN in order to connect, an investigator will need the VPN logs as well (IPsec or OpenVPN).
Logs Archieved In S3
CloudTrail logs are not available forever. If the company backs up all events as they come and sends them to a S3 bucket. How to query them? To search for a particular string within the logs. For example, by event_id or username.
Bash script below will look through the files recursively and run grep
against each. This is very time consuming, so you’d better narrow down the search and choose the keyword wisely.
โ ๏ธ Make sure youโve logged in the Log Archive first and you have set up the profile for SSO in
~/.aws/config
.
keyword="bucket-name"
for file in $(aws s3 ls --profile ProfileName --recursive "s3://bucket-name/path/to/CloudTrail/logs/")
do
if echo $file | grep -q "unique string"; then
echo "========================================"
echo "Looking for keywords in the file \e[36m$file...\e[0m"
echo ""
result=$(aws s3 cp "s3://bucket-name/$file" --profile ProfileName - | gzip -d - )
if echo $result | grep -q $keyword; then
echo $result | jq -C .
echo $result | jq -C . >> result.txt
fi
echo "========================================"
fi
done
$(aws s3 ls --profile ProfileName --recursive "s3://bucket-name/path/to/CloudTrail/logs/")
will return a table: time - bytes - path (each on a separate line). So, when iterating through the files in S3 buckets, you’ll get time after the first iteration, size in bytes after the second one and the last iteration will get you the path itself. For the script to work, we need path only, that’s why I added this line if echo $file | grep -q "bucket-name"; then. This command aws s3 cp "s3://bucket-name/$file" --profile Backup - | gzip -d -
will copy the contents of the file to the stdout (-
) and pipe to gzip
(since log files are usually compressed) and send to stdout again (-
). If the file has the $keyword
(if echo $result | grep -q $keyword; then
) I run jq -C .
(coloured jq
output) and also save the result in a text file (append).
CloudWatch
Similarly, session activity audit logs can be captured in S3 or CloudWatch logs.
SQS
This can be used for various things but one of the usages could be a log buffer.
๐ BTFM
aws sqs list-queues --profile ProfileName
aws sqs send-message --queue-url <value> --message-body <value>
aws sqs receive-message --queue-url <value>
๐งช I would like to see if messages will disappear from the queue once
receive-message
was invoked.
Security Monkey
???
References
[1] IR in AWS